flume单agent安装

官网下载[较慢] | 蓝奏云盘下载

1、创建flume文件夹，安装telnet远程协议软件
mkdir /usr/apps/flume
yum install -y telnet
2、上传安装包到flume文件夹下
#rz命令，如果没有就安装：yum install -y lrzsz
rz
3、flume-env.sh环境变量配置
cd  /usr/apps/flume/apache-flume-1.8.0-bin/conf
    • （1）创建配置文件
cp flume-env.sh.template flume-env.sh
    • （2）编辑配置文件
vim  flume-env.sh

#vim和vi命令同理，都是编辑文件，在Xshell中vim更能多彩【有颜色】

    • （3）在flume-env.sh中添加
export JAVA_HOME=/usr/apps/java/jdk1.8.0_221
#你的jdk路径
4、配置flume文件系统环境变量
vim /etc/profile

在最后两行添加以下内容

#你的flume文件夹路径
export FLUME_HOME=/usr/apps/flume/apache-flume-1.8.0-bin
export PATH=$PATH:$FLUME_HOME/bin

使环境变量立即生效

source /etc/profile
5、配置flume采集方案
    • (1)进入目录flume的conf目录
cd /usr/apps/flume/apache-flume-1.8.0-bin/conf
    • (2)copy配置文件
cp flume-conf.properties.template  flume-conf.properties
    • (3)编辑配置文件
vim  flume-conf.properties
    • (4)更改内容：

29行更改为“netcat”，并添加属性👇
#查看行号命令为：set nu[esc+英文冒号+set nu]小写

agent.sources.seqGenSrc.type = netcat
agent.sources.seqGenSrc.bind = localhost
agent.sources.seqGenSrc.port = 44444

#源文件如下，自行对比

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#  http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.


# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called 'agent'

agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = loggerSink

# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = netcat
agent.sources.seqGenSrc.bind = localhost
agent.sources.seqGenSrc.port = 44444
# The channel can be defined as follows.
agent.sources.seqGenSrc.channels = memoryChannel

# Each sink's type must be defined
agent.sinks.loggerSink.type = logger

#Specify the channel the sink should use
agent.sinks.loggerSink.channel = memoryChannel

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 100
6、启动单个agent
NOTE:注意自己的flume路径
flume-ng agent --conf conf --conf-file /usr/apps/flume/apache-flume-1.8.0-bin/conf/flume-conf.properties --name agent -Dflume.root.logger=INFO,console
7、再新打开一个master窗口
    • (1)执行Telnet命令
telnet localhost 44444
    • (2)测试
NOTE:输入hello 然后回车
8、切换到原始的master节点
NOTE:可以查看到flume采集到 刚才输入的hello信息
20/12/07 14:51:14 INFO node.Application: Starting Sink loggerSink
20/12/07 14:51:14 INFO node.Application: Starting Source seqGenSrc
20/12/07 14:51:14 INFO source.NetcatSource: Source starting
20/12/07 14:51:15 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
<!--上方为启动信息，下方为采集到的信息-->
20/12/07 14:52:03 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 0D                               hello. }
20/12/07 14:52:09 INFO sink.LoggerSink: Event: { headers:{} body: E4 BD A0 E5 A5 BD 0D                            ....... }
20/12/07 14:52:15 INFO sink.LoggerSink: Event: { headers:{} body: 77 6F 72 08 08 08 0D                            wor.... }
20/12/07 14:52:20 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 0D                               hello. }
20/12/07 14:52:22 INFO sink.LoggerSink: Event: { headers:{} body: 77 6F 72 6C 64 0D                               world. }
20/12/07 14:52:31 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 2C 77 6F 72 6C 64 21 0D          hello,world!. }

单flume总结

安装flume、telnet服务

在conf文件夹中统一修改文件：

cp flume-env.sh.template flume-env.sh：jdk、cp flume-conf.properties.template flume-conf.properties添加修改三行

修改profile配置文件，添加flume环境变量

启动agent服务

复制一个master窗口，连接接口【telnet localhost 44444】

测试

查看

多agent安装

首先启动hadoop集群三个节点。master用来汇聚日志，slave1，slave2用来采集日志

1、master主机下载安装包：

官网下载[较慢] | 蓝奏云盘下载

2、创建flume文件夹
mkdir /usr/apps/flume
3、上传安装包到flume文件夹下
cd /usr/apps/flume
rz

解压：tar -zxvf apache-flume-1.8.0-bin.tar.gz

4、flume-env.sh环境变量配置
cd  /usr/apps/flume/apache-flume-1.8.0-bin/conf
    • (1)创建配置文件
cp flume-env.sh.template flume-env.sh
    • (2)编辑配置文件
vim  flume-env.sh
    • (3)在flume-env.sh中添加

注意自己的jdk路径

export JAVA_HOME=/usr/apps/java/jdk1.8.0_221
5、配置flume文件系统环境变量
vim  /etc/profile
    • (1)在最后两行添加

注意你的flume文件路径

export FLUME_HOME=/usr/apps/flume/apache-flume-1.8.0-bin
export PATH=$PATH:$FLUME_HOME/bin
    • (2)使环境变量立即生效
source /etc/profile
6、将创建好的配置远程复制到slave1、slave2节点
    • (1)复制安装包
scp -r /flume/  root@slave1:/
scp -r /flume/  root@slave2:/
    • (2)复制环境变量
scp -r /etc/profile root@slave1:/etc/
scp -r /etc/profile root@slave2:/etc/
    • (3)到slave1、slave2执行source命令
source  /etc/profile
7、slave1、slave2创建日志文件
    • (1)slave1执行：
mkdir  /usr/apps/flume/logs
cd  /usr/apps/flume/logs

vim  access.log   添加内容：slave1  access.log 
vim  nginx.log  添加内容：slave1  nginx.log 
vim   web.log  添加内容：slave1  web.log
    • (2)slave2执行：
mkdir /usr/apps/flume/logs
cd /usr/apps/flume/logs

vim  access.log   添加内容：slave2  access.log 
vim  nginx.log  添加内容：slave2  nginx.log 
vim   web.log  添加内容：slave2  web.log
8、master、slave1、slave2创建采集日志配置文件
    • (1)slave1配置文件：
cd  /usr/apps/flume/apache-flume-1.8.0-bin/conf
vim  flume-conf.properties

添加内容如下：

#命名
a1.sources = r1 r2 r3
a1.sinks = k1
a1.channels = c1
#资源1
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /usr/apps/flume/logs/access.log
a1.sources.r1.channels = c1
#由于3个资源同时发送给node03，node03不认识都是哪个路径资源发送的，所以需要拦截器，标识一下
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = static
a1.sources.r1.interceptors.i1.key = key
a1.sources.r1.interceptors.i1.value = access_log
#资源2
a1.sources.r2.type = exec
a1.sources.r2.command = tail -F /usr/apps/flume/logs/nginx.log
a1.sources.r2.channels = c1
#拦截器
a1.sources.r2.interceptors = i2
a1.sources.r2.interceptors.i2.type = static
a1.sources.r2.interceptors.i2.key = key
a1.sources.r2.interceptors.i2.value = nginx_log
#资源3
a1.sources.r3.type = exec
a1.sources.r3.command = tail -F /usr/apps/flume/logs/web.log
a1.sources.r3.channels = c1
#拦截器
a1.sources.r3.interceptors = i3
a1.sources.r3.interceptors.i3.type = static
a1.sources.r3.interceptors.i3.key = key
a1.sources.r3.interceptors.i3.value = web_log
#sink
a1.sinks.k1.type = avro
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = master
a1.sinks.k1.port = 41414
#channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
    • (2)将slave1的采集配置文件远程复制到slave2

注意路径

scp -r /usr/apps/flume/apache-flume-1.8.0-bin/conf/flume-conf.properties root@slave2:/usr/apps/flume/apache-flume-1.8.0-bin/conf/
    • (3)创建master采集配置文件
cd  /flume/apache-flume-1.8.0-bin/conf
vim  flume-conf.properties

添加内容如下：

#命名
a1.sources = r1
a1.channels = c1
a1.sinks = k1
#资源
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 192.168.1.4//注意IP
a1.sources.r1.port = 41414
#定义通道
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
#定义sink，注意（修改）IP【master】
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path =hdfs://master:9000/flume/logs/%{key}/%y-%m-%d/
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat = Text
9、master、slave1、slave2启动flume agent
    • 三个节点分别执行：

注意路径

flume-ng agent --conf conf --conf-file /usr/apps/flume/apache-flume-1.8.0-bin/conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console
10、查看日志采集结果
在Xshell中新开启一个master窗口

查看hdfs上产生文件路径：

hadoop  fs   -lsr /

查看文件内容：

hadoop fs -cat  /文件路径

多agent总结

flume-env.sh环境变量配置，用到jdk路径

配置flume文件系统环境变量【profile】

将创建好的配置远程复制到slave1、slave2节点

三台节点初始化环境变量【source /etc/profile】

slave1、slave2创建日志文件夹logs

slave1、slave2创建编辑日志文件

master【独立】、slave1、slave2创建采集日志配置文件【slave1 = slave2】(slave1用scp命令发送给slave2)

三个节点分别启动agent

查看日志采集结果

NOTE:经测试无误，如有错误请认真查看

如有问题请点击下方联系笔者

联系笔者QQ | 联系笔者邮箱