安装

解压安装包

解压FLume 安装包

tar -zxvf apache-flume-1.9.0-bin.tar.gz -C ./

环境变量设置

export MAVEN_HOME=/opt/module/apache-maven-3.6.3
export PATH=$PATH:$MAVEN_HOME/bin

source /etc/profile

Flume 配置文件

配置Flume conf目录下的flume-env.sh文件，设置JAVA_HOME

export JAVA_HOME=/usr/local/jdk1.8.0_251

运行

运行flume-ng version 查看是否安装成功

flume-ng version

监听日志数据并上传HDFS

在conf下配置管道文件

a1.sources = s1
a1.sinks = sk1
a1.channels = c1

a1.sources.s1.type = spooldir
a1.sources.s1.spoolDir = /opt/module/hadoop-3.1.3/logs/

a1.channels.c1.type = memory

a1.sinks.sk1.type = hdfs
a1.sinks.sk1.hdfs.path = hdfs://master:9000/tmp/flume
a1.sinks.sk1.hdfs.fileType = DataStream

a1.sources.s1.channels = c1
a1.sinks.sk1.channel = c1

命令执行

flume-ng agent -c conf -f conf/toHDFS.conf -n a1 -Dflume.root.logger=DEBUG,console

hdfs 查看上传日志

hdfs dfs -ls /tmp/flume