Flume

Ingest data in HDFS(Solr, HBase) from different sources.

Configuring an agent

Have a configuration file flume.conf (java properties file) Configuration example for file feed and output to logger.

agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1

agent1.sources.source1.channels = channel1
agent1.sources.source1.type = spooldir
agent1.sources.source1.spoolDir = /tmp/spooldir

agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.type = logger

agent1.channels.channel1.type = file

Create the source folder

mkdir /tmp/spooldir

Start an agent

flume-ng agent --name <agent_name> --conf <config_dir> --conf-file <config_file_path.conf> [-Dflume.root.logger=INFO,console]

Configure a memory channel

agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000 # nb events