🪅
Hadoop tools
  • Introduction
  • Ingestion
    • Sqoop
    • Flume
  • Transformation
    • Pig
    • Hive
    • Spark - Scala
      • Examples
    • Spark - Python
  • NoSQL
    • HBase
  • Big Data Principles
  • Big Data Architectures
Powered by GitBook
On this page
  • Configuring an agent
  • Start an agent
  • Configure a memory channel

Was this helpful?

  1. Ingestion

Flume

Ingest data in HDFS(Solr, HBase) from different sources.

Configuring an agent

Have a configuration file flume.conf (java properties file) Configuration example for file feed and output to logger.

agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1

agent1.sources.source1.channels = channel1
agent1.sources.source1.type = spooldir
agent1.sources.source1.spoolDir = /tmp/spooldir

agent1.sinks.sink1.channel = channel1
agent1.sinks.sink1.type = logger

agent1.channels.channel1.type = file

Create the source folder

mkdir /tmp/spooldir

Start an agent

flume-ng agent --name <agent_name> --conf <config_dir> --conf-file <config_file_path.conf> [-Dflume.root.logger=INFO,console]

Configure a memory channel

agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000 # nb events
PreviousSqoopNextTransformation

Last updated 5 years ago

Was this helpful?