Apache Flume Sink | Ignite Documentation

Ignite Summit 2023 — Watch on demand — Register now!


Apache Flume Sink


Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. (https://github.com/apache/flume).

IgniteSink is a Flume sink that extracts events from an associated Flume channel and injects into an Ignite cache.

IgniteSink and its dependencies have to be included in the agent’s classpath, as described in the following subsection, before starting the Flume agent.

Setting Up

  1. Create a transformer by implementing EventTransformer interface.

  2. Create ignite directory inside plugins.d directory which is located in ${FLUME_HOME}. If the plugins.d directory is not there, create it.

  3. Build it and copy to ${FLUME_HOME}/plugins.d/ignite-sink/lib.

  4. Copy other Ignite-related jar files from Apache Ignite distribution to ${FLUME_HOME}/plugins.d/ignite-sink/libext to have them as shown below.

    `-- ignite
     |-- lib
     |   `-- ignite-flume-transformer-x.x.x.jar <-- your jar
     `-- libext
         |-- cache-api-1.0.0.jar
         |-- ignite-core-x.x.x.jar
         |-- ignite-flume-ext.x.x.x.jar <-- IgniteSink
         |-- ignite-spring-x.x.x.jar
         |-- spring-aop-4.1.0.RELEASE.jar
         |-- spring-beans-4.1.0.RELEASE.jar
         |-- spring-context-4.1.0.RELEASE.jar
         |-- spring-core-4.1.0.RELEASE.jar
         `-- spring-expression-4.1.0.RELEASE.jar
  5. In Flume configuration file, specify Ignite configuration XML file’s location with cache properties (see flume/src/test/resources/example-ignite.xml for a basic example) with the cache name specified for cache creation. Also specify the cache name (same as in Ignite configuration file), your `EventTransformer’s implementation class, and, optionally, batch size. All properties are shown in the table below (required properties are in bold).

    Property Name Description Default Value




    The component type name. Needs to be org.apache.ignite.stream.flume.IgniteSink



    Ignite configuration XML file



    Cache name. Same as in igniteCfg



    Your implementation of org.apache.ignite.stream.flume.EventTransformer



    Number of events to be written per transaction


The sink configuration part of agent named a1 can look like this:

a1.sinks.k1.type = org.apache.ignite.stream.flume.IgniteSink
a1.sinks.k1.igniteCfg = /some-path/ignite.xml
a1.sinks.k1.cacheName = testCache
a1.sinks.k1.eventTransformer = my.company.MyEventTransformer
a1.sinks.k1.batchSize = 100

After specifying your source and channel (see Flume’s docs), you are ready to run a Flume agent.