Apache Flume Sink | Ignite Documentation

Ignite Summit 2025 – Call for Speakers Now Open – Learn more 

Apache Ignite logo
Edit

Apache Flume Sink

Overview

Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. (https://github.com/apache/flume).

IgniteSink is a Flume sink that extracts events from an associated Flume channel and injects into an Ignite cache.

IgniteSink and its dependencies have to be included in the agent’s classpath, as described in the following subsection, before starting the Flume agent.

Setting Up

  1. Create a transformer by implementing EventTransformer interface.

  2. Create ignite directory inside plugins.d directory which is located in ${FLUME_HOME}. If the plugins.d directory is not there, create it.

  3. Build it and copy to ${FLUME_HOME}/plugins.d/ignite-sink/lib.

  4. Copy other Ignite-related jar files from Apache Ignite distribution to ${FLUME_HOME}/plugins.d/ignite-sink/libext to have them as shown below.

    plugins.d/
`-- ignite
 |-- lib
 |   `-- ignite-flume-transformer-x.x.x.jar <-- your jar
 `-- libext
     |-- cache-api-1.0.0.jar
     |-- ignite-core-x.x.x.jar
     |-- ignite-flume-ext.x.x.x.jar <-- IgniteSink
     |-- ignite-spring-x.x.x.jar
     |-- spring-aop-4.1.0.RELEASE.jar
     |-- spring-beans-4.1.0.RELEASE.jar
     |-- spring-context-4.1.0.RELEASE.jar
     |-- spring-core-4.1.0.RELEASE.jar
     `-- spring-expression-4.1.0.RELEASE.jar

  5. In Flume configuration file, specify Ignite configuration XML file’s location with cache properties (see flume/src/test/resources/example-ignite.xml for a basic example) with the cache name specified for cache creation. Also specify the cache name (same as in Ignite configuration file), your `EventTransformer’s implementation class, and, optionally, batch size. All properties are shown in the table below (required properties are in bold).

    Property Name Description Default Value

    channel

    -

    type

    The component type name. Needs to be org.apache.ignite.stream.flume.IgniteSink

    -

    igniteCfg

    Ignite configuration XML file

    -

    cacheName

    Cache name. Same as in igniteCfg

    -

    eventTransformer

    Your implementation of org.apache.ignite.stream.flume.EventTransformer

    -

    batchSize

    Number of events to be written per transaction

    100

The sink configuration part of agent named a1 can look like this:

a1.sinks.k1.type = org.apache.ignite.stream.flume.IgniteSink
a1.sinks.k1.igniteCfg = /some-path/ignite.xml
a1.sinks.k1.cacheName = testCache
a1.sinks.k1.eventTransformer = my.company.MyEventTransformer
a1.sinks.k1.batchSize = 100

After specifying your source and channel (see Flume’s docs), you are ready to run a Flume agent.

© 2024 The Apache Software Foundation.
Apache, Apache Ignite, the Apache feather and the Apache Ignite logo are either registered trademarks or trademarks of The Apache Software Foundation.