Apache Storm Streamer | Ignite Documentation

Ignite Summit 2024 — Call For Speakers Now Open — Learn more

Edit

Apache Storm Streamer

Apache Ignite Storm Streamer module provides streaming via Storm to Ignite.

Starting data transfer to Ignite can be done with the following steps.

  1. Import Ignite Storm Streamer Module In Maven Project. If you are using Maven to manage dependencies of your project, you can add Storm module dependency like this (replace ${ignite-storm-ext.version} with actual Ignite Storm Extension version you are interested in):

    <project xmlns="http://maven.apache.org/POM/4.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                            http://maven.apache.org/xsd/maven-4.0.0.xsd">
        ...
        <dependencies>
            ...
            <dependency>
                <groupId>org.apache.ignite</groupId>
                <artifactId>ignite-storm-ext</artifactId>
                <version>${ignite-storm-ext.version}</version>
            </dependency>
            ...
        </dependencies>
        ...
    </project>
  2. Create an Ignite configuration file (see example-ignite.xml in modules/storm/src/test/resources/example-ignite.xml) and make sure it is accessible from the streamer.

  3. Make sure your key-value data input to the streamer is specified with the field named ignite (or a different one you configure with StormStreamer.setIgniteTupleField(…​)). See TestStormSpout.declareOutputFields(…​) for an example.

  4. Create a topology with the streamer, make a jar file with all dependencies and run the following

    storm jar ignite-storm-streaming-jar-with-dependencies.jar my.company.ignite.MyStormTopology