Twitter Streamer | Ignite Documentation

Ignite Summit 2024 — Call For Speakers Now Open — Learn more

Edit

Twitter Streamer

Ignite Twitter Streamer module consumes tweets from Twitter and feeds the transformed key-value pairs <tweetId, text> into Ignite.

To stream data from Twitter into Ignite, you need to:

  1. Import Ignite Twitter Module with Maven and replace ${ignite-twitter-ext.version} with the actual Ignite Twitter Extension version you are interested in.

    <dependency>
      <groupId>org.apache.ignite</groupId>
      <artifactId>ignite-twitter-ext</artifactId>
      <version>${ignite-twitter-ext.version}</version>
    </dependency>
  2. In your code, set the necessary parameters and start the streamer, like so:

    IgniteDataStreamer dataStreamer = ignite.dataStreamer("myCache");
    dataStreamer.allowOverwrite(true);
    dataStreamer.autoFlushFrequency(10);
    
    OAuthSettings oAuthSettings = new OAuthSettings("setting1", "setting2", "setting3", "setting4");
    
    TwitterStreamer<Integer, String> streamer = new TwitterStreamer<>(oAuthSettings);
    streamer.setIgnite(ignite);
    streamer.setStreamer(dataStreamer);
    
    Map<String, String> params = new HashMap<>();
    params.put("track", "apache, twitter");
    params.put("follow", "3004445758");
    
    streamer.setApiParams(params);// Twitter Streaming API params.
    streamer.setEndpointUrl(endpointUrl);// Twitter streaming API endpoint.
    streamer.setThreadsCount(8);
    
    streamer.start();

Refer to Twitter streaming API documentation for more information on various streaming parameters.