Namespace Apache.Ignite.Core.Datastream

Classes

DataStreamerDefaults

Data streamer configuration defaults.

StreamTransformer<TK, TV, TArg, TRes>

Convenience adapter to transform update existing values in streaming cache based on the previously cached value.

StreamVisitor<TK, TV>

Convenience adapter to visit every key-value tuple in the stream. Note that the visitor does not update the cache.

Interfaces

IDataStreamer<TK, TV>

Data streamer loads data efficiently into cache. Updates are buffered and mapped to primary nodes to ensure minimal data movement and optimal resource utilization.

Note that streamer loads data to remote nodes in parallel, so cache updates can be reordered.

Also note that IDataStreamer is not the only way to load data into cache. Alternatively you can use LoadCache(Action<TK, TV>, Object[]) method to load data from underlying data store. You can also use standard cache put and putAll operations as well, but they most likely will not perform as well as this class for loading data. And finally, data can be loaded from underlying data store on demand, whenever it is accessed - for this no explicit data loading step is needed.

IDataStreamer supports the following configuration properties:

  • PerNodeBufferSizeWhen entries are added to data streamer they are not sent to Ignite right away and are buffered internally for better performance and network utilization. This setting controls the size of internal per-node buffer before buffered data is sent to remote node. Default value is 1024.
  • PerThreadBufferSizeWhen entries are added to data streamer they are not sent to Ignite right away and are buffered internally on per thread basis for better performance and network utilization. This setting controls the size of internal per-thread buffer before buffered data is sent to remote node. Default value is 4096.
  • PerNodeParallelOperationsSometimes data may be added to the data streamer faster than it can be put in cache. In this case, new buffered load messages are sent to remote nodes before responses from previous ones are received. This could cause unlimited heap memory utilization growth on local and remote nodes. To control memory utilization, this setting limits maximum allowed number of parallel buffered load messages that are being processed on remote nodes. If this number is exceeded, then data streamer add/remove methods will block to control memory utilization. Default value is 16.
  • AutoFlushFrequencyAutomatic flush frequency in milliseconds. Essentially, this is the time after which the streamer will make an attempt to submit all data added so far to remote nodes. Note that there is no guarantee that data will be delivered after this concrete attempt (e.g., it can fail when topology is changing), but it won't be lost anyway. Disabled by default (default value is 0).
  • IsolatedDefines if data streamer will assume that there are no other concurrent updates and allow data streamer choose most optimal concurrent implementation. Default value is false.

All members are thread-safe and may be used concurrently from multiple threads.

IStreamReceiver<TK, TV>

Updates cache with batch of entries. Usually it is enough to configure AllowOverwrite property and appropriate internal cache receiver will be chosen automatically. But in some cases custom implementation may help to achieve better performance.