Data streamer configuration defaults.
Convenience adapter to transform update existing values in streaming cache based on the previously cached value.
Convenience adapter to visit every key-value tuple in the stream. Note that the visitor does not update the cache.
Data streamer is responsible for loading external data into cache. It achieves it by properly buffering updates and properly mapping keys to nodes responsible for the data to make sure that there is the least amount of data movement possible and optimal network and memory utilization.
Note that streamer will load data concurrently by multiple internal threads, so the data may get to remote nodes in different order from which it was added to the streamer.
Also note that
IDataStreamer is not the only way to load data into cache.
Alternatively you can use
LoadCache(Action<TK, TV>, Object)
method to load data from underlying data store. You can also use standard cache
putAll operations as well, but they most likely will not perform
as well as this class for loading data. And finally, data can be loaded from underlying
data store on demand, whenever it is accessed - for this no explicit data loading step
IDataStreamer supports the following configuration properties:
- PerNodeBufferSizeWhen entries are added to data streamer they are not sent to Ignite right away and are buffered internally for better performance and network utilization. This setting controls the size of internal per-node buffer before buffered data is sent to remote node. Default value is 1024.
- PerThreadBufferSizeWhen entries are added to data streamer they are not sent to Ignite right away and are buffered internally on per thread basis for better performance and network utilization. This setting controls the size of internal per-thread buffer before buffered data is sent to remote node. Default value is 4096.
- PerNodeParallelOperationsSometimes data may be added to the data streamer faster than it can be put in cache. In this case, new buffered load messages are sent to remote nodes before responses from previous ones are received. This could cause unlimited heap memory utilization growth on local and remote nodes. To control memory utilization, this setting limits maximum allowed number of parallel buffered load messages that are being processed on remote nodes. If this number is exceeded, then data streamer add/remove methods will block to control memory utilization. Default value is 16.
- AutoFlushFrequencyAutomatic flush frequency in milliseconds. Essentially, this is the time
after which the streamer will make an attempt to submit all data added so far to remote
nodes. Note that there is no guarantee that data will be delivered after this concrete
attempt (e.g., it can fail when topology is changing), but it won't be lost anyway.
Disabled by default (default value is
- IsolatedDefines if data streamer will assume that there are no other concurrent
updates and allow data streamer choose most optimal concurrent implementation. Default value
All members are thread-safe and may be used concurrently from multiple threads.
Updates cache with batch of entries. Usually it is enough to configure AllowOverwrite property and appropriate internal cache receiver will be chosen automatically. But in some cases custom implementation may help to achieve better performance.