Apache Ignite 2.16.0: Cache dumps, Calcite engine stabilization, JDK 14+ bug fixes

As of December 25, 2023, Apache Ignite 2.16 has been released. You can directly check the full list of resolved Important JIRA tasks but let's briefly overview some valuable improvements.

Cache dumps

Ignite has persistent cache snapshots and this feature is highly appreciated by Ignite users. This release introduces another way to make a copy of user data - a cache dump.

The cache dump is essentially a file that contains all entries of a cache group at the time of dump creation. Dump is consistent like a snapshot, which means all entries that existed in the cluster at the moment of dump creation will be included in the dump file. Meta information of dumped caches and binary meta are also included in the dump.

Main differences from cache snapshots:

Supports in-memory caches that a snapshot feature does not support.
Takes up less disk space. The dump contains only the cache entries as-is.
Can be used for offline data processing.

The IgniteSnapshot interface provides a method to create a dump:

ignite.snapshot().createDump("dump-name", Arrays.asList("cacheGrp1", "cacheGrp2")).get();

You can read a dump using the Dump Reader application:


                  // Consumer should be implemented to receive all data stored in cache dump.
DumpConsumer cnsmr = new DumpConsumerImpl(..);

DumpReaderConfiguration cfg = new DumpReaderConfiguration(new File("path-to-dump"), cnsmr);

new DumpReader(cfg, log).run();

Calcite engine: added hints and diagnostic tools, became more stable

The Calcite engine is now fully equipped with metrics, events and the performance statistics tool. Moreover, query plans and other useful properties have been added to the performance statistics report.

The following hints have been introduced to help the optimizer make optimizations more rational or build an execution plan faster:

FORCE_INDEX / NO_INDEX - Forces or disables index scan.
ENFORCE_JOIN_ORDER - Forces join order as appears in a query. Fastens building of joins plan.

There are 10+ bugs and performance fixes in the release.

Java thin client: Service Awareness feature

In previous versions of Ignite, the thin client invoked the service grid on a random server node. A service may not be deployed on each node. If a node misses the invoked service, the invocation request is redirected to a proper node. This additional network hop adds overhead and can now be avoided.

With Service Awareness, the thin client knows where service instances are deployed and sends the request to the correct node.

Note that the service topology is updated asynchronously starting with the first service invocation. Thus, some invocation redirects are still possible.

The feature is activated if:

The Partition Awareness property is enabled in a client configuration (it is enabled by default).
Cluster and thin client versions are 2.16.0 and higher.

CDC: Realtime mode

Since version 2.12 Ignite supports the CDC functionality. It's implemented with a background process that processes archived WAL segments. The delay between the creation of an event and it's processing by this process contains the time WAL segments were archived. Some users are intolerant of this lag. For such users, Ignite now provides functionality that makes it possible to process events almost instantly after they are created. Please refer to the CdcManager interface javadocs for more details.

Other improvements and changes

Fixed JDK 14-21 support issues.
Cluster Management API has been implemented to unify management command invokers via various protocols - CLI, JMX, REST. The Ignite now provides JMX command invoker.
Operations on atomic caches within transactions are finally forbidden. The system property IGNITE_ALLOW_ATOMIC_OPS_IN_TX has been removed.
The community agreed to remove MVCC. The CacheAtomicityMode#TRANSACTIONAL_SNAPSHOT cache mode has been removed.
Mixed mode cache groups are now forbidden, but IGNITE_ALLOW_MIXED_CACHE_GROUPS system option may temporarily allow them.
The ignite-ml and cassandra modules have been migrated to the Ignite extensions.
100+ small improvements and bug fixes.

Anything else?

See the release notes to learn about all of the new features and improvements.

Reach out to us on the community user list for more questions, details, and feedback.

Sincerely yours, Ignite contributors and committers