Durable Memory

Apache Ignite is based on the Durable Memory architecture that allows storing and processing data and indexes both in memory and on disk when the Ignite Native Persistence feature is enabled.

The durable memory architecture helps achieve the performance and scale of in-memory computing together with the disk durability and strong consistency in one system.

Ignite's Durable Memory operates in a way similar to the Virtual Memory of operating systems such as Linux. However, one significant difference between these two is that the Durable Memory, in addition to keeping the whole or partial data set in memory, always keeps the whole data set with indexes on disk (assuming that Ignite Native Persistence is enabled), while Virtual Memory uses the disk when it runs out of RAM, for swapping purposes only.

Ignite Native Persistence

Ignite Persistence is the most flexible, scalable, and convenient way of persisting data in Ignite. It is widely used in scenarios where applications need a distributed memory-centric database

Ignite native persistence is a distributed, ACID, and SQL-compliant disk store that transparently integrates with Ignite's durable memory. Ignite persistence is optional and can be turned on and off. When turned off Ignite becomes a pure in-memory store.

Following are the advantages and characteristics of Apache Ignite as a platform when Durable Memory and Ignite Native Persistence are used together:

In-Memory
  • Off-Heap memory
  • Removes noticeable GC pauses
  • Automatic Defragmentation
  • Predictable memory consumption
  • Boosts SQL performance
On Disk
  • Optional Persistence
  • Support of flash, SSD, Intel 3D Xpoint
  • Stores superset of data
  • Fully Transactional
    • Write-Ahead-Log (WAL)
  • Instantaneous Cluster Restarts

3rd Party Persistence

Ignite can be used as a caching layer (aka. data grid) above an existing 3rd party database - RDBMS, NoSQL, or HDFS. This mode is used to accelerate the underlying database that persists the data. Ignite stores data in memory, distributed across multiple nodes providing fast data access. It reduces the network overhead caused due to frequent data movement between an application and the database. However, there are some limitations in comparison to the native persistence. For instance, SQL queries will be executed only on the data that is in RAM, thus, requiring to preload all the data set from disk to memory beforehand.

Swap Space

If you do not want to use Ignite native persistence or 3rd party persistence, you can enable swapping, in which case, Ignite in-memory data will be moved to the swap space located on disk if you run out of RAM. When swap space is enabled, Ignites stores data in memory mapped files (MMF) whose content will be swapped to disk by the OS depending on the current RAM consumption. The swap space is mostly used to avoid out of memory errors (OOME) that might happen if RAM consumption goes beyond its capacity and you need more time to scale the cluster out to​ redistribute the data sets evenly.

More Information

Durable Memory

Native Persistence

3rd Party Persistence

Swap Space