In-Memory Data Grid

Ignite data grid is a distributed, transactional key-value store. Unlike other in-memory data grids (IMDG), Ignite enables storing data both, in memory and on disk. On top of the standard key-value data operations, including ACID transactions and all the JCache (JSR 107) functionality, Ignite also supports SQL queries over its data with the support for distributed joins.

The data grid has been built from the ground up to linearly scale to hundreds of nodes with strong semantics for data locality and affinity data routing to reduce redundant data noise. It can be viewed as a distributed partitioned hash map with every cluster node owning a portion of the overall data. This way the more cluster nodes we add, the more data we can cache.

Ignite data grid is one of the fastest implementations of ACID transactions or atomic data updates in distributed clusters today. We know it because we constantly benchmark it ourselves.

The data in the data grid can be stored either purely in memory, or can also be persisted to disk. If Ignite native persistence is enabled, then data and indexes will be persisted natively by Ignite on every cluster node. In this case, memory only serves as a smaller caching layer of the overall persisted data set. All queries and transactions span the whole data set stored on disk.

The in-memory data grid can also improve performance and scalability by integrating with existing 3rd party databases, like RDBMS, NoSQL, or Hadoop-based storages. This approach does not require rip-and-replace of the existing data, but has its limitations. For example, SQL or scan queries will only include the results stored in memory, and not in the external database, since Ignite does not have any knowledge of the external data.

Read more about different types of persistence in Ignite.

Code Examples:
                            Ignite ignite = Ignition.ignite();

                            // Get an instance of named cache.
                            final IgniteCache<Integer, String> cache = ignite.cache("cacheName");

                            // Store keys in cache.
                            for (int i = 0; i < 10; i++)
                                cache.put(i, Integer.toString(i));

                            // Retrieve values from cache.
                            for (int i = 0; i < 10; i++)
                                System.out.println("Got [key=" + i + ", val=" + cache.get(i) + ']');

                            // Remove objects from cache.
                            for (int i = 0; i < 10; i++)
                                cache.remove(i);

                            // Atomic put-if-absent.
                            cache.putIfAbsent(1, "1");

                            // Atomic replace.
                            cache.replace(1, "1", "2");
                        
                            Ignite ignite = Ignition.ignite();

                            // Clone every object we get from cache, so we can freely update it.
                            IgniteCache<Integer, Account> cache = ignite.cache("cacheName");

                            try (IgniteTx tx = Ignition.ignite().transactions().txStart()) {
                                Account acct = cache.get(acctId);

                                assert acct != null;

                                // Deposit $20 into account.
                                acct.setBalance(acct.getBalance() + 20);

                                // Store updated account in cache.
                                cache.put(acctId, acct);

                                tx.commit();
                            }
                        
                            Ignite ignite = Ignition.ignite();

                            // Get an instance of named cache.
                            final GridCache<String, Integer> cache = ignite.cache("cacheName");

                            // Lock cache key "Hello".
                            Lock lock = cache.lock("Hello");

                            lock.lock();

                            try {
                                cache.put("Hello", 11);
                                cache.put("World", 22);
                            }
                            finally {
                                lock.unlock();
                            }
                        
                            IgniteCache<Long, Person> cache = ignite.cache("mycache");

                            SqlFieldsQuery sql = new SqlFieldsQuery(
                              "select concat(firstName, ' ', lastName) from Person");

                            // Select concatinated first and last name for all persons.
                            try (QueryCursor<List<?>> cursor = cache.query(sql)) {
                              for (List<?> row : cursor)
                                System.out.println("Full name: " + row.get(0));
                            }
                        
                            IgniteCache<Long, Person> personCache = ignite.cache("personCache");

                            // Select with join between Person and Organization to
                            // get the names of all the employees of a specific organization.
                            SqlFieldsQuery sql = new SqlFieldsQuery(
                                "select p.name  "
                                    + "from Person p, \"orgCache\".Organization o where "
                                    + "p.orgId = o.id "
                                    + "and o.name = ?");

                            // Execute the query and obtain the query result cursor.
                            try (QueryCursor<List<?>> cursor =  personCache.query(sql.setArgs("Ignite"))) {
                                for (List<?> row : cursor)
                                    System.out.println("Person name=" + row);
                            }
                        
                            IgniteCache<Long, Person> personCache = ignite.cache("personCache");

                            // Select average age of people working within different departments.
                            SqlFieldsQuery sql = new SqlFieldsQuery(
                                "select avg(p.age) as avg_age, d.name as dpmt_name, o.name as org_name "
                                    + "from Person p, \"depCache\".Department d, \"orgCache\".Organization o "
                                    + "where p.depid = d.id and d.orgid = o.id "
                                    + "group by d.name, o.name "
                                    + "order by avg_age";

                            // Execute the query and obtain the query result cursor.
                            try (QueryCursor<List<?>> cursor =  personCache.query(sql.setArgs("Ignite"))) {
                                for (List<?> row : cursor)
                                    System.out.println("Average age by department and organization: " + row);
                            }
                        
GitHub Examples:

Also see data grid examples available on GitHub.

Data Grid Features

Feature Description
Key-Value Store

Ignite data grid is a key-value store which can store data both, in-memory and on-disk. It can be viewed as a distributed partitioned hash map, with every cluster node owning a portion of the overall data. This way the more cluster nodes we add, the more data we can store.

Unlike other key-value stores, Ignite determines data locality using a pluggable hashing algorithm. Every client can determine which node a key belongs to by plugging it into a hashing function, without a need for any special mapping servers or name nodes.

SQL Support

In addition to key-value API, Ignite data grid has complete SQL support, including primary and secondary indexes and distributed JOINs.

Durable Memory

Ignite Durable Memory allows storing and processing data and indexes both, in memory and on disk. The in-memory data, including indexes, is always stored and managed off-heap, completely removing any type of Garbage Collection overhead.

JCache (JSR 107)

Ignite is a 100% compliant implementation of JCache (JSR 107) specification. JCache provides a very simple to use, yet very powerful API for data caching.

Some of the JCache API features include:

  • Basic Cache Operations
  • ConcurrentMap APIs
  • Collocated Processing (EntryProcessor)
  • Events and Metrics
  • Pluggable Persistence

Partitioning & Replication

Depending on the configuration, Ignite can either partition or replicate data. Unlike REPLICATED mode, where data is fully replicated across all nodes in the cluster, in PARTITIONED mode Ignite will equally split the data across multiple cluster nodes.

Ignite also allows to configure multiple backup copies to guarantee data resiliency in case of node failures.

Collocated Processing

Ignite allows executing any native Java, C++, and .NET/C# code directly on the server-side, close to the data, in collocated fashion.

Self-Healing Cluster

Ignite cluster can self-heal, where clients automatically reconnect in case of failures, slow clients are automatically kicked out, and data from failed nodes is automatically propagated to other nodes in the grid.

Client-side Near Caches

Near cache is local client-side cache that stores the most recently and most frequently accessed data.

ACID Transactions

Ignite provides fully ACID compliant distributed transactions that ensure guaranteed consistency.

Ignite supports OPTIMISTIC and PESSIMISTIC concurrency modes as well as READ_COMMITTED, REPEATABLE_READ, and SERIALIZABLE isolation levels.

Ignite transactions utilize 2PC protocol with many one-phase-commit optimizations whenever applicable.

Deadlock-Free Transactions

Ignite supports deadlock-free, optimistic transactions, which do not acquire any locks, and free users from worrying about the lock order. Such transactions also provide much better performance.

Transactional Entry Processor

Ignite transactional entry processor allows executing collocated user logic on the server side within a transaction.

Cross-Partition Transactions

In Ignite, transactions can be performed on all partitions of a cache across the whole cluster.

Locks

Ignite allows developers to define explicit locks enforcing mutual exclusion on cached objects.

Continuous Queries

Continuous queries are useful for cases when you want to execute a query and then continue to get notified about the data changes that fall into your query filter.

Native Persistence

Ignite Native Persistence is a distributed ACID and SQL-compliant disk store that transparently integrates with Ignite's Durable Memory as an optional persistence layer, storing data and indexes on SSD, Flash, 3D XPoint, or any other types of storages.

Database Integration

Ignite can automatically integrate with external databases, like RDBMS, NoSQL, HDFS and others.

Write-Through

Write-Through mode allows updating the data in the database.

Read-Through

Read-Through mode allows reading the data from the database.

Write-Behind Caching

Ignite provides an option to asynchronously perform updates to the database via Write-Behind Caching.

Web Session Clustering

Ignite data grid is capable of caching web sessions of all Java Servlet containers that follow Java Servlet 3.0 Specification, including Apache Tomcat, Eclipse Jetty, Oracle WebLogic, and others.

Web sessions caching becomes useful when running a cluster of app servers to improve performance and scalability of the servlet container.

Hibernate L2 Caching

Ignite data grid can be used as Hibernate Second-Level Cache (or L2 cache), which can significantly speed-up the persistence layer of your application.

Spring Caching

Ignite provides Spring-annotation-based way to enable caching for Java methods so that the result of a method execution is stored in the Ignite cache. If later the same method is called with the same set of parameters, the result will be retrieved from the cache instead of actually executing the method.

Spring Data

Apache Ignite implements Spring Data CrudRepository interface that not only supports basic CRUD operations but also provides access to the Apache Ignite SQL capabilities via the unified Spring Data API.

XA/JTA

Ignite can be configured with a Java Transaction API (JTA) transaction manager lookup class.

OSGI Support

 
Binary Protocol

Apache Ignite stores data in caches as BinaryObjects that allows you to:

  • Read a serialized object's field without full object deserialization.
  • Dynamically change an object's structure.
  • Dynamically create an object.