High-Performance Computing With Apache Ignite

High-performance computing (HPC) is the ability to process data and perform complex calculations at high speeds. Using Apache Ignite® as a high-performance compute cluster, you can turn a group of commodity machines or a cloud environment into a distributed supercomputer of interconnected Ignite nodes. Ignite enables speed and scale by processing records in memory and reducing network utilization with APIs for data and compute-intensive calculations. Those APIs implement the MapReduce paradigm and allow you to run arbitrary tasks across the cluster of nodes.

High-Performance Computing With Apache Ignite

Co-located Processing

Ignite uses the notion of co-located processing to guide HPC workloads implementations in distributed in-memory environments. Co-located processing increases the performance of your complex calculations by running them straight on the Ignite cluster nodes. These calculations are done only on local data sets available on the nodes, thus avoiding data shuffling over the network and resulting in orders of magnitude increase in performance.

To exploit the co-located processing in practice, first, you need to co-locate data by storing related records on the same cluster node. As an example of related or co-located data, consider your bank account and transactions posted to it. Once you set accountID as an affinity key for the Transactions table, you'll instruct Ignite to store all transactions for your accountId on the same node that keeps the record of your account in the Accounts table. Now let's say a payment processing system sends a compute task that verifies previous transactions of your account. Since the data is co-located, Ignite will execute this task directly on the node that stores your account record with all completed transactions and finish the verification locally on that machine instead of pulling all the transactions back to the application over the network. This method of executing a task on the node where the data resides provides exceptionally high performance.The effect is even more significant when the system needs to process millions of transactions per second, verifying billions of previously completed payments.

Compute APIs

Ignite provides compute APIs (also known as compute grid) for creating and scheduling custom tasks of arbitrary complexity. The APIs implement MapReduce paradigm and are presently available for Java, C#, and C++.