Accelerate Apache Spark Applications
With Apache Ignite

Horizontally scalable and shared in-memory layer

Ignite is designed to store data sets in memory across a cluster of nodes, reducing latency of Spark operations that usually need to pull date from disk-based systems.

Minimal data shuffling over the network

Ignite tries to minimize data shuffling over the network between its store and Spark applications by running certain Spark tasks, produced by RDDs or DataFrames APIs in-place on Ignite nodes.

Extra performance boost with native Ignite APIs

Use native Ignite APIs such as SQL from Spark applications directly and eliminate data shuffling completely between Spark and Ignite.

Apache Ignite provides an implementation of the Spark RDD, which allows any data and state to be shared in memory as RDDs across Spark jobs.

The Ignite RDD provides a shared, mutable view of the data stored in Ignite caches across different Spark jobs, workers, or applications.

The Ignite RDD is implemented as a view over a distributed Ignite table (aka. cache). It can be deployed with an Ignite node either within the Spark job executing process, on a Spark worker, or in a separate Ignite cluster.

This means that depending on the chosen deployment mode, the shared state may either exist only during the lifespan of a Spark application (embedded mode), or it may out-survive the Spark application (standalone mode).

The Apache Spark DataFrame API introduced the concept of a schema to describe the data, allowing Spark to manage the schema and organize the data into a tabular format.

To put it simply, a DataFrame is a distributed collection of data organized into named columns. Conceptually, it is the equivalent of a table in a relational database. It allows Spark to leverage the Catalyst query optimizer to produce much more efficient query execution plans in comparison to RDDs, which are collections of elements partitioned across the nodes of the cluster.

Ignite supports DataFrame APIs, allowing Spark to write to and read from Ignite through that interface.

Furthermore, Ignite analyzes execution plans produced by Spark's Catalyst engine and can execute parts of the plan on Ignite nodes directly, which will reduce data shuffling and consequently make your SparkSQL perform better.