Apache Ignite Blog

Deep Learning With TensorFlow

Even though it was natural to provide machine learning algorithms in Ignite out of the box, another direction was taken for deep learning capabilities. Primarily because machine learning approaches have already been adopted in businesses from big to small -- while deep learning is still being used for narrow and specific use cases.

Thus, Ignite 2.7 can boast about an official integration with TensorFlow deep learning framework that gives a way to use Ignite as a distributed storage for TensorFlow calculations. With Ignite, data scientists can store unlimited data sets across a cluster, gain performance improvements and rely on fault-tolerance of both products if an algorithm fails in the middle of an execution.

Apache Ignite was always appreciated by its users for two primary things it delivers - scalability and performance. Throughout the lifetime many distributed systems tend to do performance optimizations from a release to release while making scalability related improvements just a couple of times. It's not because the scalability is of no interest. Usually, scalability requirements are set and solved once by a distributed system and don't require significant additional interventions by engineers.

However, Apache Ignite grew to the point when the community decided to revisit its discovery subsystem that influences how well and far Ignite scales out. The goal was pretty clear - Ignite has to scale to 1000s of nodes as good as it scales to 100s now.

It took many months to get the task implemented. So, please join me in welcoming Apache Ignite 2.5 that now can be scaled easily to 1000s of nodes and goes with other exciting capabilities. Let's check out the most prominent ones.

Usually, Ignite community rolls out a new version once in 3 months, but we had to make an exception for Apache Ignite 2.4 that consumed five months in total. We could easily blame Thanksgiving, Christmas and New Year holidays for the delay and would be forgiven, but, in fact, we were forging the release you can't simply pass by.

Let's dive in and search for a big fish.

Machine Learning General Availability

Eight months ago, at the time of Apache Ignite 2.0, we put out the first APIs that formed the foundation of the Ignite's machine learning component of today. Since that time, Ignite machine learning experts and enthusiasts have been moving the liary to the general availability condition meticulously. And Ignite 2.4 became a milestone that let us consider the ML Grid to be production ready.

As promised in my initial blog post on this matter, Apache Ignite community applied security patches against the notorious Meltdown Spectre vulnerabilities and completed performance testing of general operations and workloads that are typical for Ignite deployments.

The security patches were applied only for CVE-2017-5754 (Meltdown) and CVE-2017-5753 (Spectre Variant 1) vulnerabilities. The patches for CVE-2017-5715 (Spectre Variant 2) for the hardware the community used for testing are not stable yet an can cause system reboot issues or another unpredictable behavior. 

The applied patches have shown that the performance implications are negligible - the performance drop is just in the 0 - 7% range as the figure shows:

The world was rocked after the recent disclosure of the Meltdown and Spectre vulnerabilities that literally affect almost all software ever developed. Both issues are related to the way all modern CPUs are designed and this is why they have opened unprecedented security breaches -- making the software, including Apache Ignite, vulnerable to hacker attacks.

The vulnerabilities are registered in the National Vulnerability Database under the following CVEs:

• CVE-2017-5753 — Spectre variant 1
• CVE-2017-5715 — Spectre variant 2
• CVE-2017-5754 — Meltdown

We finally made this happen! I’m happy to invite all of the software architects and engineers out there to a series of webinars that will introduce you to the fundamental capabilities of in-memory computing platforms such as Apache Ignite.

There will also be a mix of theory and practice. A lot of code examples are waiting to be shown so that you can apply the theory in practice right away.

The series consists of two parts.

Part 1: Tuesday, November 21, 2017, 11:00am PT / 2:00pm ET

To be covered:

• Cluster configuration and deployment.
• Distributed database internals (partitioning, replication).
• Data processing with key-value APIs.
• Affinity Collocation.
• Data processing with SQL.

Putting aside the regular bug fixes and performance optimizations, the Apache Ignite 2.3 release brings new SQL capabilities and Ignite persistence improvements that are worth mentioning.

SQL

Let's start with SQL first.

Apache Ignite users have consistently told us that despite all of Ignite’s SQL capabilities, it’s been at times challenging trying to figure out how to start using Ignite as an SQL database.

This was mostly caused by scattered documentation pages, lack of “getting started” guides and tutorials. We’ve remedied this oversight! All related SQL knowledge has been curated in a single documentation domain.

by Tom Diederich

This is our third community update – there’s a lot going on, so let's get started.

Apache Ignite experts have already spoken at two meetups this month, both in Silicon Valley, but there are several more scheduled this month around the world.

On Sept. 9 Apache Ignite PMC chair Denis Magda was the featured presenter at the Big Data and Cloud Meetup in Santa Clara, Calif. His talk, titled "Apache Spark and Apache Ignite: Where Fast Data Meets the IoT," was highly rated and we’re planning a hands-on workshop with meetup organizers for November.

by Tom Diederich

Igniters, here are some community highlights from the last couple week. If I missed anything, please share it here. Meetups! Did you know that Apache Ignite experts are available to speak at your meetup? And we also have spots open for YOU to speak at the following meetups that some of us co-organize:

• Apache Ignite London
• Bay Area In-Memory Computing Meetup
• NYC In-Memory Computing Meetup
• Moscow Apache Ignite Meetup

The power and beauty of in-memory computing projects are that they truly do what they state -- deliver outstanding performance improvements by moving data closer to the CPU, using RAM as a storage and spreading the data sets out across a cluster of machines relying on horizontal scalability.

However, there is an unspoken side of the story. No matter how fast a platform is, we do not want to lose the data and encounter cluster restarts or other outages. To guarantee this we need to somehow make data persistent on the disk.

Most in-memory computing projects address the persistence dilemma by giving a way to sync data back to a relational database (RDBMS). That sounds reasonable and undoubtedly works pretty well in practice, but if we dig deeper, you’ll likely encounter the following limitations: