Dynamically building database queries can be necessary for some use cases, such as UI-defined filtering. This can get challenging with LINQ frameworks like EF Core and Ignite.NET.
This is a breaking change release: The legacy service grid implementation was removed.
New Apache Calcite-based SQL engine
We've implemented a new experimental SQL engine based on Apache Calcite. Now it's possible to:
The current H2-based engine has fundamental limitations. For example:
- some queries should be splitted into 2 phases (map subquery and reduce subquery), but some of them cannot be effectively executed in 2 phases.
- H2 is a third-party database product with not-ASF license.
- The optimizer and other internal things are not supposed to work in a distributed environment.
- It's hard to make Ignite-specific changes to the H2 code, patches are often declined.
The Apache Ignite versions lower than 2.11.1 are vulnerable to CVE-2021-44832 which is related to the
ignite-log4j2 module usage.
The release also fixes 10+ CVE’s of various modules. See release notes for more details.
Change Data Capture
Change Data Capture (CDC) is a data processing pattern used to asynchronously receive entries that have been changed on the local node so that action can be taken using the changed entry.
Apache Ignite with Log4j Vulnerability
All the following conditions must be met:
- The Apache Ignite version lower than 2.11.0 is used (since these vulnerabilities are already fixed in 2.11.1, 2.12, and upper versions);
ignite-logj42is used by Apache Ignite and located in the
libsdirectory (by default it is located in the
libs/optionaldirectory, so these deployments are not affected);
The Java version in use is older than the following versions:
11.0.1. This is due to the fact that later versions set the JVM property
falseby default, which disables JNDI loading of classes from arbitrary URL code bases.
NOTE: Relying only on the Java version as a protection against these vulnerabilities is very risky and has not been tested.
The new Apache Ignite 2.11 was released on September 17, 2021. It can be considered to be a greater extent as a stabilization release that closed a number of technical debts of the internal architecture and bugs. Out of more than 200 completed tasks, 120 are bug fixes. However, some valuable improvements still exist, so let's take a quick look at them together.
Partition awareness is enabled by default in the 2.11 release and allows thin clients to send query requests directly to the node that owns the queried data. Without partition awareness, an application executes all queries and operations via a single server node that acts as a proxy for the incoming requests.
When Apache Ignite entered the Apache Software Foundation (ASF) Incubator in 2014, it took less than a year for the project and its community to graduate from the Incubator and become a top-level project for the ASF. Since then, Ignite has experienced a significant and steady growth in popularity, and it has been used by thousands of application developers and architects to create high-performance and scalable applications used by millions of people daily. In this article, we’ll recap the achievements of Ignite in 2020-2021.
Ignite is Ranked as a Top 5 Project
The ASF has ranked Apache ignite as a Top 5 project in various categories since 2017. That year, Ignite was in the Top 5 of Apache Project Repositories by Commits and most active Apache mailing lists. Today, the momentum continues, and Ignite continues to be ranked as a Top 5 project in multiple categories: second on the Top 5 big data user lists, third on the Top 5 big data dev lists, second on the Top 5 of all user lists, third on the Top 5 repos by size.
With thousands of changes contributed to Apache Ignite 2.8 that enhanced almost all the components of the platform, it’s possible to overlook some of the improvements that can convince you to upgrade to this version sooner than later. While a quick check of the release notes will help to discover anticipated bug fixes, this article aims to guide through enhancements every Ignite developer should be aware of.
New Subsystem for Production Monitoring and Tracing
Several months of constant work on IEP-35: Monitoring & Profiling has resulted in the creation of a robust and elastic subsystem for production monitoring and diagnostic (aka. profiling). This was influenced by the needs of many developers who deployed Ignite in critical environments and were asking for a foundation that can be integrated with many external monitoring tools and be expanded easily.
The new subsystem consists of several registries that group individual metrics related to a specific Ignite component. For instance, you will find registries for cache, compute, or service grid APIs. Since the registries are designed to be generic, specific exporters can observe the state of Ignite via a myriad of tools supporting various protocols. By default, Ignite 2.8 introduces exporters for monitoring interfaces such as log files, JMX and SQL views, and contemporary ones such as OpenCensus.
Apache Ignite was always appreciated by its users for two primary things it delivers - scalability and performance. Throughout the lifetime many distributed systems tend to do performance optimizations from a release to release while making scalability related improvements just a couple of times. It's not because the scalability is of no interest. Usually, scalability requirements are set and solved once by a distributed system and don't require significant additional interventions by engineers.
However, Apache Ignite grew to the point when the community decided to revisit its discovery subsystem that influences how well and far Ignite scales out. The goal was pretty clear - Ignite has to scale to 1000s of nodes as good as it scales to 100s now.
It took many months to get the task implemented. So, please join me in welcoming Apache Ignite 2.5 that now can be scaled easily to 1000s of nodes and goes with other exciting capabilities. Let's check out the most prominent ones.
Usually, Ignite community rolls out a new version once in 3 months, but we had to make an exception for Apache Ignite 2.4 that consumed five months in total. We could easily blame Thanksgiving, Christmas and New Year holidays for the delay and would be forgiven, but, in fact, we were forging the release you can't simply pass by.
Let's dive in and search for a big fish.
Machine Learning General Availability
Eight months ago, at the time of Apache Ignite 2.0, we put out the first APIs that formed the foundation of the Ignite's machine learning component of today. Since that time, Ignite machine learning experts and enthusiasts have been moving the liary to the general availability condition meticulously. And Ignite 2.4 became a milestone that let us consider the ML Grid to be production ready.
Putting aside the regular bug fixes and performance optimizations, the Apache Ignite 2.3 release brings new SQL capabilities and Ignite persistence improvements that are worth mentioning.
Let's start with SQL first.
Apache Ignite users have consistently told us that despite all of Ignite’s SQL capabilities, it’s been at times challenging trying to figure out how to start using Ignite as an SQL database.
This was mostly caused by scattered documentation pages, lack of “getting started” guides and tutorials. We’ve remedied this oversight! All related SQL knowledge has been curated in a single documentation domain.