October 20th, Q&A session: Get you issues solved and questions answered!

GitHub logo
Edit

New Metrics System

Overview

Warning
Experimental

Ignite 2.8 introduced a new mechanism for collecting metrics, which is intended to replace the legacy metrics system. This section explains the new system and how you can use it to monitor your cluster.

Let’s explore the basic concepts of the new metrics system in Ignite. First, there are different metrics. Each metric has a name and a return value. The return value can be a simple value like String, long, or double, or can represent a Java object. Some metrics represent Histograms.

And then there are different ways to export the metrics — what we call exporters. To put it another way, the exporter are different ways you can access the metrics. Each exporter always gives access to all available metrics.

Ignite includes the following exporters:

  • JMX

  • SQL Views

  • Log files

  • OpenCensus

You can create a custom exporter by implementing the MetricExporterSpi interface.

Metric Registers

Metrics are grouped into categories (called registers). Each register has a name. The full name of a specific metric within the register consists of the register name followed by a dot, followed by the name of the metric: <register_name>.<metric_name>. For example, the register for data storage metrics is called io.datastorage. The metric that return the storage size is called io.datastorage.StorageSize.

The list of all registers and the metrics they contain are described here.

Metric Exporters

If you want to enable metrics, configure one or multiple metric exporters in the node configuration. This is a node-specific configuration, which means it enables metrics only on the node where it is specified.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="metricExporterSpi">
        <list>
            <bean class="org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi"/>
            <bean class="org.apache.ignite.spi.metric.sql.SqlViewMetricExporterSpi"/>
            <bean class="org.apache.ignite.spi.metric.log.LogExporterSpi"/>
            <bean class="org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi"/>
        </list>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

cfg.setMetricExporterSpi(new JmxMetricExporterSpi(), new SqlViewMetricExporterSpi());

Ignite ignite = Ignition.start(cfg);
This API is not presently available for C++. You can use XML configuration.

The following sections describe the exporters available in Ignite by default.

JMX

org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi exposes metrics via JMX beans.

IgniteConfiguration cfg = new IgniteConfiguration();

JmxMetricExporterSpi jmxExporter = new JmxMetricExporterSpi();

//export cache metrics only
jmxExporter.setExportFilter(mreg -> mreg.name().startsWith("cache."));

cfg.setMetricExporterSpi(jmxExporter);
This API is not presently available for C++.

SQL View

org.apache.ignite.spi.metric.sql.SqlViewMetricExporterSpi exposes metrics via the SYS.METRICS view. Each metric is displayed as a single record. You can use any supported SQL tool to view the metrics:

> select name, value from SYS.METRICS where name LIKE 'cache.myCache.%';
+-----------------------------------+--------------------------------+
|                NAME               |             VALUE              |
+-----------------------------------+--------------------------------+
| cache.myCache.CacheTxRollbacks    | 0                              |
| cache.myCache.OffHeapRemovals     | 0                              |
| cache.myCache.QueryCompleted      | 0                              |
| cache.myCache.QueryFailed         | 0                              |
| cache.myCache.EstimatedRebalancingKeys | 0                         |
| cache.myCache.CacheEvictions      | 0                              |
| cache.myCache.CommitTime          | [J@2eb66498                    |
....

This is how you can configure the SQL View exporter:

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="metricExporterSpi">
        <list>
            <bean class="org.apache.ignite.spi.metric.sql.SqlViewMetricExporterSpi"/>
        </list>
    </property>
</bean>
IgniteConfiguration cfg = new IgniteConfiguration();

SqlViewMetricExporterSpi jmxExporter = new SqlViewMetricExporterSpi();

//export cache metrics only
jmxExporter.setExportFilter(mreg -> mreg.name().startsWith("cache."));

cfg.setMetricExporterSpi(jmxExporter);
This API is not presently available for C++. You can use XML configuration.

Log

org.apache.ignite.spi.metric.log.LogExporterSpi prints the metrics to the log file at regular intervals (1 min by default) at INFO level.

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="metricExporterSpi">
        <list>
            <bean class="org.apache.ignite.spi.metric.log.LogExporterSpi"/>
        </list>
    </property>
</bean>

If you use programmatic configuration, you can change the print frequency as follows:

IgniteConfiguration cfg = new IgniteConfiguration();

LogExporterSpi logExporter = new LogExporterSpi();
logExporter.setPeriod(600_000);

//export cache metrics only
logExporter.setExportFilter(mreg -> mreg.name().startsWith("cache."));

cfg.setMetricExporterSpi(logExporter);

Ignite ignite = Ignition.start(cfg);

OpenCensus

org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi adds integration with the OpenCensus library.

To use the OpenCensus exporter:

  1. Enable the 'ignite-opencensus' module.

  2. Add org.apache.ignite.spi.metric.opencensus.OpenCensusMetricExporterSpi to the list of exporters in the node configuration.

  3. Configure OpenCensus StatsCollector to export to a specific system. See OpenCensusMetricsExporterExample.java for an example and OpenCensus documentation for additional information.

Configuration parameters:

  • filter - predicate that filters metrics.

  • period - export period.

  • sendInstanceName - if enabled, a tag with the Ignite instance name is added to each metric.

  • sendNodeId - if enabled, a tag with the Ignite node id is added to each metric.

  • sendConsistentId - if enabled, a tag with the Ignite node consistent id is added to each metric.

Histograms

The metrics that represent histograms are available in the JMX exporter only. Histogram metrics are exported as a set of values where each value corresponds to a specific bucket and is available through a separate JMX bean attribute. The attribute names of a histogram metric have the following format:

{metric_name}_{low_bound}_{high_bound}

where

  • {metric_name} - the name of the metric.

  • {low_bound} - start of the bound. 0 for the first bound.

  • {high_bound} - end of the bound. inf for the last bound.

Example of the metric names if the bounds are [10,100]:

  • histogram_0_10 - less than 10.

  • histogram_10_100 - between 10 and 100.

  • histogram_100_inf - more than 100.