Ignite 3
Edit

Performance Statistics

Overview

Ignite provides a built-in tool for cluster profiling.

You can collect performance statistics from the cluster and then build the performance report.

What Is Collected

Performance statistics includes the following records:

Category Records

Cache operations

Cache API and cache store operations: get, put, remove, invoke, lock, bulk operations, conflict operations, and cache store load, write, and delete operations

Cache metadata

Cache start records with cache IDs and cache names

Transactions

Transaction commit and rollback records with affected cache IDs, start time, and duration

Queries

SQL, SCAN, and INDEX query records with query text or cache/index details, query ID, start time, duration, and success flag

Query details

Query logical and physical reads, processed rows, and custom query properties

Tasks and jobs

Compute task and job records with task names, session IDs, execution time, queue time, affinity partition, and timeout flag

Persistence

Checkpoint timing records and page write throttling records

System views

System view schemas and rows captured when statistics collection starts (some views are skipped, see the list below)

The following system views are skipped because they are large or duplicate data that is already collected as performance records:

  • baseline.node.attributes

  • node.attributes

  • metrics

  • caches

  • sql.queries

  • nodes

  • cacheGroupPageLists

  • dataRegionPageLists

  • partitionStates

  • statisticsPartitionData

  • metastorage

  • distributed.metastorage

Collecting Statistics

JMX interface and Control Script are used to manage statistics collecting.

Each node collects performance statistics in binary files. These files are placed under the Ignite_work_directory/perf_stat/ directory. Regular performance events are written to node-{nodeId}.prf. If a file with the same name already exists, Ignite adds an index to the file name: node-{nodeId}-{index}.prf.

When statistics collection starts, each node also writes a snapshot of system views (System Views) to node-{nodeId}-system-views.prf. If a file with the same name already exists, Ignite adds an index: node-{nodeId}-system-views-{index}.prf. The rotate operation rotates only regular performance event files.

Warning

Sensitive Information

Performance statistics files can contain sensitive information, including query text, cache names, task names, node attributes, and values exposed by system views. Store and share these files according to your security policies.

Performance statistics files are used to build the report offline.

For regular performance events, nodes use off-heap cyclic buffer to temporarily store serialized statistics. The writer thread flushes buffer to the file when the flush size is reached. Some statistics are skipped if the buffer overflows due to a slow disk. See the properties section for customization.

Each statistics collection process creates new files on nodes. Each next file has the same name with the corresponding index. See the examples below:

  • node-faedc6c9-3542-4610-ae10-4ff7e0600000.prf

  • node-faedc6c9-3542-4610-ae10-4ff7e0600000-1.prf

  • node-faedc6c9-3542-4610-ae10-4ff7e0600000-2.prf

  • node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views.prf

  • node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views-1.prf

Building the Report

Ignite provides a tool to generate the report from performance statistics files. The tool is published in the ignite-extensions repository as performance-statistics-ext extension.

Follow these steps to build the performance report:

  1. Stop collecting statistics and place files from all nodes under an empty directory. For example:

    /path_to_files/
        ├── node-162c7147-fef8-4ea2-bd25-8653c41fc7fa.prf
        ├── node-162c7147-fef8-4ea2-bd25-8653c41fc7fa-system-views.prf
        ├── node-7b8a7c5c-f3b7-46c3-90da-e66103c00001.prf
        ├── node-7b8a7c5c-f3b7-46c3-90da-e66103c00001-system-views.prf
        ├── node-faedc6c9-3542-4610-ae10-4ff7e0600000.prf
        └── node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views.prf
  2. Run the script from the release package of the tool:

    performance-statistics-tool/build-report.sh path_to_files

The performance report is created in the new directory under the performance statistics files path: path_to_files/report_yyyy-MM-dd_HH-mm-ss/. Open report_yyyy-MM-dd_HH-mm-ss/index.html in the browser to see the report.

You can also print raw performance statistics records to the console or to a file by using print-statistics.sh from the performance-statistics-ext release package. For example, to print checkpoint timing and page write throttling records:

performance-statistics-tool/print-statistics.sh path_to_files --ops CHECKPOINT,PAGES_WRITE_THROTTLE

To write the output to a file, use the --out parameter:

performance-statistics-tool/print-statistics.sh path_to_files --ops CHECKPOINT,PAGES_WRITE_THROTTLE --out checkpoints.json

Management

The following section provides information on JMX, Control Script and system properties.

JMX

You can manage the performance statistics collection via the PerformanceStatisticsMBean interface:

Method Description

start()

Start collecting performance statistics in the cluster.

stop()

Stop collecting performance statistics in the cluster.

rotate()

Rotate collecting performance statistics in the cluster.

started()

True if performance statistics collection is started.

Control Script

You can manage the performance statistics collection via the Control Script.

control.sh --performance-statistics [start|stop|rotate|status]
control.bat --performance-statistics [start|stop|rotate|status]

Parameters:

Parameter Description

start

Start collecting performance statistics in the cluster.

stop

Stop collecting performance statistics in the cluster.

rotate

Rotate collecting performance statistics in the cluster.

status

Get status of collecting performance statistics in the cluster.

System properties

Property Type Default Value Description

IGNITE_PERF_STAT_FILE_MAX_SIZE

Long

32 Gb

Maximum performance statistics file size in bytes. Performance statistics collection is stopped when the file size is exceeded.

IGNITE_PERF_STAT_BUFFER_SIZE

Integer

32 Mb

Performance statistics off heap buffer size in bytes.

IGNITE_PERF_STAT_FLUSH_SIZE

Integer

8 Mb

Minimal performance statistics batch size to be flushed in bytes.

IGNITE_PERF_STAT_CACHED_STRINGS_THRESHOLD

Integer

10240

Maximum performance statistics cached strings threshold. String caching is stopped when the threshold is exceeded.