Edit

Benchmarking Ignite With Yardstick Framework

Yardstick Ignite Benchmarks

Apache Ignite benchmarks are written on top of the Yardstick Framework, allowing you to measure the performance of various Apache Ignite components and modules. The documentation below describes how to execute and configure pre-assembled benchmarks. If you need to add new benchmarks or build existing one, then refer to the instructions from Ignite’s DEVNOTES.txt file in the source directory.

Visit the Yardstick Repository for more details on the resulting graphs generation and how the framework works.

Running Ignite Benchmarks Locally

The simplest way to start with benchmarking is to use one of the executable scripts available under the benchmarks/bin directory:

./bin/benchmark-run-all.sh config/benchmark-sample.properties

The command above will benchmark the cache put operations for a distributed atomic cache. The results of the benchmark will be added to an auto-generated output/results-{DATE-TIME}+\ directory.

If the ./bin/benchmark-run-all.sh command is executed as-is without any parameters and modifications in the configuration file, then all the available benchmarks will be executed on a local machine using the config/benchmark.properties configuration. In case of any issues, refer to the logs that are added to an auto-generated output/logs-{DATE-TIME} directory.

For more information about available benchmarks and configuration parameters, refer to the Existing Benchmarks and Properties And Command Line Arguments sections below.

Running Ignite Benchmarks Remotely

To benchmark Apache Ignite across several remote hosts:

Go to config/ignite-remote-config.xml and replace <value>127.0.0.1:47500..47509</value> with actual IPs of all the remote hosts. Refer to the documentation section below if you prefer to use another kind of IP finder: Cluster Configuration
Go to config/benchmark-remote-sample.properties and replace localhost with actual IPs of the remote hosts in the following places: SERVERS=localhost,localhost DRIVERS=localhost,localhost where the DRIVER is a host (usually an Ignite client node) that executes benchmarking logic. SERVERS are Ignite nodes that are benchmarked. Replace the localhost occurrences in the same places in the config/benchmark-remote.properties file if you plan to execute a full set of benchmarks available.
Upload Ignite Yardstick Benchmarks to one of your DRIVERS host in its own working directory.
Log in on the remote host that will be the DRIVER, and execute the following command:
./bin/benchmark-run-all.sh config/benchmark-remote-sample.properties

By default, all the necessary files will be automatically uploaded from the host in which you run the command above to every other host to the same path. If you prefer to do it manually set the AUTO_COPY variable in property file to false.

The command above will benchmark the cache put operation for a distributed atomic cache. The results of the benchmark will be added to an auto-generated output/results-{DATE-TIME} directory.

If you want to execute all the available benchmarks across the remote hosts, then execute the following command on the DRIVER side:

./bin/benchmark-run-all.sh config/benchmark-remote.properties

Existing Benchmarks

The following benchmarks are provided by default:

GetBenchmark - benchmarks atomic distributed cache get operation.
PutBenchmark - benchmarks atomic distributed cache put operation.
PutGetBenchmark - benchmarks atomic distributed cache put and get operations together.
PutTxBenchmark - benchmarks transactional distributed cache put operation.
PutGetTxBenchmark - benchmarks transactional distributed cache put and get operations together.
SqlQueryBenchmark - benchmarks distributed SQL query over cached data.
SqlQueryJoinBenchmark - benchmarks distributed SQL query with a Join over cached data.
SqlQueryPutBenchmark - benchmarks distributed SQL query with simultaneous cache updates.
AffinityCallBenchmark - benchmarks affinity call operation.
ApplyBenchmark - benchmarks apply operation.
BroadcastBenchmark - benchmarks broadcast operations.
ExecuteBenchmark - benchmarks execute operations.
RunBenchmark - benchmarks running task operations.
PutGetOffHeapBenchmark - benchmarks atomic distributed cache put and get operations together off-heap.
PutGetOffHeapValuesBenchmark - benchmarks atomic distributed cache put value operations off-heap.
PutOffHeapBenchmark - benchmarks atomic distributed cache put operations off-heap.
PutOffHeapValuesBenchmark - benchmarks atomic distributed cache get value operations off-heap.
PutTxOffHeapBenchmark - benchmarks transactional distributed cache put operation off-heap.
PutTxOffHeapValuesBenchmark - benchmarks transactional distributed cache put value operation off-heap.
SqlQueryOffHeapBenchmark -benchmarks distributed SQL query over cached data off-heap.
SqlQueryJoinOffHeapBenchmark - benchmarks distributed SQL query with a Join over cached data off-heap.
SqlQueryPutOffHeapBenchmark - benchmarks distributed SQL query with simultaneous cache updates off-heap.
PutAllBenchmark - benchmarks atomic distributed cache batch put operation.
PutAllTxBenchmark - benchmarks transactional distributed cache batch put operation.

Properties And Command Line Arguments

Note that this section only describes configuration parameters specific to Ignite benchmarks, and not for Yardstick framework. To run Ignite benchmarks and generate graphs, you will need to run them using the Yardstick framework scripts in the bin folder.

Refer to the Yardstick Documentation for common Yardstick properties and command line arguments for running Yardstick scripts.

The following Ignite benchmark properties can be defined in the benchmark configuration:

-b <num> or --backups <num> - Number of backups for every key.
-cfg <path> or --Config <path> - Path to Ignite configuration file.
-cs or --cacheStore - Enable or disable cache store readThrough, writeThrough.
-cl or --client - Client flag. Use this flag if you running more than one DRIVER, otherwise additional drivers would behave like a servers.
-nc or --nearCache - Near cache flag.
-nn <num> or --nodeNumber <num> - Number of nodes (automatically set in benchmark.properties); used to wait for the specified number of nodes to start.
-sm <mode> or -syncMode <mode> - Synchronization mode (defined in CacheWriteSynchronizationMode`).
-r <num> or --range - Range of keys that are randomly generated for cache operations.
-rd or --restartdelay - Restart delay in seconds.
-rs or --restartsleep - Restart sleep in seconds.
-rth <host> or --restHost <host> - REST TCP host.
-rtp <num> or --restPort <num> - REST TCP port, indicates that a Ignite node is ready to process Ignite Clients.
-ss or --syncSend - Flag indicating whether synchronous send is used in TcpCommunicationSpi.
-txc <value> or --txConcurrency <value> - Cache transaction concurrency control, either OPTIMISTIC or PESSIMISTIC (defined in CacheTxConcurrency).
-txi <value> or --txIsolation <value> - Cache transaction isolation (defined in CacheTxIsolation).
-wb or --writeBehind - Enable or disable writeBehind for cache store.

For example, if you want to run 2 IgniteNode servers on localhost with the PutBenchmark benchmark, number of backups set to 1, and synchronization mode set to PRIMARY_SYNC, then the following configuration should be specified in the benchmark.properties file:

SERVER_HOSTS=localhost,localhost
...

# Note that -dn and -sn, which stand for data node and server node,
# are native Yardstick parameters and are documented in
# Yardstick framework.
CONFIGS="-b 1 -sm PRIMARY_SYNC -dn PutBenchmark`IgniteNode"

Building From Sources

Run ./mvnw clean package -Pyardstick -pl modules/yardstick -am -DskipTests in the Apache Ignite root directory.

This command will compile the project and also unpack the scripts from yardstick-resources.zip file to modules/yardstick/target/assembly/bin directory.

Artifacts can be found in the modules/yardstick/target/assembly directory.

Custom Ignite Benchmarks

All benchmarks extend the AbstractBenchmark class. A new benchmark should also extend this abstract class and implement the test method (this is the method that actually tests performance).

© 2024 The Apache Software Foundation.
Apache, Apache Ignite, the Apache feather and the Apache Ignite logo are either registered trademarks or trademarks of The Apache Software Foundation.