Edit

Data Rebalancing

Overview

When a new node joins the cluster, some partitions are relocated to the new node so that the data remains distributed equally in the cluster. This process is called data rebalancing.

If an existing node permanently leaves the cluster and backups are not configured, you lose the partitions stored on this node. When backups are configured, one of the backup copies of the lost partitions becomes a primary partition and the rebalancing process is initiated.

Caution

Data rebalancing is triggered by changes in the Baseline Topology. In pure in-memory clusters, the default behavior is to start rebalancing immediately when a node leaves or joins the cluster (the baseline topology changes automatically). In clusters with persistence, the baseline topology has to be changed manually (default behavior), or can be changed automatically when automatic baseline adjustment is enabled.

Configuring Rebalancing Mode

Ignite supports both synchronous and asynchronous rebalancing. In the synchronous mode, any operation on the cache data is blocked until rebalancing is finished. In the asynchronous mode, the rebalancing process is done asynchronously. You can also disable rebalancing for a particular cache.

To change the rebalancing mode, set one of the following values in the cache configuration.

SYNC — Synchronous rebalancing mode. In this mode, any call to the cache public API is blocked until rebalancing is finished.
ASYNC — Asynchronous rebalancing mode. Distributed caches are available immediately and load all necessary data from other available cluster nodes in the background.
NONE — In this mode no rebalancing takes place, which means that caches are either loaded on demand from the persistent storage whenever data is accessed, or populated explicitly.

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:util="http://www.springframework.org/schema/util" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="         http://www.springframework.org/schema/beans         http://www.springframework.org/schema/beans/spring-beans.xsd         http://www.springframework.org/schema/util         http://www.springframework.org/schema/util/spring-util.xsd">
    <bean class="org.apache.ignite.configuration.IgniteConfiguration" id="ignite.cfg">
        <property name="cacheConfiguration">
            <list>
                <bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="mycache"/>
                    <!-- enable synchronous rebalance mode -->
                    <property name="rebalanceMode" value="SYNC"/>
                </bean>
            </list>
        </property>

    </bean>
</beans>

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.ignite.snippets;

import org.apache.ignite.Ignite;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheRebalanceMode;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;

public class RebalancingConfiguration {

    public static void main(String[] args) {
        RebalancingConfiguration rc = new RebalancingConfiguration();

        rc.configure();
    }

    void configure() {
        IgniteConfiguration cfg = new IgniteConfiguration();

        CacheConfiguration cacheCfg = new CacheConfiguration("mycache");

        cacheCfg.setRebalanceMode(CacheRebalanceMode.SYNC);

        // Start a node.
        Ignite ignite = Ignition.start(cfg);


        ignite.close();
    }

}

IgniteConfiguration cfg = new IgniteConfiguration
{
    CacheConfiguration = new[]
    {
        new CacheConfiguration
        {
            Name = "mycache",
            RebalanceMode = CacheRebalanceMode.Sync
        }
    }
};

// Start a node.
var ignite = Ignition.Start(cfg);

This API is not presently available for C++. You can use XML configuration.

Configuring Rebalance Thread Pool

By default, rebalancing is performed in one thread on each node. It means that at each point in time only one thread is used to transfer batches from one node to another, or to process batches coming from the remote node.

You can increase the number of threads that are taken from the system thread pool and used for rebalancing. A system thread is taken from the pool every time a node needs to send a batch of data to a remote node or needs to process a batch that came from a remote node. The thread is relinquished after the batch is processed.

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:util="http://www.springframework.org/schema/util" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="         http://www.springframework.org/schema/beans         http://www.springframework.org/schema/beans/spring-beans.xsd         http://www.springframework.org/schema/util         http://www.springframework.org/schema/util/spring-util.xsd">
    <bean class="org.apache.ignite.configuration.IgniteConfiguration" id="ignite.cfg">
        <property name="rebalanceThreadPoolSize" value="4"/>

    </bean>
</beans>

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.ignite.snippets;

import org.apache.ignite.Ignite;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheRebalanceMode;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;

public class RebalancingConfiguration {

    public static void main(String[] args) {
        RebalancingConfiguration rc = new RebalancingConfiguration();

        rc.configure();
    }

    void configure() {
        IgniteConfiguration cfg = new IgniteConfiguration();

        cfg.setRebalanceThreadPoolSize(4);

        // Start a node.
        Ignite ignite = Ignition.start(cfg);


        ignite.close();
    }

}

This API is not presently available for C#/.NET. You can use XML configuration.

This API is not presently available for C++. You can use XML configuration.

Caution

System thread pool is widely used internally by all the cache related operations (put, get, etc.), SQL engine, and other modules. Setting the size of the rebalancing thread pool to a large value may significantly increase rebalancing performance at the cost of decreased throughput.

Rebalance Message Throttling

When data is transferred from one node to another, the whole data set is split into batches and each batch is sent in a separate message. You can configure the batch size and the amount of time the node waits between messages.

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:util="http://www.springframework.org/schema/util" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="         http://www.springframework.org/schema/beans         http://www.springframework.org/schema/beans/spring-beans.xsd         http://www.springframework.org/schema/util         http://www.springframework.org/schema/util/spring-util.xsd">
    <bean class="org.apache.ignite.configuration.IgniteConfiguration" id="ignite.cfg">
        <!-- Set batch size. -->
        <property name="rebalanceBatchSize" value="#{2 * 1024 * 1024}"/>
        <!-- Set batches prefetch count. -->
        <property name="rebalanceBatchesPrefetchCount" value="3"/>
        <!-- Set throttle interval. -->
        <property name="rebalanceThrottle" value="100"/>

    </bean>
</beans>

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.ignite.snippets;

import org.apache.ignite.Ignite;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheRebalanceMode;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;

public class RebalancingConfiguration {

    public static void main(String[] args) {
        RebalancingConfiguration rc = new RebalancingConfiguration();

        rc.configure();
    }

    void configure() {
        IgniteConfiguration cfg = new IgniteConfiguration();


        cfg.setRebalanceBatchSize(2 * 1024 * 1024);
        cfg.setRebalanceBatchesPrefetchCount(3);
        cfg.setRebalanceThrottle(100);

        // Start a node.
        Ignite ignite = Ignition.start(cfg);


        ignite.close();
    }

}

IgniteConfiguration cfg = new IgniteConfiguration
{
    RebalanceBatchSize = 2 * 1024 * 1024,
    RebalanceThrottle = new TimeSpan(0, 0, 0, 0, 100),
    RebalanceBatchesPrefetchCount = 3
};

// Start a node.
var ignite = Ignition.Start(cfg);

This API is not presently available for C++. You can use XML configuration.

Other Properties

The following table lists the properties of IgniteConfiguration related to rebalancing:

Caution

rebalanceDelay and related API’s are deprecated and will be removed in the next releases.

Property Description Default Value

Property	Description	Default Value
`rebalanceThreadPoolSize`	Rebalance thread pool size. Limit of threads used for rebalance. See Configuring Rebalance Thread Pool	min(4, max(1, AVAILABLE_PROC_CNT / 4))
`rebalanceBatchSize`	The size in bytes of a single rebalance message. The rebalancing algorithm splits the data on every node into multiple batches prior to sending it to other nodes.	512KB
`rebalanceBatchesPrefetchCount`	Rebalance batches prefetch count.	3
`rebalanceThrottle`	See Rebalance Message Throttling.	0 (throttling disabled)
`rebalanceOrder`	The order in which rebalancing should be done. Rebalance order can be set to a non-zero value for caches with SYNC or ASYNC rebalance modes only. Rebalancing for caches with smaller rebalance order is completed first. By default, rebalancing is not ordered.	0
`rebalanceTimeout`	Timeout for pending rebalancing messages when they are exchanged between the nodes.	10 seconds
`rebalanceDelay`	[Deprecated] A delay in milliseconds before the rebalancing process starts after a node joins or leaves the topology. Rebalancing delay is useful if you plan to restart nodes or start multiple nodes at once or one after another and don’t want to repartition and rebalance the data until all nodes are started.	0 (no delay)

rebalanceThreadPoolSize

Rebalance thread pool size. Limit of threads used for rebalance. See Configuring Rebalance Thread Pool

min(4, max(1, AVAILABLE_PROC_CNT / 4))

rebalanceBatchSize

The size in bytes of a single rebalance message. The rebalancing algorithm splits the data on every node into multiple batches prior to sending it to other nodes.

512KB

rebalanceBatchesPrefetchCount

Rebalance batches prefetch count.

rebalanceThrottle

See Rebalance Message Throttling.

0 (throttling disabled)

rebalanceOrder

The order in which rebalancing should be done. Rebalance order can be set to a non-zero value for caches with SYNC or ASYNC rebalance modes only. Rebalancing for caches with smaller rebalance order is completed first. By default, rebalancing is not ordered.

rebalanceTimeout

Timeout for pending rebalancing messages when they are exchanged between the nodes.

10 seconds

rebalanceDelay

[Deprecated] A delay in milliseconds before the rebalancing process starts after a node joins or leaves the topology. Rebalancing delay is useful if you plan to restart nodes or start multiple nodes at once or one after another and don’t want to repartition and rebalance the data until all nodes are started.

0 (no delay)

Monitoring Rebalancing Process

You can monitor the rebalancing process for specific caches using JMX.

© 2024 The Apache Software Foundation.
Apache, Apache Ignite, the Apache feather and the Apache Ignite logo are either registered trademarks or trademarks of The Apache Software Foundation.