Hazelcast is slow when putting data into a distributed map
I have two hazelcast nodes (16GB memory, 4 cores per node). When I try to put it into a distributed map, hazelcast is very slow (1904 puts/s) but if I shut down a node, the performance improves (30000 puts/s). Can anyone help me improve the performance of my multi-node? Thanks
Solution
Please check your configuration. You must have a synchronous backup because your put query completes when the data is copied to another node. This is the default configuration.
You can use asynchronous backup to improve performance. But this will hamper the consistency of the system.
More about consistency:
In the context of the CAP theorem, Hazelcast is an AP product. Therefore, the goal of Best-Effort Consistency is replication, and both sync and async backups are lazy replication models. As explained in the page; The difference between the two options is;
- Synchronous backup, caller blocks until the backup copy applies backup updates and acknowledgements are sent back to the caller
- Asynchronous backups work in a fire-and-forget fashion.
Below, take a look at Hazelcast from Section of the Reference Manual:
Hazelcast’s replication technique enables Hazelcast clusters to offer high throughput. However, due to temporary situations in the system, such as network interruption, backup replicas can miss some updates and diverge from the primary. Backup replicas can also hit long GC pauses or VM pauses, and fall behind the primary, which is a situation called as replication lag. If a Hazelcast partition primary replica member crashes while there is a replication lag between itself and the backups, strong consistency of the data can be lost.