Every client instance runs an extra cluster tend thread that periodically polls server nodes for cluster status. In a single-process/multi-threaded environment, only one client instance is required. This instance is shared across all threads. However, in a multi-process/single-thread environment, multiple client instances are required, which means that multiple cluster tend threads are simultaneously polling server nodes for cluster status. Shared memory resolves this inefficiency.
Multi-process/single-threaded environments are common for languages like Python. The Aerospike Python client is implemented using calls to the Aerospike C client, gaining access to the C client's shared memory capabilities.
When shared memory is enabled, cluster state (including nodes and data partition maps) is stored in a shared memory segment. Only one cluster tend owner process polls server nodes for cluster status and writes to this shared memory segment. All other client processes read cluster status from this shared memory segment, and do not poll the server nodes.
One shared memory segment stores state for a single cluster. Multiple clusters require multiple shared memory segments. Each shared memory segment is identified by the user configurable
Configure shared memory using these fields in
use shm— Set to true to use shared memory for cluster tending. Shared memory is useful when operating in single-thread mode with multiple client processes. This model is used by wrapper languages such as Python. When enabled, data partition maps are maintained by only one process and all other processes use these shared memory maps. Default: false
shm_key— Identifier for the shared memory segment associated with the target Aerospike cluster. Each shared memory segment contains state for one Aerospike cluster. If there are multiple Aerospike clusters, a different shm_key must be defined for each cluster. Default: 0xA7000000
shm_max_nodes— Set the maximum number of server nodes allowed using shared memory and the size of the fixed shared memory segment. Default: 16
Ensure that you leave a cushion between actual server node count and
shm_max_nodes so that you can add new nodes without rebooting the client.
shm_max_namespaces— Set the maximum number of namespaces, which sizes the fixed shared memory segment. Default: 8
Ensure that you leave a cushion between actual namespace count and
shm_max_namespaces so that you can add new namespaces without rebooting the client.
shm_takeover_threshold_sec— Take over shared memory cluster tending if the cluster is not tended by this threshold within n seconds. Default: 30
Enable Shared Memory
To enable shared memory before calling
as_config_add_host(&config, host, port);
config.use_shm = true;
config.shm_key = 0xA7000000;
config.shm_max_nodes = 16;
config.shm_max_namespaces = 8;
config.shm_takeover_threshold_sec = 30;
as_status status = aerospike_connect(&client, &err);
The shared memory segment is automatically created by the client if it does not already exist on that machine.
The first client process owns cluster tending by acquiring a shared memory lock.
Ensuing client processes read cluster status from the shared memory segment, but do not cluster tend nor write to shared memory.
Applications must call
aerospike_close()before exiting to release the shared memory lock. Another client process automatically acquires the lock and becomes the new cluster tender.
If the application exits without releasing the lock, another client process will take over cluster tending after the takeover threshold is reached (
shm_takeover_threshold_sec, default = 30 seconds). Cluster status can still be read during this lag, but cluster tending (polling servers for cluster status) does not commence until another client process takes over.
You can view shared memory segments with this command:
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0xA7000000 622592 root 666 263224 0
To change shared memory configuration variables (for example, to increase max nodes), a new shared memory segment must be created. There are two ways to upgrade your application:
- One Shot Upgrade
Close all applications.
Delete the shared memory segment (if it still exists):
ipcrm -M 0xA7000000
Restart your applications.
- Rolling Upgrade
- Set a different shared memory key in your new application:New application instances use the new shared memory segment, automatically created by the first new application instance. Old applications instances continue to use the old shared memory segment.
config.shm_key = 0xA7000001;
- Gradually stop all old application instances and replace them with new application instances.
- Delete the old shared memory segment (if it still exists):
ipcrm -M 0xA7000000