Hi, we are using aerospike : Aerospike Enterprise Edition build 3.9.1-158-g1e8db6e currently. We want to enable rack awareness to our cluster. For that we are moving our cluster to version 3.13 first and then we are planning to change configuration to enable rack awareness dynamically. What can be the best way to do this and avoid any downtime, such that all migrations run smoothly.
Are you manually creating your config files/racks? Are you using RF>=2? Are you in AP or CP mode? How many racks do you plan to specify? I believe the process would just be to get everything upgraded and stable, then dynamically apply the rack-id and recluster. That should be it. If things get hairy, to backout I believe you just set rack-id to 0 across all nodes and recluster.
We are using RF=2 and trying to ensure high availability. We have a cluster of 10 nodes and memory utilisation of each node is around 70 percent. We have configured high water mark at 75% of memory. In case we use 2 partitions (racks) i.e, 5 nodes in each rack, then in case of a rack failure the memory of each node after partition would jump up to over 75%?? and hence eviction of records ??
What should be ideal rack size for a 10 node cluster with each node utilising 70% memory ?
Also, one more thing is that once we enable rack ids dynamically to nodes, what should be strategy as instantly allocating rack ids to all 10 nodes might start lots of migrations, and that might cause high latencies.
70% memory usage is pretty high. Taking a node out and you’ll be at 78% already. It sounds like you need to expand. Since half of your architecture is on 1 rack, and you are planning for a rack failure scenario- you may want to set paxos-single-replica-limit - https://www.aerospike.com/docs/reference/configuration/#paxos-single-replica-limit . If you set this to 5, this would switch the cluster to RF=1 if you lost a rack. So you wouldn’t be completely overloaded, but you’d be in a vulnerable state.
The smaller the racks, the less the effect… and if you have all nodes on different racks, then there’s really no point to using rack-aware. That’s up to you on how you want to balance it.
As far as the migrations go, you need to just throttle them. There are lots of knobs and ways to make this process take days if you wish to be that conservative; https://www.aerospike.com/docs/operations/manage/migration/
First and foremost though, you really need to fix your memory usage. I’d be scared of an OOM-killer event if I were you.
Hi, we tried rack awareness for a cluster of 4 nodes. For that we upgraded cluster from Aerospike Enterprise Edition build 3.9.1-158-g1e8db6e to aerospike community version 3.13. and then made a partition placement group with 4 partitions(racks) and each node placed was given a different rack id.
After this procedure what we observed is the latency spike in cluster. What can be reason ? Moving to community version from enterprise patch ?? Or using partition placement group ?? any other reason ?
Probably latency due to migrations. May want to throttle things back. We can only speculate without seeing histograms/screenshots/configs/etc.
Thanks for response.
Strange thing what i observed after the following steps :
-
Upgraded cluster to 13.3
-
Two nodes in partition number 1 while other two nodes in partition 2.
-
Switched to heartbeat protocol version to v3 and paxos-protocol version to v5
-
rolling restart of cluster after updating config.
-
asinfo -v “racks:” shows : ns=test:rack_0=BB9CE6A9F390006,BB98E5D013AD406,BB9DCA5DC559106,BB94E9A22170D06
-
Dynamicaly assigned rack id 1 to two nodes in partition number one and rack-id 2 to two nodes in partition number two.
-
Added these rack ids in config and restarted one node to trigger migrations.
-
Racks of all 4 nodes in cluster were changed.
asinfo -v “racks:” shows : ns=test:rack_1=BB9CE6A9F390006,BB98E5D013AD406:rack_2=BB9DCA5DC559106,BB94E9A22170D06
-
If i am using RF=2 and stopped two nodes on rack 1, still there is no data loss which suggests that cluster has become rack aware.
STRANGE THING IS WHEN I GIVE COMMAND
asadm -e "info" output is :
**NODE** **BUILD** **Rack Aware**
10.0.49.131:3000 C-3.13.0.10 **none**
10.0.49.134:3000 C-3.13.0.10 **none**
10.0.49.158:3000 C-3.13.0.10 **none**
10.0.49.203:3000 C-3.13.0.10 **none**
This shows Rackaware Mode as none for all nodes and logs also displays this line :
Jul 03 2019 06:42:01 GMT: INFO (socket): (socket.c:2567) Node port 3001, node ID bb9ce6a9f390006, rack-aware 10.0.49.203
Jul 03 2019 06:42:01 GMT: INFO (config): (cfg.c:3755) Rack Aware mode not enabled
Given that rack ids are different and there is no data loss after shutting down 2 nodes, asadm -e "info" still shows rack aware not enabled. Is cluster actually rack enabled or not? If not, then why are rack ids different and why there is no data loss after shutting down 2 nodes with same rack id ?
Each node in its own rack is equivalent to not using rack-aware.
Probably migrations, in EE with ‘Rapid Rebalance’ only the data that changed needed to be migrated, CE migrates all data.
You need to update your aerospike-tools package to get the latest adadm.
The rack-aware message coming from socket is related to the pre paxos-protocol switch version of rack aware. You can ignore this message post switch, the message was removed in later releases.