Drive failure... now what?


Running: aerospike-server-community-3.12.0-1.el6.x86_64

I have a three node cluster and one node had a drive failure (SSD) - total data loss, but not a big deal since it was just a cache. Now I need to throw another drive in there and get that asd instance back in the cluster. Please keep in mind that I am not an experienced aerospike user/admin - know just enough to get what I need working. My config is below - I am using multicast for heartbeat - based on what I have read, all I need to do is prep the new SSD, replace the failed drive, fire asd back up and presto - everything should be cool.

It can’t be that easy. What am I missing? Any advice on checks to run after getting things back up and running?

# Aerospike database configuration file.

service {
	user aerospike
	group aerospike
	paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
	pidfile /var/run/aerospike/
        service-threads 8
        transaction-queues 8
        transaction-threads-per-queue 8	
        proto-fd-max 15000

logging {
	# Log file must be an absolute path.
	file /var/log/aerospike/aerospike.log {
		context any info

network {
	service {
		address any
		port 3000

	heartbeat {
		mode multicast
		port 9918

		# To use unicast-mesh heartbeats, remove the 3 lines above, and see
		# aerospike_mesh.conf for alternative.

		interval 150
		timeout 10

	fabric {
		port 3001

	info {
		port 3003

namespace tmcache {
	replication-factor 1
	memory-size 24G # around 250 million records
	default-ttl 30d
        high-water-disk-pct 80
        high-water-memory-pct 85
        storage-engine device {
		device /dev/sdc
		scheduler-mode noop
                ## WARNING: you can raise, but cannot lower without zeroing disk
		write-block-size 256K


Should be that simple. Make sure you write 0’s across the drive to ensure you’re not introducing junk data into the cluster though. ‘dd if=/dev/zero of=/dev/sdc bs=1M’ or ‘blkdiscard /dev/sdc’. Just make sure its the same disk speed/size so no surprises :slight_smile: