Add Namespace: Adding New Namespace

Knowledge Base General
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

Question:

I have a working configuration with 1 Namespace. I tried to add 2 more namespaces. It halted migrations and I received the following errors when I restarted the node:

Sep 16 2014 16:52:42 GMT: WARNING (drv_ssd): (drv_ssd.c::2518) read header: device /dev/sdb previous namespace Namespace1 now Namespace2, check config or erase device
Sep 16 2014 16:52:42 GMT: CRITICAL (drv_ssd): (drv_ssd.c:ssd_load_devices:3447) unable to read disk header /dev/sdb: No such file or directory

Here are my configuration files:

BEFORE

namespace Namespace1 {
    replication-factor 2
	memory-size 32G
	default-ttl 30d # 30 days, use 0 to never expire/evict.

    storage-engine device {
	    # shouldn't mount these devices
		device /dev/sdb
		device /dev/sdc

        # ssd optimizations
		scheduler-mode noop
		write-block-size 128K
	}
}

AFTER

namespace Namespace1 {

    replication-factor 2
	memory-size 20G
	default-ttl 30d
	high-water-memory-pct 80
	high-water-disk-pct 50

    storage-engine device {
	    # shouldn't mount these devices
		device /dev/sdb
		device /dev/sdc

        # ssd optimizations
		scheduler-mode noop
		write-block-size 128K
	}
}

namespace Namespace2 {
    replication-factor 2
	memory-size 28G
	default-ttl 90d # 30 days, use 0 to never expire/evict.
	high-water-memory-pct 60
	high-water-disk-pct 50
	stop-writes-pct 90
	storage-engine device {
	    # shouldn't mount these devices
		device /dev/sdb
		device /dev/sdc
		# ssd optimizations
		scheduler-mode noop
		write-block-size 128K
	}
}

namespace Namespace3 {
    replication-factor 2
	memory-size 3G
	default-ttl 0
	high-water-memory-pct 60
	high-water-disk-pct 50
	stop-writes-pct 90
	storage-engine device {
	    # shouldn't mount these devices
		device /dev/sdb
		device /dev/sdc
		# ssd optimizations
		scheduler-mode noop
		write-block-size 128K
	}
}

Answer:

From the configuration files, it looks like you are reusing the same exact SSD’s and partitions for all namespace (device /dev/sdb device /dev/sdc) This will not work as a partition should only belong to one namespace.

Recommendation:

  1. For adding and removing a namespace in server versions 3.13.0.1 running the updated clustering protocol and versions after it, you can add the namespace configuration and do a rolling restart of the cluster nodes.

Note: For server versions prior to 3.13.0.1, you will have to restart entire cluster. For production sanity, the entire cluster has to updated and restarted together. Else the machines with different namespace configurations will not be part of same cluster until namespace definition is same across all machines of the desired cluster.

  1. You could divide your disk into 3 partitions (IE : sdb1, sdb2, sdb3 and sdc1, sdc2, sdc3) or go with separate drives per namespace.

More info on SSD’s can be found at: http://www.aerospike.com/docs/operations/plan/ssd/

  1. Devices should be unique across namespaces. You can use partitions instead if you don’t have enough unique devices.

  2. Also, when you are re-purposing a disk from one namespace to another, you need to wipe out the disk (dd if=/dev/zero of=/dev/destination/disk) before using it with the new namespace.

  3. This is handled gracefully in Aerospike server version 4.0.0.4. The following WARNING will be logged if device/files are not wiped clean previously or if accidentally reused again in another namespace. This would prevent issues with stuck migration or errors in read/writes.

Jun 15 2018 01:11:49 GMT: WARNING (drv_ssd): (drv_ssd.c:2466) read header: device /dev/sdc4 previous namespace ns1 now ns2, check config or erase device
Jun 15 2018 01:11:49 GMT: FAILED ASSERTION (drv_ssd): (drv_ssd.c:3202) unable to read disk header /dev/sdc4

NOTE: The context for aerospike.conf is case sensitive. All entries in aerospike.conf are lower case.

Notes

Keywords

SAME DEVICE DIFFERENT NAMESPACE

Timestamp

06/14/2018