How to Add, Replace, & Remove disks


#1

How to Add, Replace, & Remove disks

Synopsys

This document discusses the add/replace/remove storage device feature added as of Aerospike 3.3.5. With this change, Aerospike is able to replace faulty disks and/or add additional storage devices with a fast-start or cold-start.

Purpose

Before these releases, removing a faulty disk or expanding capacity was a major ordeal. The user would have to zeroize all disks associated with the namespace regardless of which tasks were being performed. If there were multiple nodes needing changes then the user would need to wait for migrations to complete between each change. Since the feature was added, the user no longer needs to zeroize all disks, and only has to wait for migrations when replacing a disk.

Replacing a Storage Device

Hot-Swappable Devices

Inevitably storage devices fail, when this occurs Aerospike is able to recover quickly. If your nodes are equipped with hot-swappable drives, simply replace the faulty device and zeroize the new device and update Aerospike’s configuration to use the new device in place of the old device, then stop and fast-start the server. If multiple nodes require device replacements then perform the same procedure on the other nodes being sure to wait for migrations to complete before continuing to the next node.

WARNING: It is critical that the user maintains the same configuration order as before. If the second device failed then the replacement device must be the second device, preceded by the original preceding device and followed by the original devices that followed.

WARNING: For Aerospike versions 3.10.1 or newer ([AER-5273] - (KVS) Read duplicate resolution should only be done when repeatable read is turned on), many client_delete_not_found and client_read_not_found may occur when replacing disks. Setting the Namespace context read-consistency-level-override to all would address this.

Non-Hot-Swappable Devices

If the devices aren’t hot-swappable then you will have to stop the server to add the devices, in which case a cold-restart is not avoidable. The same procedure as for Hot-Swappable Devices applies just that it is now just a recommendation that the device order is maintained.

Expanding Storage Capacity

If you are expanding storage by replacing lesser capacity disks with larger capacity see “Replacing a Storage Device” instead.

Hot-Swappable Devices

To expand storage simply add the new devices to the node and zeroize them. Then, in the Aerospike’s configuration file append the devices to the end of the namespace’s storage devices, then stop and fast-start. If expanding multiple nodes simply perform the same procedure on each node, wait for the service to become ready.

NOTE: For versions not running the new cluster protocol introduced in versions 3.13/3.14, it may be advisable to wait for migrations to complete between each node (depending on consistency requirements).

Additionally, in Aerospike versions 3.8.3 or newer, rapid rebalance is included which greatly improves migration time.

WARNING: It is critical that the user appends the new devices to the end of the namespace storage devices (ie. the new devices must be the last ones in the list). This is important in order to take advantage of fast restart (if config permits it) as the index stores the device’s ids which are based on their order and would cause data misreads if a namesapce is fast restarted with a different order in devices. Cold restarts are not impacted.

WARNING: When adding a new device to a namespace, the data will not automatically rebalance across all the storage devices. As records are added / updated / their blocks defragmented, they will be re-distributed based on the new set of storage devices (per their digest hash). Depending on the workload, this could lead to low avail-pct situations on the previously used storage devices. Overtime, as all records get updated, the distribution should be very balanced. It may be necessary, though, to force such rebalance by force updating all the records (for example using the touch command), or temporarily forcing more aggressive defragmentation (increasing defrag-lwm-pct). This would require close monitoring to avoid inadvertently impacting performance.

Non-Hot-Swappable Devices

If the devices aren’t hot-swappable then you will have to stop the server to add the devices, in which case a cold-start is not avoidable. The same procedure as for Hot-Swappable Devices applies just that it is no longer important that the device be appended to the end.

Removing Storage Capacity

We can only support removing a device and fast-start if the device happens to be the last device defined for the namespace. In either case, the user should remove the device from the configuration. If the device was the last device defined then stop and fast-start, otherwise stop and cold-start. If multiple nodes require change then wait for the service to be ready and migrations to complete before continuing to the next node.

WARNING: For removal of a device on a configuration supporting fast-restart, we only support removing the last device defined for the namespace.

Data Distribution across devices

When expanding capacity or not maintaining the original order of the devices, the data is not redistributed all at once. Data is redistributed as new writes and defragmentation writes move existing records around. If a record hashes to a different device it will be queued for the appropriate device. Performance degradation caused by the redistribution should be negligible.

Procedures

Adding a Hot-Swappable device in Aerospike version 3.3.5 and above

The steps listed below should be followed only when using Hot-Swappable devices and Aerospike versions 3.3.5 and above. If expanding multiple nodes simply perform the same procedure for each node, waiting for the service to become ready. For greater detail see the Upgrade Hardware documented instructions. See: http://www.aerospike.com/docs/operations/upgrade/hardware

 For each node in the cluster (one node at a time)
 
 a. Stop Aerospike

    sudo service aerospike stop

 b. Add the new disk

 c. When the machine is ready, zeroize only the new disk
                  
    dd if=/dev/zero of=/dev/DEVICE bs =1M

      OR if blkdiscard is available:

    sudo blkdiscard /dev/DEVICE

 d. Add device by modifying the configuration file (e.g. aerospike.conf)

    Example of adding 2 new devices

    Before:
    namespace test {
           replication-factor 2
           memory-size 4G
           default-ttl 30d

           storage-engine device {
               device /dev/sdb1
               device /dev/sdb2
               write-block-size 128K
       	}
    }

    After:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d

    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
          	  	    device /dev/sdb3 # New device
          	  	    device /dev/sdb4 # New device
    		        write-block-size 128K
    	  }
    }

 e. [If necessary] Set read-consistency-level-override by modifying the configuration file (e.g. aerospike.conf)
 
    Only when replacing a disk in Aerospike version 3.10.1 or newer
    
    Before:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d
        
    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
          	        device /dev/sdb3 # New device
          	        device /dev/sdb4 # New device
    		        write-block-size 128K
       	}
    }

    After:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d
        read-consistency-level-override all #to reduce client_delete_not_found & client_read_not_found errors

    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
          	        device /dev/sdb3 # New device
          	        device /dev/sdb4 # New device
    		        write-block-size 128K
    	  }
    }
    
 
 f. Start Aerospike (fast-restart)

    sudo service aerospike start

 g. Verify Aerospike service is ready
 
    In the aerospike log search for "service ready: soon there will be cake!"
 
    sudo tail -f /var/log/aerospike/aerospike.log | grep "cake"
    
 h. Wait for Migrations to complete (optional)
 
    In the info output Namespace Information wait for Pending Migrates (tx%,rx%) to read (0,0)
    
    asadm -h 127.0.0.1 -e "info namespace"

Adding a Non Hot-Swappable device in Aerospike version 3.3.5 and above

The steps listed below should be followed only when using Non Hot-Swappable devices and Aerospike versions 3.3.5 and above. If expanding multiple nodes simply perform the same procedure for each node, waiting for the service to become ready. For greater detail see the Upgrade Hardware documented instructions. See: http://www.aerospike.com/docs/operations/upgrade/hardware

 For each node in the cluster (one node at a time)
 
 a. Stop Aerospike

    sudo service aerospike stop

 b. Add the new disk

 c. When the machine is ready, zeroize only the new disk
                  
    dd if=/dev/zero of=/dev/DEVICE bs =1M

      OR if blkdiscard is available:

    sudo blkdiscard /dev/DEVICE

 d. Add device by modifying the configuration file (e.g. aerospike.conf)

    Example of adding 2 new devices

    Before:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d

    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
    		  write-block-size 128K
       	}
    }

    After:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d

    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
          	        device /dev/sdb3 # New device
                    device /dev/sdb4 # New device
    		        write-block-size 128K
    	  }
    }

 e. Set read-consistency level-override by modifying the configuration file (e.g. aerospike.conf)
    
    Only when replacing a disk in Aerospike version 3.10.1 or newer
      
    Before:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d
            
    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
          	        device /dev/sdb3 # New device
          	        device /dev/sdb4 # New device
                    write-block-size 128K
       	}
    }
        
    After:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d
        read-consistency-level-override all #to reduce client_delete_not_found & client_read_not_found  errors
          
    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
          	  	device /dev/sdb3 # New device
          	  	device /dev/sdb4 # New device
    		        write-block-size 128K
    	  }
    }
    
 f. Start Aerospike (Cold-Start)

    sudo service aerospike start

 g. Verify Aerospike service is ready
 
    In the aerospike log search for "service ready: soon there will be cake!"
 
    sudo tail -f /var/log/aerospike/aerospike.log | grep "cake"
    
 h. Wait for Migrations to complete (optional)
 
    In the info output Namespace Information wait for Pending Migrates (tx%,rx%) to read (0,0)
    
    asadm -h 127.0.0.1 -e "info namespace"

Adding a device in Aerospike version 3.2.9 or below

The steps listed below should be followed only when using Aerospike versions 3.2.9 and below.

 For each node in the cluster (one node at a time)
 a. Take a backup of the node (optional if your replication factor > 1)

    for each namespace: 

    asbackup -h 127.0.0.1 -n NAMESPACE -d /path/to/backup/NAMESPACE -l 127.0.0.1:3000 -b

 b. Stop Aerospike

    /etc/init.d/aerospike stop

 c. Shutdown the machine and add the new disks

 d. When the machine is ready, zeroize all disks (new and old) in the affected namespace(s)
              
    for each device in affected namespace(s):

    # For devices that do not support TRIM:
    dd of=/dev/zero if=/dev/DEVICE bs=1M

    # For TRIM supported devices:
    sudo blkdiscard /dev/DEVICE

 e. Start Aerospike

    /etc/init.d/aerospike start

 f. If backup(s) were taken restore the backup(s)
    for each namespace:

    asrestore -h 127.0.0.1 -n NAMESPACE -d /path/to/backup/NAMESPACE

Removing a device

If removing multiple nodes simply perform the same procedure for each node, waiting for the service to become ready. If the device was the last device defined in the namespace’s storage listing then Aerospike will fast-start. If the device was the not the last device defined in the namespace’s storage listing then Aerospike will cold-start.

WARNING: If wanting to remove a device with a fast-start, we only support removing the last device defined for the namespace.

NOTE: To reuse a device after removal from the Namespace simply follow the instructions contained herein for Adding a device.

 a. As a safeguard, take a backup of the node (optional if your replication factor > 1)

    for each namespace: 

    asbackup -h 127.0.0.1 -n NAMESPACE -d /path/to/backup/NAMESPACE -l 127.0.0.1:3000 -b
    
 b. Wait for Migrations to complete
 
    In the info output Namespace Information wait for Pending Migrates (tx%,rx%) to read (0,0)
    
    asadm -h 127.0.0.1 -e "info namespace"

 c. Stop Aerospike

    /etc/init.d/aerospike stop
    
 d. Remove the disk
 
 e. Remove device by modifying the configuration file (e.g. aerospike.conf)
  
    Either comment out or delete the device to be removed
     
    Example of removing a device
    
    Before:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d
    
    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
          	  	device /dev/sdb3 # device being removed
    		  write-block-size 128K
       	}
    }
    
    After:
    namespace test {
     	  replication-factor 2
    	  memory-size 4G
    	  default-ttl 30d
      
    	  storage-engine device {
    		        device /dev/sdb1
    		        device /dev/sdb2
    		        write-block-size 128K
    	  }
    }
      
 f. Start Aerospike
  
    sudo service aerospike start

 g. Verify Aerospike service is ready
 
    In the aerospike log search for "service ready: soon there will be cake!"
 
    sudo tail -f /var/log/aerospike/aerospike.log | grep "cake"
    
 h. Wait for Migrations to complete
 
    In the info output Namespace Information wait for Pending Migrates (tx%,rx%) to read (0,0)
    
    asadm -h 127.0.0.1 -e "info namespace"

References

Keywords

Add Remove Replace Upgrade Hardware Devices Zeroize Configuration Fast Warm Cold Start Migrations

Timestamp

03/24/2017 12/15/2017


Removing SSD's from namespace
How does Aerospike protect my data in case of a SSD failure?
Cold restart failed - tombstone found in CE after reboot
What is the difference between device_available_pct and device_free_pct