Aerospike Configuration Review


#1

Continuing the discussion from Wrong metrics from AS:

# cat /etc/aerospike/aerospike.conf
    service {
        user                              root
        group                             root
        run-as-daemon
        paxos-single-replica-limit        1
        paxos-recovery-policy             manual
        auto-dun                          false
        auto-undun                        false
    
        pidfile                           /var/run/aerospike/asd.pid
    
        service-threads                   24
        fabric-workers                    36
    
        transaction-queues                12
        transaction-threads-per-queue     3
        transaction-pending-limit         30
    
    
        nsup-startup-evict                true
        nsup-period                       120
        nsup-queue-escape                 10
        nsup-queue-hwm                    500
        nsup-queue-lwm                    1
    
        storage-benchmarks                false
        microbenchmarks                   false
    
        respond-client-on-master-completion    true
    
        proto-fd-max                      50000
    
        hist-track-thresholds              1,4,8,16,32,64,128,256,512,1024,2048,4096,8192
    
    }
    
    logging {
        file /var/log/aerospike/aerospike.log {
            context any info
        }
    }
    
    network {
        service {
            address                       any
            port                          3000
            access-address                x.x.x.x
            reuse-address
        }
        heartbeat {
            mode multicast
            address 239.1.99.222
            port 9918
    
            interval 1500
            timeout 10
        }
        fabric {
            port 3001
        }
        info {
            address                       any
            port                          3003
        }
    }
    
    namespace ssd {
        replication-factor                2
        default-ttl                       0
        memory-size                       118G
        high-water-memory-pct             99
        high-water-disk-pct               80
        stop-writes-pct                   80
    
        storage-engine device {
            scheduler-mode                noop
            device                        /dev/sda4
            device                        /dev/sdb4
            write-block-size              128k
            defrag-lwm-pct                50
            min-avail-pct                 1
            data-in-memory                true
            max-write-cache               134217728
            post-write-queue              2048
        }
    
    }

#2

@garicknk, I noticed a few potential problems with your config and thought I would bring them up here.

Please see the “# NOTE” lines below.

# cat /etc/aerospike/aerospike.conf
    service {
        user                              root
        group                             root
        run-as-daemon
        paxos-single-replica-limit        1
        paxos-recovery-policy             manual
        auto-dun                          false
        auto-undun                        false
    
        pidfile                           /var/run/aerospike/asd.pid
    
        service-threads                   24  # NOTE: This shouldn't be an issue, but for most use cases 4 is more than enough threads.
        fabric-workers                    36
    
        transaction-queues                12 # NOTE: Since you are using persistence we typically see better performance with 8 queues and 8 threads per queue or 4 queues and 16 threads per queue.
        transaction-threads-per-queue     3
        transaction-pending-limit         30
    
    
        nsup-startup-evict                true
        nsup-period                       120
        nsup-queue-escape                 10
        nsup-queue-hwm                    500
        nsup-queue-lwm                    1
    
        storage-benchmarks                false
        microbenchmarks                   false
    
        respond-client-on-master-completion    true # NOTE: In the next release this can be configured per namespace, and per transaction... though initially only the C client will support per transaction.
    
        proto-fd-max                      50000
    
        hist-track-thresholds              1,4,8,16,32,64,128,256,512,1024,2048,4096,8192
    
    }
    
    logging {
        file /var/log/aerospike/aerospike.log {
            context any info
        }
    }
    
    network {
        service {
            address                       any
            port                          3000
            access-address                x.x.x.x
            reuse-address
        }
        heartbeat {
            mode multicast
            address 239.1.99.222
            port 9918
    
            interval 1500  # NOTE: interval of 1500 and timeout of 10 means that it will take 15 seconds for the cluster to recognize that a node has left.
            timeout 10
        }
        fabric {
            port 3001
        }
        info {
            address                       any
            port                          3003
        }
    }
    
    namespace ssd {
        replication-factor                2
        default-ttl                       0
        memory-size                       118G
        high-water-memory-pct             99 # NOTE: if all your records are written with 0 TTL, these high-water-[memory/disk]-pct will not have any impact.
        high-water-disk-pct               80
        stop-writes-pct                   80 # NOTE: Awesome, I was surprised to see this; when a node isn't able to evict, we typically recommend a stop-writes-pct at 80%.
    
        storage-engine device {
            scheduler-mode                noop
            device                        /dev/sda4
            device                        /dev/sdb4
            write-block-size              128k
            defrag-lwm-pct                50
            min-avail-pct                 1  # NOTE: this is potentially dangerous, it is conceivable that your node could write more than 1% of your data in the 120 second NSUP interval which would result in a situation requiring a node restart to recover from. 
            data-in-memory                true
            max-write-cache               134217728
            post-write-queue              2048  # NOTE: This is meaningless for data-in-memory.
        }
    
    }

#3

Hi @kporter, big thanks for comments. On next cluster update, we test threads settings.

interval - is set one some time on network problems in our hoster now is:

    interval                      100
    timeout                       10

high-water-memory-pct - not all records, some set use ttl changes, some use ttl 0 and we use cleanup service with some strange logic :smile: )))

min-avail-pct - its timed parameter. Hotser tightened the installation of a new server. We have reached the size of free space on noded and lower parameters to exclude data drops on some time. Now is 5


#4

Please check following configuration, we are able to do 1000 QPS using java application asynch client, post which at 1500 QPS , the read threads keep waiting and latency shoots up really high in reading from aerospike. What could be wrong? we are using r3.2xlarge machine for aerospike db (8 core and 64 gb ram) and the java application that connects to aerospike is 4core 16gb ram machine.

# Aerospike database configuration file.

service {
	user root
	group root
	paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
	pidfile /var/run/aerospike/asd.pid
	service-threads 4
	transaction-queues 4
	transaction-threads-per-queue 4
	proto-fd-max 50000
}

logging {
	# Log file must be an absolute path.
	file /var/log/aerospike/aerospike.log {
		context any info
	}
}

network {
	service {
		address any
		port 3000
	}

	heartbeat {
		mode multicast
		multicast-group 239.1.99.222
		port 9918

		# To use unicast-mesh heartbeats, remove the 3 lines above, and see
		# aerospike_mesh.conf for alternative.

		interval 150
		timeout 10
	}

	fabric {
		port 3001
	}

	info {
		port 3003
	}
}

#namespace test {
	#replication-factor 2
	#memory-size 4G
	#default-ttl 30d # 30 days, use 0 to never expire/evict.

	#storage-engine memory
#}

namespace user {
    replication-factor 1
    memory-size 14G
    high-water-memory-pct 85
    high-water-disk-pct 85
    stop-writes-pct 85
    default-ttl 7d # 30 days, use 0 to never expire/evict.

    #storage-engine memory

    # To use file storage backing, comment out the line above and use the
    # following lines instead.
    storage-engine device {
        file /opt/aerospike/data/user.dat
        filesize 50G
        data-in-memory false # Store data in memory in addition to file.
        defrag-lwm-pct 60
    }
}

namespace dsp_user_sync {
    replication-factor 1
    memory-size 14G
    high-water-memory-pct 85
    high-water-disk-pct 85
    stop-writes-pct 85
    default-ttl 30d # 30 days, use 0 to never expire/evict.

    #storage-engine memory

    # To use file storage backing, comment out the line above and use the
    # following lines instead.
    storage-engine device {
        file /opt/aerospike/data/user_sync.dat
        filesize 50G
        data-in-memory false # Store data in memory in addition to file.
    }
}

#5

Generally useful to have the version of Aerospike as well.

Which type of operation are you performing when you say QPS?

3.12.0+:

If you are using 3.12.0 then could remove:

	service-threads 4
	transaction-queues 4
	transaction-threads-per-queue 4

Aerospike will then automatically configure service-threads and transaction-queues to the number of CPUs and the default for transaction-threads-per-queue is 4.

Before 3.12.0:

If you are prior to 3.12.0 then the general recommendation is to set both service-threads and transaction-queues to the number of cores (8 in your case). For higher core counts we typically recommend setting transaction-threads-per-queue 3, but in this case I would compare 3, 4, and 8.

Either:

In addition to this, what type of storage are you using: SSD or Rotational?

You are using file which means you are going through the filesystem which will add overhead. If possible I suggest using device which removes the filesystem layer.

Another suggestion is to spend a bit of RAM on a post write cache, see post-write-queue.


#6

@kporter is there some area where we can reference all the config changes that have taken place? I’d like to see if there is anything else that has changed in 3.12 that I should be aware of, like what you mentioned above…


#7

These aren’t deprecated btw, the defaults have just been changed from a static number to being defined by the number of CPUs.

For configuration changes, keep an eye on the release notes and you can also check the configuration reference.