@garicknk, I noticed a few potential problems with your config and thought I would bring them up here.
Please see the ā# NOTEā lines below.
# cat /etc/aerospike/aerospike.conf
service {
user root
group root
run-as-daemon
paxos-single-replica-limit 1
paxos-recovery-policy manual
auto-dun false
auto-undun false
pidfile /var/run/aerospike/asd.pid
service-threads 24 # NOTE: This shouldn't be an issue, but for most use cases 4 is more than enough threads.
fabric-workers 36
transaction-queues 12 # NOTE: Since you are using persistence we typically see better performance with 8 queues and 8 threads per queue or 4 queues and 16 threads per queue.
transaction-threads-per-queue 3
transaction-pending-limit 30
nsup-startup-evict true
nsup-period 120
nsup-queue-escape 10
nsup-queue-hwm 500
nsup-queue-lwm 1
storage-benchmarks false
microbenchmarks false
respond-client-on-master-completion true # NOTE: In the next release this can be configured per namespace, and per transaction... though initially only the C client will support per transaction.
proto-fd-max 50000
hist-track-thresholds 1,4,8,16,32,64,128,256,512,1024,2048,4096,8192
}
logging {
file /var/log/aerospike/aerospike.log {
context any info
}
}
network {
service {
address any
port 3000
access-address x.x.x.x
reuse-address
}
heartbeat {
mode multicast
address 239.1.99.222
port 9918
interval 1500 # NOTE: interval of 1500 and timeout of 10 means that it will take 15 seconds for the cluster to recognize that a node has left.
timeout 10
}
fabric {
port 3001
}
info {
address any
port 3003
}
}
namespace ssd {
replication-factor 2
default-ttl 0
memory-size 118G
high-water-memory-pct 99 # NOTE: if all your records are written with 0 TTL, these high-water-[memory/disk]-pct will not have any impact.
high-water-disk-pct 80
stop-writes-pct 80 # NOTE: Awesome, I was surprised to see this; when a node isn't able to evict, we typically recommend a stop-writes-pct at 80%.
storage-engine device {
scheduler-mode noop
device /dev/sda4
device /dev/sdb4
write-block-size 128k
defrag-lwm-pct 50
min-avail-pct 1 # NOTE: this is potentially dangerous, it is conceivable that your node could write more than 1% of your data in the 120 second NSUP interval which would result in a situation requiring a node restart to recover from.
data-in-memory true
max-write-cache 134217728
post-write-queue 2048 # NOTE: This is meaningless for data-in-memory.
}
}
Hi
@kporter, big thanks for comments.
On next cluster update, we test threads settings.
interval - is set one some time on network problems in our hoster now is:
interval 100
timeout 10
high-water-memory-pct - not all records, some set use ttl changes, some use ttl 0 and we use cleanup service with some strange logic )))
min-avail-pct - its timed parameter. Hotser tightened the installation of a new server. We have reached the size of free space on noded and lower parameters to exclude data drops on some time. Now is 5
Please check following configuration, we are able to do 1000 QPS using java application asynch client,
post which at 1500 QPS , the read threads keep waiting and latency shoots up really high in reading
from aerospike. What could be wrong? we are using r3.2xlarge machine for aerospike db (8 core and 64 gb ram) and the java application that connects to aerospike is 4core 16gb ram machine.
# Aerospike database configuration file.
service {
user root
group root
paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
pidfile /var/run/aerospike/asd.pid
service-threads 4
transaction-queues 4
transaction-threads-per-queue 4
proto-fd-max 50000
}
logging {
# Log file must be an absolute path.
file /var/log/aerospike/aerospike.log {
context any info
}
}
network {
service {
address any
port 3000
}
heartbeat {
mode multicast
multicast-group 239.1.99.222
port 9918
# To use unicast-mesh heartbeats, remove the 3 lines above, and see
# aerospike_mesh.conf for alternative.
interval 150
timeout 10
}
fabric {
port 3001
}
info {
port 3003
}
}
#namespace test {
#replication-factor 2
#memory-size 4G
#default-ttl 30d # 30 days, use 0 to never expire/evict.
#storage-engine memory
#}
namespace user {
replication-factor 1
memory-size 14G
high-water-memory-pct 85
high-water-disk-pct 85
stop-writes-pct 85
default-ttl 7d # 30 days, use 0 to never expire/evict.
#storage-engine memory
# To use file storage backing, comment out the line above and use the
# following lines instead.
storage-engine device {
file /opt/aerospike/data/user.dat
filesize 50G
data-in-memory false # Store data in memory in addition to file.
defrag-lwm-pct 60
}
}
namespace dsp_user_sync {
replication-factor 1
memory-size 14G
high-water-memory-pct 85
high-water-disk-pct 85
stop-writes-pct 85
default-ttl 30d # 30 days, use 0 to never expire/evict.
#storage-engine memory
# To use file storage backing, comment out the line above and use the
# following lines instead.
storage-engine device {
file /opt/aerospike/data/user_sync.dat
filesize 50G
data-in-memory false # Store data in memory in addition to file.
}
}
Aerospike will then automatically configure service-threads and transaction-queues to the number of CPUs and the default for transaction-threads-per-queue is 4.
Before 3.12.0:
If you are prior to 3.12.0 then the general recommendation is to set both service-threads and transaction-queues to the number of cores (8 in your case). For higher core counts we typically recommend setting transaction-threads-per-queue 3, but in this case I would compare 3, 4, and 8.
Either:
In addition to this, what type of storage are you using: SSD or Rotational?
You are using file which means you are going through the filesystem which will add overhead. If possible I suggest using device which removes the filesystem layer.
Another suggestion is to spend a bit of RAM on a post write cache, see post-write-queue.
@kporter is there some area where we can reference all the config changes that have taken place? Iād like to see if there is anything else that has changed in 3.12 that I should be aware of, like what you mentioned aboveā¦
)))