@kporter, i been also facing the same issues at my setup, as i use single AS community server, we have high volumes of data and it is stored with 300seconds on ttl on each record, but it never expires any, here is a snapshot of some metrics which might help you.
[ec2-user@ip-172-30-1-180 ~]$ cat /etc/aerospike/aerospike.conf
# Aerospike database configuration file.
service {
user root
group root
paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
pidfile /newvolume/var/run/aerospike/asd.pid
service-threads 16
transaction-queues 16
transaction-threads-per-queue 4
proto-fd-max 99999
proto-fd-idle-ms 10000
nsup-period 10
}
logging {
# Log file must be an absolute path.
file /newvolume/var/log/aerospike/aerospike.log {
context any info
}
}
network {
service {
address any
port 3000
reuse-address
access-address 172.30.1.180 virtual
}
heartbeat {
mode multicast
address 239.1.99.222
port 9918
# To use unicast-mesh heartbeats, remove the 3 lines above, and see
# aerospike_mesh.conf for alternative.
interval 150
timeout 10
}
fabric {
port 3001
}
info {
port 3003
}
}
namespace centralns {
replication-factor 1
memory-size 108G
default-ttl 0
storage-engine device {
file /newvolume/opt/aerospike/data/centralns.dat
filesize 1000G
write-block-size 128K
}
}
[ec2-user@ip-172-30-1-180 ~]$ asmonitor
Enter help for commands
1 hosts in cluster: 172.30.1.180:3000
Aerospike Interactive Shell, version 3.7.5
Monitor> info
===NODES===
2017-05-19 12:49:01.669654
Sorting by IP, in Ascending order:
ip:port Build Cluster Cluster Free Free Migrates Node Principal Replicated Sys
. Size Visibility Disk Mem . ID ID Objects Free
. . . pct pct . . . . Mem
172.30.1.180:3000 3.7.5.1 1 true 93 94 (0,0) BB9B6941919D10A BB9B6941919D10A 99,237,126 94
Number of nodes displayed: 1
===NAMESPACE===
Total (unique) objects in cluster for centralns : 99,237,140
Note: Total (unique) objects is an under estimate if migrations are in progress.
ip/namespace Avail Evicted Master Repl Stop Used Used Used Used hwm hwm
Pct Objects Objects Factor Writes Disk Disk Mem Mem Disk Mem
. . . . . . % . % . .
172.30.1.180/centralns 92 0 99,237,140 1 false 64.28 G 7 5.91 G 6 50 60
Number of rows displayed: 1
Monitor> exit
Config files location : /home/ec2-user/.asmonitor/asmonitor.conf
Folder used by asmonitor for files : /home/ec2-user/.asmonitor/
[ec2-user@ip-172-30-1-180 ~]$ asinfo -l -v 'statistics'
cluster_size=1
cluster_key=F98896FAC496E218
cluster_integrity=true
objects=99270160
sub-records=0
total-bytes-disk=1073741824000
used-bytes-disk=69042935296
free-pct-disk=93
total-bytes-memory=115964116992
used-bytes-memory=6353290240
data-used-bytes-memory=0
index-used-bytes-memory=6353290240
sindex-used-bytes-memory=0
free-pct-memory=94
stat_read_reqs=6085393098
stat_read_reqs_xdr=0
stat_read_success=2121457454
stat_read_errs_notfound=3963935641
stat_read_errs_other=0
stat_write_reqs=322758276
stat_write_reqs_xdr=0
stat_write_success=322758276
stat_write_errs=0
stat_xdr_pipe_writes=0
stat_xdr_pipe_miss=0
stat_delete_success=223057803
stat_rw_timeout=0
udf_read_reqs=0
udf_read_success=0
udf_read_errs_other=0
udf_write_reqs=0
udf_write_success=0
udf_write_err_others=0
udf_delete_reqs=0
udf_delete_success=0
udf_delete_err_others=0
udf_lua_errs=0
udf_scan_rec_reqs=0
udf_query_rec_reqs=0
udf_replica_writes=0
stat_proxy_reqs=0
stat_proxy_reqs_xdr=0
stat_proxy_success=0
stat_proxy_errs=0
stat_ldt_proxy=0
stat_cluster_key_err_ack_dup_trans_reenqueue=0
stat_expired_objects=114504343
stat_evicted_objects=0
stat_deleted_set_objects=0
stat_evicted_objects_time=0
stat_zero_bin_records=0
stat_nsup_deletes_not_shipped=162294291
stat_compressed_pkts_received=0
err_tsvc_requests=21554
err_tsvc_requests_timeout=0
err_out_of_space=0
err_duplicate_proxy_request=0
err_rw_request_not_found=13
err_rw_pending_limit=0
err_rw_cant_put_unique=0
geo_region_query_count=0
geo_region_query_cells=0
geo_region_query_points=0
geo_region_query_falsepos=0
fabric_msgs_sent=0
fabric_msgs_rcvd=0
paxos_principal=BB9B6941919D10A
migrate_msgs_sent=0
migrate_msgs_recv=0
migrate_progress_send=0
migrate_progress_recv=0
migrate_num_incoming_accepted=0
migrate_num_incoming_refused=0
queue=4
transactions=4040145455
reaped_fds=261460
scans_active=0
basic_scans_succeeded=0
basic_scans_failed=1
aggr_scans_succeeded=0
aggr_scans_failed=0
udf_bg_scans_succeeded=0
udf_bg_scans_failed=0
batch_index_initiate=3508538128
batch_index_queue=1:0,1:0,1:0,3:0
batch_index_complete=3508538122
batch_index_timeout=0
batch_index_errors=0
batch_index_unused_buffers=250
batch_index_huge_buffers=0
batch_index_created_buffers=29081
batch_index_destroyed_buffers=28825
batch_initiate=0
batch_queue=0
batch_tree_count=0
batch_timeout=0
batch_errors=0
info_queue=0
delete_queue=16812
proxy_in_progress=0
proxy_initiate=0
proxy_action=0
proxy_retry=0
proxy_retry_q_full=0
proxy_unproxy=0
proxy_retry_same_dest=0
proxy_retry_new_dest=0
write_master=322758278
write_prole=0
read_dup_prole=0
rw_err_dup_internal=0
rw_err_dup_cluster_key=0
rw_err_dup_send=0
rw_err_write_internal=0
rw_err_write_cluster_key=0
rw_err_write_send=0
rw_err_ack_internal=0
rw_err_ack_nomatch=0
rw_err_ack_badnode=0
client_connections=2667
waiting_transactions=0
tree_count=0
record_refs=99270178
record_locks=1
migrate_tx_objs=0
migrate_rx_objs=0
ongoing_write_reqs=0
err_storage_queue_full=0
partition_actual=4096
partition_replica=0
partition_desync=0
partition_absent=0
partition_zombie=0
partition_object_count=99270161
partition_ref_count=4118
system_free_mem_pct=94
sindex_ucgarbage_found=0
sindex_gc_locktimedout=0
sindex_gc_inactivity_dur=0
sindex_gc_activity_dur=0
sindex_gc_list_creation_time=0
sindex_gc_list_deletion_time=0
sindex_gc_objects_validated=0
sindex_gc_garbage_found=0
sindex_gc_garbage_cleaned=0
system_swapping=false
err_replica_null_node=0
err_replica_non_null_node=0
err_sync_copy_null_master=0
storage_defrag_corrupt_record=0
err_write_fail_prole_unknown=0
err_write_fail_prole_generation=0
err_write_fail_unknown=0
err_write_fail_key_exists=0
err_write_fail_generation=0
err_write_fail_generation_xdr=0
err_write_fail_bin_exists=0
err_write_fail_parameter=0
err_write_fail_incompatible_type=0
err_write_fail_noxdr=0
err_write_fail_prole_delete=0
err_write_fail_not_found=0
err_write_fail_key_mismatch=0
err_write_fail_record_too_big=0
err_write_fail_bin_name=0
err_write_fail_bin_not_found=0
err_write_fail_forbidden=0
stat_duplicate_operation=0
uptime=38517
stat_write_errs_notfound=0
stat_write_errs_other=0
heartbeat_received_self=255951
heartbeat_received_foreign=0
query_reqs=0
query_success=0
query_fail=0
query_abort=0
query_avg_rec_count=0
query_short_running=0
query_long_running=0
query_short_queue_full=0
query_long_queue_full=0
query_short_reqs=0
query_long_reqs=0
query_agg=0
query_agg_success=0
query_agg_err=0
query_agg_abort=0
query_agg_avg_rec_count=0
query_lookups=0
query_lookup_success=0
query_lookup_err=0
query_lookup_abort=0
query_lookup_avg_rec_count=0
[ec2-user@ip-172-30-1-180 ~]$ asinfo -v 'get-config:' -l
transaction-queues=16
transaction-threads-per-queue=4
transaction-duplicate-threads=0
transaction-pending-limit=20
migrate-threads=1
migrate-max-num-incoming=256
migrate-rx-lifetime-ms=60000
proto-fd-max=99999
proto-fd-idle-ms=10000
proto-slow-netio-sleep-ms=1
transaction-retry-ms=1000
transaction-max-ms=1000
transaction-repeatable-read=false
ticker-interval=10
log-local-time=false
microbenchmarks=false
storage-benchmarks=false
ldt-benchmarks=false
scan-max-active=100
scan-max-done=100
scan-max-udf-transactions=32
scan-threads=4
batch-index-threads=4
batch-threads=4
batch-max-requests=5000
batch-max-buffers-per-queue=255
batch-max-unused-buffers=256
batch-priority=200
nsup-delete-sleep=100
nsup-period=10
nsup-startup-evict=true
paxos-retransmit-period=5
paxos-single-replica-limit=1
paxos-max-cluster-size=32
paxos-protocol=v3
paxos-recovery-policy=manual
write-duplicate-resolution-disable=false
respond-client-on-master-completion=false
replication-fire-and-forget=false
info-threads=16
allow-inline-transactions=true
use-queue-per-device=false
snub-nodes=false
prole-extra-ttl=0
max-msgs-per-type=-1
service-threads=16
fabric-workers=16
pidfile=/newvolume/var/run/aerospike/asd.pid
memory-accounting=false
udf-runtime-gmax-memory=18446744073709551615
udf-runtime-max-memory=18446744073709551615
sindex-builder-threads=4
sindex-data-max-memory=ULONG_MAX
query-threads=6
query-worker-threads=15
query-priority=10
query-in-transaction-thread=0
query-req-in-query-thread=0
query-req-max-inflight=100
query-bufpool-size=256
query-batch-size=100
query-priority-sleep-us=1
query-short-q-max-size=500
query-long-q-max-size=500
query-rec-count-bound=18446744073709551615
query-threshold=10
query-untracked-time-ms=1000
query-pre-reserve-partitions=false
service-address=0.0.0.0
service-port=3000
access-address=172.30.1.180
reuse-address=true
fabric-port=3001
fabric-keepalive-enabled=true
fabric-keepalive-time=1
fabric-keepalive-intvl=1
fabric-keepalive-probes=10
network-info-port=3003
heartbeat-mode=multicast
heartbeat-protocol=v2
heartbeat-address=239.1.99.222
heartbeat-port=9918
heartbeat-interval=150
heartbeat-timeout=10
enable-security=false
privilege-refresh-period=300
report-authentication-sinks=0
report-data-op-sinks=0
report-sys-admin-sinks=0
report-user-admin-sinks=0
report-violation-sinks=0
syslog-local=-1
enable-xdr=false
xdr-namedpipe-path=NULL
forward-xdr-writes=false
xdr-delete-shipping-enabled=true
xdr-nsup-deletes-enabled=false
stop-writes-noxdr=false
reads-hist-track-back=1800
reads-hist-track-slice=10
reads-hist-track-thresholds=1,8,64
writes_master-hist-track-back=1800
writes_master-hist-track-slice=10
writes_master-hist-track-thresholds=1,8,64
proxy-hist-track-back=1800
proxy-hist-track-slice=10
proxy-hist-track-thresholds=1,8,64
udf-hist-track-back=1800
udf-hist-track-slice=10
udf-hist-track-thresholds=1,8,64
query-hist-track-back=1800
query-hist-track-slice=10
query-hist-track-thresholds=1,8,64
query_rec_count-hist-track-back=1800
query_rec_count-hist-track-slice=10
query_rec_count-hist-track-thresholds=1,8,64
[ec2-user@ip-172-30-1-180 ~]$ asinfo -v 'hist-dump:ns=centralns;hist=ttl'
centralns:ttl=100,864,5060364,4443960,3856727,3406033,3064149,2753621,2608490,2229002,2031532,1684548,1373003,1065835,770459,466249,165430,328,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,306361,198587,171518,156951,126786,117620,110467,98720,80378,78223,69918,69741,57858,55356,59457,53118,542274;
[ec2-user@ip-172-30-1-180 ~]$ asinfo -v 'get-config:context=namespace;id=centralns' -l
memory-size=115964116992
high-water-disk-pct=50
high-water-memory-pct=60
evict-tenths-pct=5
stop-writes-pct=90
cold-start-evict-ttl=4294967295
repl-factor=1
default-ttl=0
max-ttl=0
conflict-resolution-policy=generation
single-bin=false
ldt-enabled=false
ldt-page-size=8192
enable-xdr=false
sets-enable-xdr=true
ns-forward-xdr-writes=false
allow-nonxdr-writes=true
allow-xdr-writes=true
disallow-null-setname=false
total-bytes-memory=115964116992
read-consistency-level-override=off
write-commit-level-override=off
migrate-order=5
migrate-sleep=1
migrate-tx-partitions-initial=0
migrate-tx-partitions-remaining=0
migrate-rx-partitions-initial=0
migrate-rx-partitions-remaining=0
migrate-tx-partitions-imbalance=0
total-bytes-disk=1073741824000
defrag-lwm-pct=50
defrag-queue-min=0
defrag-sleep=1000
defrag-startup-minimum=10
flush-max-ms=1000
fsync-max-sec=0
max-write-cache=67108864
min-avail-pct=5
post-write-queue=256
data-in-memory=false
file=/newvolume/opt/aerospike/data/centralns.dat
filesize=1073741824000
writethreads=1
writecache=67108864
obj-size-hist-max=100
and most of the time i see in logs is this, i hope nsup is not running or something.
May 19 2017 07:23:13 GMT: INFO (info): (thr_info.c::4996) tree_counts: nsup 0 scan 0 dup 0 wprocess 0 migrx 0 migtx 0 ssdr 0 ssdw 0 rw 4
Let me give you the pin point picture.