Hi
I am testing an aerospike cluster with real data (25M recrods) and am getting unexpectedly slow performance for aggregations. The system is running on AWS as a cluster of 2 machines both r3XXLarge instance types in the same zone. I have 1 nic on each machine and have run the afterburner script as well as setting RPS for each machine. I have set the query-threads to 8 (as that’s the core count). I have a secondary index which I am performing the aggregation against as follows: function count(s) function mapper(rec) return 1 end local function reducer(v1, v2) return v1 + v2 end return s: map(mapper) : reduce(reducer) end
The results from aql are: aggregate analytics_getdatafordate.count() on testdata.pk where apikeyid = 7
±---------+ | count | ±---------+ | 12505912 | ±---------+ 1 row in set (25.152 secs)
The secondary key is a numeric. This is effectively the same as running a ‘count(*) from’ query on an SQL database. On a development machine with the SQL database this data comes from the query response time is 2 seconds. Below is the result of the get-config:
transaction-queues=8 transaction-threads-per-queue=3 transaction-duplicate-threads=0 transaction-pending-limit=20 migrate-threads=1 migrate-xmit-priority=40 migrate-xmit-sleep=500 migrate-read-priority=10 migrate-read-sleep=500 migrate-xmit-hwm=10 migrate-xmit-lwm=5 migrate-max-num-incoming=256 migrate-rx-lifetime-ms=60000 proto-fd-max=15000 proto-fd-idle-ms=60000 proto-slow-netio-sleep-ms=1 transaction-retry-ms=1000 transaction-max-ms=1000 transaction-repeatable-read=false dump-message-above-size=134217728 ticker-interval=10 microbenchmarks=false storage-benchmarks=false ldt-benchmarks=false scan-priority=200 scan-sleep=1 batch-threads=4 batch-max-requests=5000 batch-priority=200 nsup-delete-sleep=100 nsup-period=120 nsup-startup-evict=true paxos-retransmit-period=5 paxos-single-replica-limit=1 paxos-max-cluster-size=32 paxos-protocol=v3 paxos-recovery-policy=manual write-duplicate-resolution-disable=false respond-client-on-master-completion=false replication-fire-and-forget=false info-threads=16 allow-inline-transactions=true use-queue-per-device=false snub-nodes=false fb-health-msg-per-burst=0 fb-health-msg-timeout=200 fb-health-good-pct=50 fb-health-bad-pct=0 auto-dun=false auto-undun=false prole-extra-ttl=0 max-msgs-per-type=-1 service-threads=8 fabric-workers=16 pidfile=/var/run/aerospike/asd.pid memory-accounting=false udf-runtime-gmax-memory=18446744073709551615 udf-runtime-max-memory=18446744073709551615 sindex-populator-scan-priority=3 sindex-data-max-memory=18446744073709551615 query-threads=8 query-worker-threads=15 query-priority=10 query-in-transaction-thread=0 query-req-in-query-thread=0 query-req-max-inflight=100 query-bufpool-size=256 query-batch-size=100 query-sleep=1 query-job-tracking=false query-short-q-max-size=500 query-long-q-max-size=500 query-rec-count-bound=4294967295 query-threshold=10 query-untracked-time=1000000 service-address=0.0.0.0 service-port=3000 mesh-seed-address-port=10.0.101.126:3002 reuse-address=true fabric-port=3001 network-info-port=3003 enable-fastpath=true heartbeat-mode=mesh heartbeat-protocol=v2 heartbeat-address=10.0.101.216 heartbeat-port=3002 heartbeat-interval=150 heartbeat-timeout=10 enable-security=false privilege-refresh-period=300 report-authentication-sinks=0 report-data-op-sinks=0 report-sys-admin-sinks=0 report-user-admin-sinks=0 report-violation-sinks=0 syslog-local=-1 enable-xdr=false forward-xdr-writes=false xdr-delete-shipping-enabled=true xdr-nsup-deletes-enabled=false stop-writes-noxdr=false reads-hist-track-back=1800 reads-hist-track-slice=10 reads-hist-track-thresholds=1,8,64 writes_master-hist-track-back=1800 writes_master-hist-track-slice=10 writes_master-hist-track-thresholds=1,8,64 proxy-hist-track-back=1800 proxy-hist-track-slice=10 proxy-hist-track-thresholds=1,8,64 writes_reply-hist-track-back=1800 writes_reply-hist-track-slice=10 writes_reply-hist-track-thresholds=1,8,64 udf-hist-track-back=1800 udf-hist-track-slice=10 udf-hist-track-thresholds=1,8,64 query-hist-track-back=1800 query-hist-track-slice=10 query-hist-track-thresholds=1,8,64 query_rec_count-hist-track-back=1800 query_rec_count-hist-track-slice=10 query_rec_count-hist-track-thresholds=1,8,64
Thanks
Craig.