FAQ – How do I monitor incoming client traffic and connections to Aerospike?
Monitoring client traffic and connections might be required in the following situations or for general review of a cluster.
- To confirm whether traffic has moved away from a node which was quiesced.
- To check if there is high incoming and/or churning client connections impacting applications.
- To check client conectivity during general testing.
The Aerospike DB has a client connections upper limit of proto-fd-max per node, beyond which incoming connections will be rejected. Thus, it is essential to monitor the client connections and configure a cluster appropriately. This article will cover monitoring recommendations.
Verify incoming traffic
A quick and easy way to check for basic read or write traffic on an Aerospike cluster is to use the ‘show latencies’ command from asadm:
Admin> show latencies read Latency (2018-10-25 23:49:19 UTC Node Ops/Sec %>1ms %>8ms %>64ms aero-cluster1_1:3000 9.3 0.0 0.0 0.0 aero-cluster1_2:3000 0.0 0.0 0.0 0.0 aero-cluster1_3:3000 7.5 0.0 0.0 0.0 aero-cluster1_4:3000 8.4 0.0 0.0 0.0 Number of rows: 1 write Latency (2018-10-25 23:49:19 UTC Node Ops/Sec %>1ms %>8ms %>64ms aero-cluster1_1:3000 2.0 0.0 0.0 0.0 aero-cluster1_2:3000 0.0 0.0 0.0 0.0 aero-cluster1_3:3000 3.2 0.0 0.0 0.0 aero-cluster1_4:3000 4.5 0.0 0.0 0.0 Number of rows: 1
The Ops/Sec column indicates the throughput. A throughput of 0 would obviously mean the node is not receiving any transaction of the given type and could imply connectivity issues with the clients. In the example above, a 4 node cluster is showing a low read/write throughput on 3 out of the 4 nodes. The
show latencies command would also show traffic for batch and secondary index queries. For scans, one would have to look at the scan related metrics or at the log file.
For checking the number of connections, one can look at the statistics, use the server generated log file or leverage Linux system tools such as
client_connections metric keeps track of all incoming client connections to an Aerospike node.
Other connections that are used for inter-node connectivity are
watch directive can be used within asadm:
Admin> watch show stat like conn [ 2020-10-28 20:38:58 'show stat like conn' sleep: 2.0s iteration: 1 ] ~Service Statistics (2020-10-28 20:38:58 UTC)~ NODE : 9f24c3755080:3000 client_connections : 5 fabric_connections : 0 heartbeat_connections: 0
Aerospike Server Logs
Connections can be monitored by tailing the Aerospike logs:
tail -f /var/log/aerospike.log | grep proto Oct 28 2020 20:53:04 GMT: INFO (info): (ticker.c:322) fds: proto (0,22,22) heartbeat (0,0,0) fabric (0,0,0) Oct 28 2020 20:53:14 GMT: INFO (info): (ticker.c:322) fds: proto (0,25,25) heartbeat (0,0,0) fabric (0,0,0)
The description of this log line can be found in the log reference manual. In the above example, one notices that there are no active connections but there were 3 connections opened and then closed in the 10 second interval.
Aerospike Monitoring Stack
The Aerospike Monitoring Stack provides dashboards showing a large variety of metrics…
The previously mentioned metrics are generated by the Aerospike Database. In order to view open sockets at the operating system level,
netstat is a good tool to use. For example:
For issues causing a high number of client connections, refer to the High Client Connections in Aerospike article.
For details on OS open files or
proto-fd-maxtuning, check the article titled Understanding
PROTO CLIENT CONNECTIONS TRAFFIC