How do I monitor client traffic and connections to Aerospike?

Knowledge Base Operating
1

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

FAQ – How do I monitor incoming client traffic and connections to Aerospike?

Context

Monitoring client traffic and connections might be required in the following situations or for general review of a cluster.

  1. To confirm whether traffic has moved away from a node which was quiesced.
  2. To check if there is high incoming and/or churning client connections impacting applications.
  3. To check client conectivity during general testing.

The Aerospike DB has a client connections upper limit of proto-fd-max per node, beyond which incoming connections will be rejected. Thus, it is essential to monitor the client connections and configure a cluster appropriately. This article will cover monitoring recommendations.

Verify incoming traffic

A quick and easy way to check for basic read or write traffic on an Aerospike cluster is to use the ‘show latencies’ command from asadm:

Admin> show latencies
                read Latency (2018-10-25 23:49:19 UTC
                Node       Ops/Sec   %>1ms   %>8ms   %>64ms
aero-cluster1_1:3000           9.3     0.0     0.0      0.0
aero-cluster1_2:3000           0.0     0.0     0.0      0.0
aero-cluster1_3:3000           7.5     0.0     0.0      0.0
aero-cluster1_4:3000           8.4     0.0     0.0      0.0
Number of rows: 1

                write Latency (2018-10-25 23:49:19 UTC
                Node        Ops/Sec   %>1ms   %>8ms   %>64ms
aero-cluster1_1:3000            2.0     0.0     0.0      0.0
aero-cluster1_2:3000            0.0     0.0     0.0      0.0
aero-cluster1_3:3000            3.2     0.0     0.0      0.0
aero-cluster1_4:3000            4.5     0.0     0.0      0.0
Number of rows: 1

The Ops/Sec column indicates the throughput. A throughput of 0 would obviously mean the node is not receiving any transaction of the given type and could imply connectivity issues with the clients. In the example above, a 4 node cluster is showing a low read/write throughput on 3 out of the 4 nodes. The show latencies command would also show traffic for batch and secondary index queries. For scans, one would have to look at the scan related metrics or at the log file.

For checking the number of connections, one can look at the statistics, use the server generated log file or leverage Linux system tools such as netstat.

Aerospike Metrics

The client_connections metric keeps track of all incoming client connections to an Aerospike node. Other connections that are used for inter-node connectivity are heartbeat_connections and fabric_connections.

The watch directive can be used within asadm:

Admin> watch show stat like conn
[ 2020-10-28 20:38:58 'show stat like conn' sleep: 2.0s iteration: 1 ]
~Service Statistics (2020-10-28 20:38:58 UTC)~
NODE                 :   9f24c3755080:3000   
client_connections   :   5                   
fabric_connections   :   0                   
heartbeat_connections:   0

Aerospike Server Logs

Connections can be monitored by tailing the Aerospike logs:

tail -f /var/log/aerospike.log | grep proto
Oct 28 2020 20:53:04 GMT: INFO (info): (ticker.c:322)    fds: proto (0,22,22) heartbeat (0,0,0) fabric (0,0,0)
Oct 28 2020 20:53:14 GMT: INFO (info): (ticker.c:322)    fds: proto (0,25,25) heartbeat (0,0,0) fabric (0,0,0)

The description of this log line can be found in the log reference manual. In the above example, one notices that there are no active connections but there were 3 connections opened and then closed in the 10 second interval.

Aerospike Monitoring Stack

The Aerospike Monitoring Stack provides dashboards showing a large variety of metrics…

Using netstat

The previously mentioned metrics are generated by the Aerospike Database. In order to view open sockets at the operating system level, netstat is a good tool to use. For example:

netstat -antp

Notes

Keywords

PROTO CLIENT CONNECTIONS TRAFFIC

Timestamp

October 2020