Not able to get UDF latency metrics with CE 5.1.0.4 with prometheus exporter

Hello

I am trying to scrape UDF latencies for my aerospike server with Aerospike Prometheus Exporter. But for some reason, I am not getting any latency metrics except for read ops for only one of the namespace. If I run

asadm -e ‘asinfo -v “latency:” -l’

I get the response that I am looking for. Wondering why Prom exporter is not showing the same? Attaching APE.toml

   # labels to add to the prometheus metrics for e.g. labels={zone="asia-south1-a", platform="google compute engine"}
labels={}

bind = ":9145"

# metrics server timeout in seconds
timeout = 10

# Exporter logging configuration
# Log file path (optional, logs to console by default)
# Level can be info|warning,warn|error,err|debug|trace ('info' by default)
log_file = ""
log_level = ""

[Aerospike]
db_host="localhost"
db_port=3000

key_file_passphrase=""

# timeout for sending commands to the server node in seconds
timeout=5

# Metrics Allowlist - If specified, only these metrics will be scraped. An empty list will exclude all metrics.
# Commenting out the below allowlist configs will disable metrics filtering (i.e. all metrics will be scraped).

# Namespace metrics allowlist
# namespace_metrics_allowlist=[]

# Set metrics allowlist
# set_metrics_allowlist=[]

# Node metrics allowlist
# node_metrics_allowlist=[]

# XDR metrics allowlist (only for Aerospike versions 5.0 and above)
xdr_metrics_allowlist=[
"success",
"latency_ms",
"throughput",
"lap_us"
]

# Metrics Blocklist - If specified, these metrics will be NOT be scraped.

# Namespace metrics blocklist
# namespace_metrics_blocklist=[]

# Set metrics blocklist
# set_metrics_blocklist=[]

# Node metrics blocklist
# node_metrics_blocklist=[]

# XDR metrics blocklist (only for Aerospike versions 5.0 and above)
# xdr_metrics_blocklist=[]

I think thats because latency was deprecated and replaced with latencies in 5.1.x Info Command Reference | Aerospike Documentation

https://www.aerospike.com/download/server/notes.html

top line of release notes…:

Aerospike Server CE Release Notes
5.1.0.6
Release Date: August 17, 2020
For the info command 'latency' support has been deprecated and will be removed in the next major release.
Please use the new replacement info command 'latencies' to track all latency histograms.

@Albot Are you saying I need to change the ape.toml? Or you saying we need to change Aerospike-Prometheus-Exporter code to read from latencies?

The asadm command works as expected but I understand it will be deprecated.

Oh, I misunderstood. Anything in the exporter log? Can you share output from curl against the exporter vs latency output?

Exporter logs

INFO[0000] Welcome to Aerospike Prometheus Exporter!
INFO[0000] Loading configuration file etc/aerospike-prometheus-exporter/ape.toml
INFO[0000] Listening for Prometheus on: :9145

Latency output

$ asadm -e ‘asinfo -v “latency:” -l’

Seed:        [('127.0.0.1', 3000, None)]
Config_file: /home/team/.aerospike/astools.conf, /etc/aerospike/astools.conf
host:3000 (10.39.0.28) returned:
batch-index:
{serving}-read:
{serving}-write:
{serving}-udf:
{serving}-query:
{user}-read:14:01:22-GMT,ops/sec,>1ms,>8ms,>64ms
14:01:32,0.0,0.00,0.00,0.00
{user}-write:
{user}-udf:14:01:22-GMT,ops/sec,>1ms,>8ms,>64ms
14:01:32,1863.3,1.86,0.01,0.00
{user}-query:

host:3000 (10.39.0.24) returned:
batch-index:
{serving}-read:
{serving}-write:
{serving}-udf:
{serving}-query:
{user}-read:
{user}-write:
{user}-udf:14:01:18-GMT,ops/sec,>1ms,>8ms,>64ms
14:01:28,1827.2,1.44,0.01,0.00
{user}-query:

host:3000 (10.39.0.25) returned:
batch-index:
{serving}-read:
{serving}-write:
{serving}-udf:
{serving}-query:
{user}-read:
{user}-write:
{user}-udf:14:01:13-GMT,ops/sec,>1ms,>8ms,>64ms
14:01:23,1893.4,1.28,0.01,0.00
{user}-query:

host:3000 (10.39.0.22) returned:
batch-index:
{serving}-read:
{serving}-write:
{serving}-udf:
{serving}-query:
{user}-read:
{user}-write:
{user}-udf:14:01:19-GMT,ops/sec,>1ms,>8ms,>64ms
14:01:29,1907.3,1.25,0.01,0.00
{user}-query:

Can you curl localhost:9145/metrics then post output? Then the log from the exporter unless theres no change.

@Albot: attaching the output

metricsAS.txt.gz (13.3 KB)

Yeah… Seems like a bug.

Thanks @Gunjan_Sharma for reporting this.

The output of latency: info command has changed in 5.1 for when there’s no latency data. When there’s no latency data, instead of error-no-data-yet-or-back-too-small, aerospike now returns an empty output like this - batch-index:;{test}-read:;{test}-write:;{test}-udf:;{test}-query:.

Tracking this issue on GitHub - Latencies metrics not available · Issue #32 · aerospike/aerospike-prometheus-exporter · GitHub