Can t find macro-benchmarks query latency

Operations Monitoring
1

Hi,

I lost my ns-query benchmark today, I can not have my query latency :

 # asloglatency -N ns -h query 

 Histogram : {ns}-query
 Log       : /var/log/aerospike/aerospike.log 
 From      : 2018-11-08 15:34:39
 can't find histogram query from start time tail

# asinfo -h localhost -v 'latency:' -l 
{ns}-read:12:31:44-GMT,ops/sec,>1ms,>8ms,>64ms
12:31:54,267610.3,0.00,0.00,0.00
{ns}-write:12:31:44-GMT,ops/sec,>1ms,>8ms,>64ms
12:31:54,20798.4,0.00,0.00,0.00
error-no-data-yet-or-back-too-small
error-no-data-yet-or-back-too-small

It s strange I did not change anything recently…

I try to re enable it but it still does not work :frowning:

# asinfo -v 'set-config:context=namespace;id=ns;enable-benchmarks-read=true' 
ok
2

The read, write, and query (as well as udf, proxy, and re-repl) histograms are macro level histograms that are always ‘enabled’; however, if there haven’t been any data points collected for a particular macro level histogram, it will be omitted from the logs.

By enabling ‘enable-benchmarks-read’, you have enabled microbenchmarks for read transactions. You can find more information about the servers benchmarks here: https://www.aerospike.com/docs/operations/monitor/latency/index.html.

Histograms are only maintained during the life of the Aerospike server process, if you restarted Aerospike, then you will need to allow the histogram to allow ~20 seconds for the histograms to accumulate data.

3

Thank you for your reply, much appreciated, I’m still discovering Aerospike.

After restarting Aero server and waiting 30min, I still don’t have any query latency:

error-no-data-yet-or-back-too-small

It’s strange, because I have lots of operations running and my namespace write and read op/s are good.

I m using Aerospike 3.14.1.2

Should I check something else?

Thanks

4

Are you performing any queries? If not then there will not be any data.

asadm -e "show stat like query_reqs"

Here, queries do not refer to regular gets/puts, they refer to lookups on a secondary index.

5

Yes,

asadm -e "show stat query_reqs"
.... 
query_reqs                            :   6920979 
objects                               :   32109698

. etc…

I notice this in my aero log, don’t know if it is related :

Nov 09 2018 09:44:53 GMT: WARNING (particle): (particle_list.c:1954) OP_LIST_GET_RANGE: index 0 out of bounds for ele_count 0

Nov 09 2018 09:44:53 GMT: WARNING (rw): (read.c:400) {ns} read_local: failed as_bin_cdt_read_from_client() <Digest>:0x975560e9f93076dcbf5f7a029be21fdb5f7078c8
6

Sorry my asadm command was missing a “like” keyword.

Well clearly you have been issuing queries, assuming they weren’t all failing:

asadm -e "show stat like query_fail query_reqs"

If you are issuing queries and they are completing successfully, I see no reason from the code’s perspective that the histogram wouldn’t be logged (unless you have changed your ticker-interval to be disabled or very largs).

asadm -e "show config like ticker"