How do I find occurrences of "batch_sub_read_not_found" in Aerospike logs

Operations Monitoring
1

Hello, I noticed a spike in the Datadog metric “aerospike.namespace.batch_sub_read_not_found” for an Aeropsike cluster. I was able to find the namespace and host id from the metric tags.

However, I cannot identify the same entries in Aerospike log.

Does Aerospike log occurrences of this metric (batch_sub_read_not_found) in the logs? If yes, what will the entries look like?

2

You wouldn’t be able to find those instances in the logs. The logs, by default, would only show the batch_sub_read_not_found statistic. Having said that, as of version 6.3, a new log context batch-sub was introduced which will log the type of batch sub transaction along with the digest and the originating client IP address at the DETAIL log level. You wouldn’t be able to differentiate not found from success, though. See example below from some quick code using the sandbox batch example and doing some write, some read and some read not found transactions by batches of 10 (each batch sub transaction will log a line so this should be used carefully):

May 29 2023 22:41:04 GMT: INFO (info): (ticker.c:778) {sandbox} batch-sub: tsvc (0,0) proxy (0,0,0) read (20,0,0,20,0) write (5040,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
May 29 2023 22:41:14 GMT: INFO (info): (ticker.c:778) {sandbox} batch-sub: tsvc (0,0) proxy (0,0,0) read (20,0,0,20,0) write (5040,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
May 29 2023 22:41:21 GMT: INFO (info): (thr_info.c:2298) log-set:id=0:batch-sub=detail
May 29 2023 22:41:24 GMT: INFO (info): (ticker.c:778) {sandbox} batch-sub: tsvc (0,0) proxy (0,0,0) read (20,0,0,20,0) write (5040,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 690f4b1eb7ef704dc79268a3b62fa9609280a2ea client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest d9d4879aa06629ea5fee339ccaae94c20b060d97 client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 74205cb1f55d2caf086153fac4e0690840cd0593 client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest dde37a768acf884a20b29904cbc641d35de24413 client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 4eed69ed39591cd2e1b0b519e775801be0068c0f client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 4828d97d81565ccf194eaf50dd87ce9d15545c3f client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest eb4d12f9866a93b7d94844f3a92064ef096a26e8 client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 8875f4f00330f6ebd645678c0c6c093c00494969 client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest cc975f0e2c40674050fba762690c73fd919570c4 client 127.0.0.1:48548 read
May 29 2023 22:41:27 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 6a1c823fe1123c0fe9050fa3771aa765693b8948 client 127.0.0.1:48548 read
May 29 2023 22:41:34 GMT: INFO (info): (ticker.c:778) {sandbox} batch-sub: tsvc (0,0) proxy (0,0,0) read (20,0,0,30,0) write (5040,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
May 29 2023 22:41:44 GMT: INFO (info): (ticker.c:778) {sandbox} batch-sub: tsvc (0,0) proxy (0,0,0) read (20,0,0,30,0) write (5040,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 7920441a64dcfa55a7fe17f44548d5c78d70fd47 client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 7b978614281b9fb3dad8e2cf319999013b23b41d client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 6aef8cee77b7aedb7f3069e7b58ef0375650cff5 client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 01af7511b7a03d27c16eab82366cc26a7c954f16 client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 902c9bc0a3c16135c0f25b2a77b310d0cdf3bdc8 client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest defafa1e6d01092a1b30bb7a1e24f899ef6e5a96 client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest e5857a49c10b7cae01fd38906af819f61801c32b client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 46b51ba2764650600b24b1d54dfeefa4d4ac6c42 client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest d95eef7a8f4d9ad93e0e8d7c613642001d5e1561 client 127.0.0.1:41094 read
May 29 2023 22:41:48 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest de57554c323d86067e1fc553902901cb7985c598 client 127.0.0.1:41094 read
May 29 2023 22:41:54 GMT: INFO (info): (ticker.c:778) {sandbox} batch-sub: tsvc (0,0) proxy (0,0,0) read (30,0,0,30,0) write (5040,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 7920441a64dcfa55a7fe17f44548d5c78d70fd47 client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 7b978614281b9fb3dad8e2cf319999013b23b41d client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 6aef8cee77b7aedb7f3069e7b58ef0375650cff5 client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 01af7511b7a03d27c16eab82366cc26a7c954f16 client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 902c9bc0a3c16135c0f25b2a77b310d0cdf3bdc8 client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest defafa1e6d01092a1b30bb7a1e24f899ef6e5a96 client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest e5857a49c10b7cae01fd38906af819f61801c32b client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest 46b51ba2764650600b24b1d54dfeefa4d4ac6c42 client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest d95eef7a8f4d9ad93e0e8d7c613642001d5e1561 client 127.0.0.1:41542 write
May 29 2023 22:41:56 GMT: DETAIL (batch-sub): (thr_tsvc.c:104) {sandbox} digest de57554c323d86067e1fc553902901cb7985c598 client 127.0.0.1:41542 write
May 29 2023 22:42:04 GMT: INFO (info): (ticker.c:778) {sandbox} batch-sub: tsvc (0,0) proxy (0,0,0) read (30,0,0,30,0) write (5050,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
May 29 2023 22:42:14 GMT: INFO (info): (ticker.c:778) {sandbox} batch-sub: tsvc (0,0) proxy (0,0,0) read (30,0,0,30,0) write (5050,0,0,0) delete (0,0,0,0,0) udf (0,0,0,0) lang (0,0,0,0)
1 Like
3

This topic was automatically closed 84 days after the last reply. New replies are no longer allowed.