How to change log levels in Aerospike


#1

How to change log levels in Aerospike

Context

Aerospike can report multiple levels of log event into the log file. Greater verbosity of log messages gives greater detail on what is going on within the cluster however it also increases the size of log files and makes it more time consuming to read through the log files verbatim. The default log level is INFO which provides a generally acceptable balance between detail and usability. Sometimes it may be necessary to see things at a greater detail or reduce logging message for brevity, this article discusses how this may be done.

Method

Aerospike log settings are dynamic which means they can be set without a node or cluster restart using asinfo. The following log levels are permissable:

  • CRITICAL
  • WARNING
  • INFO
  • DEBUG
  • DETAIL

Additional granularity is provided as different Aerospike components can have different logging levels. A list of available components can be obtained as follows. The following list is for Aerospike version 3.16.0.1.

$ asinfo -v "log/" -l
misc:INFO
alloc:INFO
arenax:INFO
hardware:INFO
msg:INFO
rbuffer:INFO
socket:INFO
tls:INFO
vmapx:INFO
aggr:INFO
as:INFO
batch:INFO
bin:INFO
config:INFO
clustering:INFO
compression:INFO
demarshal:INFO
drv_ssd:INFO
exchange:INFO
fabric:INFO
geo:INFO
hb:INFO
hlc:INFO
index:INFO
info:INFO
info-port:INFO
job:INFO
migrate:INFO
mon:INFO
namespace:INFO
nsup:INFO
particle:INFO
partition:INFO
paxos:INFO
predexp:INFO
proto:INFO
proxy:INFO
proxy-divert:INFO
query:INFO
record:INFO
rw:INFO
rw-client:INFO
scan:INFO
security:INFO
sindex:INFO
smd:INFO
storage:INFO
truncate:INFO
tsvc:INFO
udf:INFO
xdr:INFO

The above list may change for other versions.

3.16.0.1

Starting server version 3.16.0.1, we introduced some key components and improved logging for important errors. Example:

  • To identify record going above configured write-block-size, you can enable details level logging for ‘rw’ and ‘drv_ssd’ context.
  • To identify hot-keys or potential rogue clients hitting Aerospike servers, a new context ‘rw-client’ is now added which can be set to details to show all client transactions details (including namespace, client IP and the type of transactions such as read, write, batch or UDF).
  • To troubleshoot proxy issues, a new context ‘proxy-divert’ is now added which can be set to detail to show all proxy divert attempts including namespace, client IP, destination node and if it’s a batch read transaction.

3.14.1.2

Does not have these contexts: jem

The syntax to change a log level for all components other than XDR is as follows:

asinfo -v "log-set:id=0;<Component1>=<LogLevel>" 

The example below shows how to change UDF logging to DETAIL

asinfo -h 127.0.0.1 -p 3000 -v "log-set:id=0;udf=detail"
requested value  log-set:id=0;udf=detail
value is  ok

The logging level is switched back to the default INFO level as shown:

asinfo -h 127.0.0.1 -p 3000 -v "log-set:id=0;udf=info"
requested value  log-set:id=0;udf=info
value is  ok

3.13

As of 3.13 for some context names the leading “cf:” has been dropped. In comparison with 3.13.0.3 the following versions have a few contexts included and dropped.

3.12.1

Does not have these contexts: hardware, clustering, exchange

Incudes these contexts that are dropped as on 3.13: drv_kv

3.11.1.1

Does not have these contexts: hardware, clustering, exchange, predexp, truncate

Incudes these contexts that are dropped as on 3.13: drv_kv

From Aerospike 3.8.x onwards it is not necessary to specify the XDR port and so the format of the command would be:

asinfo -v "log-set:id=0;xdr=log_level"

The following command would set the XDR log level to WARNING_ in Aerospike 3.8 and higher.

asinfo -v "log-set:id=0;xdr=warning"

The XDR component uses slightly different syntax. This is shown below for Aerospike releases prior to 3.8.x:

asinfo -p XDR_port -v "log-set:xdr=log_level"

As an example, to switch XDR into WARNING log level on a system where XDR uses the default port of 3004, the following command would be used:

asinfo -p 3004 -v "set-log:xdr=warning"

Notes

  • The asadm tool now has a log analyser component. Logs can be loaded in to the log analyser (including logs for multiple nodes) and searched for useful information such as search strings, histogram data etc. The tool is available at:
  • This topic is covered in the documentation here:

http://www.aerospike.com/docs/reference/serverlogmessages/#changing-log-levels

Keywords

LOG LEVEL XDR DETAIL INFO WARNING CRITICAL DEBUG

Timestamp

04/25/2018


Hot Key error code 14