How to change log levels in Aerospike
Context
Aerospike can report multiple levels of log event into the log file. Greater verbosity of log messages gives greater detail on what is going on within the cluster however it also increases the size of log files and makes it more time consuming to read through the log files verbatim. The default log level is INFO which provides a generally acceptable balance between detail and usability. Sometimes it may be necessary to see things at a greater detail or reduce logging message for brevity, this article discusses how this may be done.
Method
Aerospike log settings are dynamic which means they can be set without a node or cluster restart using asinfo. The following log levels are permissable:
- CRITICAL
- WARNING
- INFO
- DEBUG
- DETAIL
Additional granularity is provided as different Aerospike components can have different logging levels. A list of available components can be obtained as follows. The following list is for Aerospike version 3.16.0.1.
$ asinfo -v "log/" -l
misc:INFO
alloc:INFO
arenax:INFO
hardware:INFO
msg:INFO
rbuffer:INFO
socket:INFO
tls:INFO
vmapx:INFO
aggr:INFO
as:INFO
batch:INFO
bin:INFO
config:INFO
clustering:INFO
compression:INFO
demarshal:INFO
drv_ssd:INFO
exchange:INFO
fabric:INFO
geo:INFO
hb:INFO
hlc:INFO
index:INFO
info:INFO
info-port:INFO
job:INFO
migrate:INFO
mon:INFO
namespace:INFO
nsup:INFO
particle:INFO
partition:INFO
paxos:INFO
predexp:INFO
proto:INFO
proxy:INFO
proxy-divert:INFO
query:INFO
record:INFO
rw:INFO
rw-client:INFO
scan:INFO
security:INFO
sindex:INFO
smd:INFO
storage:INFO
truncate:INFO
tsvc:INFO
udf:INFO
xdr:INFO
The above list may change for other versions.
3.16.0.1
Starting server version 3.16.0.1, we introduced some key components and improved logging for important errors. Example:
- To identify record going above configured write-block-size, you can enable details level logging for ‘rw’ and ‘drv_ssd’ context.
- To identify hot-keys or potential rogue clients hitting Aerospike servers, a new context ‘rw-client’ is now added which can be set to details to show all client transactions details (including namespace, client IP and the type of transactions such as read, write, batch or UDF).
- To troubleshoot proxy issues, a new context ‘proxy-divert’ is now added which can be set to detail to show all proxy divert attempts including namespace, client IP, destination node and if it’s a batch read transaction.
3.14.1.2
Does not have these contexts: jem
The syntax to change a log level for all components other than XDR is as follows:
asinfo -v "log-set:id=0;<Component1>=<LogLevel>"
The example below shows how to change UDF logging to DETAIL
asinfo -h 127.0.0.1 -p 3000 -v "log-set:id=0;udf=detail"
requested value log-set:id=0;udf=detail
value is ok
The logging level is switched back to the default INFO level as shown:
asinfo -h 127.0.0.1 -p 3000 -v "log-set:id=0;udf=info"
requested value log-set:id=0;udf=info
value is ok
3.13
As of 3.13 for some context names the leading “cf:” has been dropped. In comparison with 3.13.0.3 the following versions have a few contexts included and dropped.
3.12.1
Does not have these contexts: hardware, clustering, exchange
Incudes these contexts that are dropped as on 3.13: drv_kv
3.11.1.1
Does not have these contexts: hardware, clustering, exchange, predexp, truncate
Incudes these contexts that are dropped as on 3.13: drv_kv
From Aerospike 3.8.x onwards it is not necessary to specify the XDR port and so the format of the command would be:
asinfo -v "log-set:id=0;xdr=log_level"
The following command would set the XDR log level to WARNING_ in Aerospike 3.8 and higher.
asinfo -v "log-set:id=0;xdr=warning"
The XDR component uses slightly different syntax. This is shown below for Aerospike releases prior to 3.8.x:
asinfo -p XDR_port -v "log-set:xdr=log_level"
As an example, to switch XDR into WARNING log level on a system where XDR uses the default port of 3004, the following command would be used:
asinfo -p 3004 -v "set-log:xdr=warning"
Notes
- The asadm tool now has a log analyser component. Logs can be loaded in to the log analyser (including logs for multiple nodes) and searched for useful information such as search strings, histogram data etc. The tool is available at:
- This topic is covered in the documentation here:
http://www.aerospike.com/docs/reference/serverlogmessages/#changing-log-levels
Keywords
LOG LEVEL XDR DETAIL INFO WARNING CRITICAL DEBUG
Timestamp
04/25/2018