When I start Aerospike, after it loads data, why does CPU use remain high?


#1

hi, I restart a node and found that when loaded 69% data, it look like stop loading data,

Dec 25 2014 14:10:12 GMT: INFO (drv_ssd): (drv_ssd.c::4109) {cmmapping} loaded 1476631870 records, /dev/sdb1 69%, /dev/sdc1 69%
Dec 25 2014 14:10:14 GMT: INFO (drv_ssd): (drv_ssd.c::4109) {cmmapping} loaded 1476614240 records, /dev/sdb1 69%, /dev/sdc1 69%
Dec 25 2014 14:10:16 GMT: INFO (drv_ssd): (drv_ssd.c::4109) {cmmapping} loaded 1476597724 records, /dev/sdb1 69%, /dev/sdc1 69%
Dec 25 2014 14:10:16 GMT: INFO (drv_ssd): (drv_ssd.c::2388) device /data/aerospike/data/base.dat: used 0, contig-free 16383M (16383 wblocks), swb-free 0, n-w 0, w-q 0 w-tot 0 (0.0/s), defrag-q 0 defrag-tot 0 (0.0/s)
Dec 25 2014 14:10:16 GMT: INFO (drv_ssd): (drv_ssd.c::2388) device /data/aerospike/data/disk-base.dat: used 0, contig-free 102399M (102399 wblocks), swb-free 0, n-w 0, w-q 0 w-tot 0 (0.0/s), defrag-q 0 defrag-tot 0 (0.0/s)
Dec 25 2014 14:10:18 GMT: INFO (drv_ssd): (drv_ssd.c::4109) {cmmapping} loaded 1476580610 records, /dev/sdb1 69%, /dev/sdc1 69%
Dec 25 2014 14:10:20 GMT: INFO (drv_ssd): (drv_ssd.c::4109) {cmmapping} loaded 1476562786 records, /dev/sdb1 69%, /dev/sdc1 69%
Dec 25 2014 14:10:22 GMT: INFO (drv_ssd): (drv_ssd.c::4109) {cmmapping} loaded 1476545617 records, /dev/sdb1 69%, /dev/sdc1 69%
Dec 25 2014 14:10:24 GMT: INFO (drv_ssd): (drv_ssd.c::4109) {cmmapping} loaded 1476528120 records, /dev/sdb1 69%, /dev/sdc1 69%
Dec 25 2014 14:10:26 GMT: INFO (drv_ssd): (drv_ssd.c::4109) {cmmapping} loaded 1476510192 records, /dev/sdb1 69%, /dev/sdc1 69%

but CPU use is very high.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2120 root      20   0 92.7g  89g 1184 S 1594.4 35.4 203:53.31 /usr/bin/asd

what is aerospike doing at this moment ?


#2

You can see that Aerospike is making progress loading the data from /dev/sdc1 and /dev/sdb1 by observing that the number of loaded records is increasing.

Basically the server is cold starting (as opposed to warm starting aka fast restart). Cold starting is the process of rebuilding the in-memory index by reading the data from the disk. Fast restart is not available in community releases and allows Aerospike to reconnect to the index created by the previous run as long as the following conditions are met:

  1. The user did not specify a coldstart.
  2. The process was shut down normally.
  3. The namespace being loaded did not store data in memory. (In aerospike.conf data-in-memory is not set and the namespace’s storage-engine is device.