Aerospike start is stuck and found cryptocurrency miner script

1

I am getting this repeatedly. I noticed the CPU utilization is unnecessarily high and checked my crontab -e and found this.

Screenshot 2020-09-21 at 10.03.16 AM
Screenshot 2020-09-21 at 10.03.16 AM815×20 5.7 KB

Searched about this kdevtmpfsi using 100% of CPU? Here is how you can fix that. | by Anand S | Medium

This comes very frequently and makes the CPU utilization 100%.

I deleted the line from crontab and restarted my system and after this unable to use aql.

Seed:         127.0.0.1
User:         None
Config File:  /etc/aerospike/astools.conf /home/ec2-user/.aerospike/astools.conf 
2020-09-21 04:46:41 WARN Failed to connect to seed 127.0.0.1 3000. AEROSPIKE_ERR_CONNECTION Socket write error: 111, 127.0.0.1:3000
Error -10: Failed to connect

And whenever I try to start it is getting stuck.

[ec2-user@ip-172-31-25-49 ~]$ sudo service aerospike start
Starting and checking aerospike:

Logs:

Sep 21 2020 04:33:59 GMT: INFO (info): (ticker.c:667) {prod} scan: basic (8669,0,0) aggr (0,0,0) udf-bg (0,0,0)
Sep 21 2020 04:33:59 GMT: INFO (info): (ticker.c:692) {prod} query: basic (85948,0) aggr (0,0) udf-bg (0,0)
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-read (96547 total) msec
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:266)  (00: 0000096546) (05: 0000000001)
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-write (109019 total) msec
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:266)  (00: 0000109016) (01: 0000000001) (04: 0000000002)
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-query (85948 total) msec
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:257)  (00: 0000085823) (01: 0000000014) (02: 0000000020) (03: 0000000039)
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:266)  (04: 0000000048) (05: 0000000003) (06: 0000000001)
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-query-rec-count (13032 total) count
Sep 21 2020 04:33:59 GMT: INFO (info): (hist.c:266)  (01: 0000004743) (02: 0000008037) (03: 0000000252)
Sep 21 2020 04:33:59 GMT: WARNING (socket): (socket.c:746) (repeated:40) Timeout while connecting
Sep 21 2020 04:33:59 GMT: WARNING (hb): (hb.c:4845) (repeated:40) could not create heartbeat connection to node {10.10.1.193:3002}
Sep 21 2020 04:33:59 GMT: WARNING (socket): (socket.c:814) (repeated:40) Error while connecting socket to 10.10.1.193:3002
Sep 21 2020 04:34:05 GMT: INFO (nsup): (thr_sindex.c:493) {prod} sindex-gc start
Sep 21 2020 04:34:06 GMT: INFO (nsup): (thr_sindex.c:524) {prod} sindex-gc: Processed: 37913, found:0, deleted: 0: Total time: 912 ms
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:171) NODE-ID bb9c068d655bf02 CLUSTER-SIZE 1
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:247)    cluster-clock: skew-ms 0
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:277)    system-memory: free-kbytes 7915808 free-pct 96 heap-kbytes (2187029,2189192,2279424) heap-efficiency-pct 95.9
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:291)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:313)    fds: proto (5,937370,937365) heartbeat (0,0,0) fabric (0,12,12)
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:322)    heartbeat-received: self 0 foreign 0
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:353)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:377)    early-fail: demarshal 0 tsvc-client 7470 tsvc-batch-sub 0 tsvc-udf-sub 0
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:408) {test} objects: all 1 master 1 prole 0 non-replica 0
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:469) {test} migrations: complete
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:488) {test} memory-usage: total-bytes 99 index-bytes 64 sindex-bytes 0 data-bytes 35 used-pct 0.00
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:587) {test} client: tsvc (0,0) proxy (0,0,0) read (0,0,0,2) write (1,0,0) delete (0,0,0,1) udf (0,0,0) lang (0,0,0,0)
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:667) {test} scan: basic (0,0,0) aggr (0,0,0) udf-bg (1,0,0)
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:720) {test} udf-sub: tsvc (0,0) udf (1,0,0) lang (1,0,0,0)
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:240) histogram dump: {test}-read (2 total) msec
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:266)  (00: 0000000002)
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (1 total) msec
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:266)  (00: 0000000001)
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:408) {prod} objects: all 37013 master 37013 prole 0 non-replica 0
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:469) {prod} migrations: complete
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:488) {prod} memory-usage: total-bytes 17061292 index-bytes 2368832 sindex-bytes 1881294 data-bytes 12811166 used-pct 0.40
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:518) {prod} device-usage: used-bytes 13703648 avail-pct 99
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:587) {prod} client: tsvc (0,0) proxy (0,0,0) read (79570,0,0,16977) write (109019,0,0) delete (23421,0,0,13) udf (0,0,0) lang (0,0,0,0)
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:667) {prod} scan: basic (8669,0,0) aggr (0,0,0) udf-bg (0,0,0)
Sep 21 2020 04:34:09 GMT: INFO (info): (ticker.c:692) {prod} query: basic (85948,0) aggr (0,0) udf-bg (0,0)
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-read (96547 total) msec
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:266)  (00: 0000096546) (05: 0000000001)
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-write (109019 total) msec
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:266)  (00: 0000109016) (01: 0000000001) (04: 0000000002)
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-query (85948 total) msec
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:257)  (00: 0000085823) (01: 0000000014) (02: 0000000020) (03: 0000000039)
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:266)  (04: 0000000048) (05: 0000000003) (06: 0000000001)
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-query-rec-count (13032 total) count
Sep 21 2020 04:34:09 GMT: INFO (info): (hist.c:266)  (01: 0000004743) (02: 0000008037) (03: 0000000252)
Sep 21 2020 04:34:09 GMT: WARNING (socket): (socket.c:746) (repeated:39) Timeout while connecting
Sep 21 2020 04:34:09 GMT: WARNING (hb): (hb.c:4845) (repeated:39) could not create heartbeat connection to node {10.10.1.193:3002}
Sep 21 2020 04:34:09 GMT: WARNING (socket): (socket.c:814) (repeated:39) Error while connecting socket to 10.10.1.193:3002
Sep 21 2020 04:34:15 GMT: INFO (nsup): (thr_sindex.c:493) {prod} sindex-gc start
Sep 21 2020 04:34:16 GMT: INFO (nsup): (thr_sindex.c:524) {prod} sindex-gc: Processed: 37913, found:0, deleted: 0: Total time: 1014 ms
Sep 21 2020 04:34:19 GMT: INFO (drv_ssd): (drv_ssd.c:2134) {prod} /opt/aerospike/data/prod.dat: used-bytes 13703232 free-wblocks 16351 write-q 0 write (43,0.0) defrag-q 0 defrag-read (54,0.0) defrag-write (4,0.0)
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:171) NODE-ID bb9c068d655bf02 CLUSTER-SIZE 1
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:247)    cluster-clock: skew-ms 0
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:277)    system-memory: free-kbytes 8000844 free-pct 97 heap-kbytes (2187028,2189192,2279424) heap-efficiency-pct 95.9
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:291)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:313)    fds: proto (4,937370,937366) heartbeat (0,0,0) fabric (0,12,12)
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:322)    heartbeat-received: self 0 foreign 0
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:353)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:377)    early-fail: demarshal 0 tsvc-client 7470 tsvc-batch-sub 0 tsvc-udf-sub 0
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:408) {test} objects: all 1 master 1 prole 0 non-replica 0
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:469) {test} migrations: complete
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:488) {test} memory-usage: total-bytes 99 index-bytes 64 sindex-bytes 0 data-bytes 35 used-pct 0.00
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:587) {test} client: tsvc (0,0) proxy (0,0,0) read (0,0,0,2) write (1,0,0) delete (0,0,0,1) udf (0,0,0) lang (0,0,0,0)
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:667) {test} scan: basic (0,0,0) aggr (0,0,0) udf-bg (1,0,0)
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:720) {test} udf-sub: tsvc (0,0) udf (1,0,0) lang (1,0,0,0)
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:240) histogram dump: {test}-read (2 total) msec
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:266)  (00: 0000000002)
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:240) histogram dump: {test}-write (1 total) msec
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:266)  (00: 0000000001)
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:408) {prod} objects: all 37012 master 37012 prole 0 non-replica 0
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:469) {prod} migrations: complete
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:488) {prod} memory-usage: total-bytes 17060823 index-bytes 2368768 sindex-bytes 1881294 data-bytes 12810761 used-pct 0.40
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:518) {prod} device-usage: used-bytes 13703232 avail-pct 99
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:587) {prod} client: tsvc (0,0) proxy (0,0,0) read (79570,0,0,16979) write (109021,0,0) delete (23423,0,0,13) udf (0,0,0) lang (0,0,0,0)
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:667) {prod} scan: basic (8669,0,0) aggr (0,0,0) udf-bg (0,0,0)
Sep 21 2020 04:34:19 GMT: INFO (info): (ticker.c:692) {prod} query: basic (85948,0) aggr (0,0) udf-bg (0,0)
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-read (96549 total) msec
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:266)  (00: 0000096548) (05: 0000000001)
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-write (109021 total) msec
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:266)  (00: 0000109018) (01: 0000000001) (04: 0000000002)
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-query (85948 total) msec
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:257)  (00: 0000085823) (01: 0000000014) (02: 0000000020) (03: 0000000039)
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:266)  (04: 0000000048) (05: 0000000003) (06: 0000000001)
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:240) histogram dump: {prod}-query-rec-count (13032 total) count
Sep 21 2020 04:34:19 GMT: INFO (info): (hist.c:266)  (01: 0000004743) (02: 0000008037) (03: 0000000252)
Sep 21 2020 04:34:19 GMT: WARNING (socket): (socket.c:746) (repeated:39) Timeout while connecting
Sep 21 2020 04:34:19 GMT: WARNING (hb): (hb.c:4845) (repeated:39) could not create heartbeat connection to node {10.10.1.193:3002}
Sep 21 2020 04:34:19 GMT: WARNING (socket): (socket.c:814) (repeated:39) Error while connecting socket to 10.10.1.193:3002
Sep 21 2020 04:34:20 GMT: WARNING (socket): (socket.c:720) Error while connecting: 101 (Network is unreachable)
Sep 21 2020 04:34:20 GMT: WARNING (info): (thr_info.c:4626) No network interface addresses detected for client access
Sep 21 2020 04:34:22 GMT: INFO (as): (signal.c:194) SIGTERM received, starting normal shutdown
Sep 21 2020 04:34:22 GMT: INFO (storage): (storage.c:702) initiating storage shutdown ...
Sep 21 2020 04:34:22 GMT: INFO (storage): (storage.c:703) flushing data to storage ...
Sep 21 2020 04:34:22 GMT: INFO (storage): (storage.c:722) completed flushing to storage
Sep 21 2020 04:34:22 GMT: INFO (as): (as.c:445) finished clean shutdown - exiting

Please help me as I am getting this (cryptocurrency miner script) very frequently.

2

System was likely compromised by vulnerability in our LUA sandbox : CVE-2020-13151. If you aren’t using UDFs, they can be disabled in the latest versions (starting with 5.1.0.6).

  1. What version of Aerospike are you running?
  2. I’d consider this host compromised, I suggest doing a full system wipe and rebuild.
  3. Restrict network access to your servers, see iptables. This should be standard operating procedures.
3

I am running Aerospike Community Edition build 4.2.0.10.

4

You should absolutely secure access to your database ports. Here is an article that may help: How To secure Aerospike database servers.

Other than that, you should also consider upgrading to a recent version with the ability to disable UDF execution, through the disable-udf-execution command.

1 Like
5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.