AEROSPIKE_ERR_CLIENT Node BB9A0AEAE005452 127.0.0.1:3000 is not yet fully initialized

Operations Installation
1

I have just installed aerospike-server-community-3.16.0.6-ubuntu16.04. Aerospike starts successfully but AQL throws error

Failed to connect to seed 127.0.0.1 3000. AEROSPIKE_ERR_CLIENT Node BB9A0AEAE005452 127.0.0.1:3000 is not yet fully initialized

I have found following information on log

Jul 02 2018 06:21:18 GMT: INFO (clustering): (clustering.c:6281) join request timed out for principal bb9c01ee53e16fa Jul 02 2018 06:21:18 GMT: INFO (clustering): (clustering.c:5997) sent cluster join request to bb9c01ee53e16fa Jul 02 2018 06:21:22 GMT: INFO (clustering): (clustering.c:6281) join request timed out for principal bb9c01ee53e16fa Jul 02 2018 06:21:22 GMT: INFO (clustering): (clustering.c:5997) sent cluster join request to bb9c01ee53e16fa

I have started aerospike with defult settings

2

How many nodes in your cluster? What does your config look like? Do the messages that you posted continue? Why are you starting on version 3.16 instead of 4.2? Can you paste the last few lines of the log? It looks like you have multiple nodes that are trying to form a cluster and failing.

3

Its single node cluster and has default configuration when installed. I have changed only log to file. I need to use multiple namespace so I have used 3.X version. Previously I was running the aerospike without error for almost 4 months with single cluster. I had single namespace with both in memory and on disk. I used HDD as file. Few days ago I tried to backup namespace data by coping file on different location and after few days I tried to restore the same file. Since then there is problem. So I reinstall the aerospike the problem was still there and finally I did OS reinstall and made it fresh and tried to install aerospike and still same.

Every time I do aerospike reinstall I get same node id?

 03 2018 03:55:29 GMT: INFO (info): (ticker.c:420) {bar} migrations: complete
Jul 03 2018 03:55:29 GMT: INFO (info): (ticker.c:439) {bar} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jul 03 2018 03:55:30 GMT: INFO (clustering): (clustering.c:6281) join request timed out for principal bb9c01ee53e16fa
Jul 03 2018 03:55:30 GMT: INFO (clustering): (clustering.c:5997) sent cluster join request to bb9c01ee53e16fa
Jul 03 2018 03:55:35 GMT: INFO (clustering): (clustering.c:6281) join request timed out for principal bb9c01ee53e16fa
Jul 03 2018 03:55:35 GMT: INFO (clustering): (clustering.c:5997) sent cluster join request to bb9c01ee53e16fa
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:166) NODE-ID bb9a0aeae005452 CLUSTER-SIZE 0
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:245)    system-memory: free-kbytes 30603216 free-pct 92 heap-kbytes (2911673,2912520,4415488) heap-efficiency-pct 65.9
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:259)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:281)    fds: proto (0,133,133) heartbeat (0,0,0) fabric (12,7332,7320)
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:290)    heartbeat-received: self 517420 foreign 517383
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:320)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:375) {test} objects: all 0 master 0 prole 0 non-replica 0
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:420) {test} migrations: complete
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:439) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:375) {bar} objects: all 0 master 0 prole 0 non-replica 0
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:420) {bar} migrations: complete
Jul 03 2018 03:55:39 GMT: INFO (info): (ticker.c:439) {bar} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jul 03 2018 03:55:40 GMT: INFO (clustering): (clustering.c:6281) join request timed out for principal bb9c01ee53e16fa
Jul 03 2018 03:55:40 GMT: INFO (clustering): (clustering.c:5997) sent cluster join request to bb9c01ee53e16fa
Jul 03 2018 03:55:44 GMT: INFO (clustering): (clustering.c:6281) join request timed out for principal bb9c01ee53e16fa
Jul 03 2018 03:55:44 GMT: INFO (clustering): (clustering.c:5997) sent cluster join request to bb9c01ee53e16fa
Jul 03 2018 03:55:49 GMT: INFO (clustering): (clustering.c:6281) join request timed out for principal bb9c01ee53e16fa
Jul 03 2018 03:55:49 GMT: INFO (clustering): (clustering.c:5997) sent cluster join request to bb9c01ee53e16fa
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:166) NODE-ID bb9a0aeae005452 CLUSTER-SIZE 0
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:245)    system-memory: free-kbytes 30602444 free-pct 92 heap-kbytes (2911773,2912620,4415488) heap-efficiency-pct 65.9
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:259)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q 0 rw-hash 0 proxy-hash 0 tree-gc-q 0
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:281)    fds: proto (0,133,133) heartbeat (0,0,0) fabric (12,7332,7320)
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:290)    heartbeat-received: self 517487 foreign 517450
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:320)    fabric-bytes-per-second: bulk (0,0) ctrl (0,0) meta (0,0) rw (0,0)
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:375) {test} objects: all 0 master 0 prole 0 non-replica 0
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:420) {test} migrations: complete
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:439) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:375) {bar} objects: all 0 master 0 prole 0 non-replica 0
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:420) {bar} migrations: complete
Jul 03 2018 03:55:49 GMT: INFO (info): (ticker.c:439) {bar} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
4

You have two different node-ids appearing in the log. Heartbeat also seems to be chatting with another node. Could you share your configuration file?

5

I solved it now :). The problem was in same network there was another aerospike server running on same multicast. So it was trying to join. I changed port of multicast and it works now.

Thank You

1 Like
6

For future reference… the node id should always be the same if the machine configuration hasn’t changed. It’s just a hash of the network details like mac+ipaddr I believe… Also, " need to use multiple namespace so I have used 3.X version. " you can have multiple namespaces in 4.x version.