I’m running 2 OpenNebula VMs with CentOS.
I’ve installed CE 3.5.3 on both nodes, and AMC and the benchmarks on the 1st node. edit: Is the CE restricted to a single node? Is it something that basic?
I can add the 2nd node in AMC no problem. After adding, I can run_benchmarks and get confirmation that both nodes are reachable:
2015-03-02 20:14:16.410 INFO Thread 1 Add node BB98405080A0002 127.0.0.1:3000 2015-03-02 20:14:16.425 INFO Thread 1 Add node BB98705080A0002 10.8.5.135:3000 2015-03-02 20:14:16.500 write(tps=16 timeouts=0 errors=0) read(tps=85 timeouts=0 errors=0) total(tps=101 timeouts=0 err
But soon after, the 2nd node doesn’t get added any more:
2015-03-02 21:08:06.728 INFO Thread 1 Add node BB98405080A0002 127.0.0.1:3000 2015-03-02 21:08:06.786 write(tps=29 timeouts=0 errors=0) read(tps=123 timeouts=0 errors=0) total(tps=152 timeouts=0 errors=0)
AMC shows both nodes as up and green Cluster Visibility for some time after that, but then they both show as read. Even while run_benchmarks still works, both nodes stay red.
If I issue an aerospike service restart on the 2nd node, it rejoins, even in the middle of a run_benchmarks:
2015-03-02 22:06:30.988 write(tps=13364 timeouts=0 errors=0) read(tps=13442 timeouts=0 errors=0) total(tps=26806 timeouts=0 errors=0) 2015-03-02 22:06:31.914 INFO Thread 8 Add node BB98705080A0002 10.8.5.135:3000 2015-03-02 22:06:31.988 write(tps=5211 timeouts=0 errors=0) read(tps=5275 timeouts=0 errors=0) total(tps=10486 timeouts=0 errors=0)
And then the 2 nodes both go green in AMC.
How can I get more detail about why the 2nd node is dropping out? What errors should I normally expect from AMC about that event?