Unexpected behavior on EC2

Operations Installation
1

by amdalal » Thu Oct 10, 2013 8:51 pm

Hello all,

In my test setup on EC2 I have done following:

  1. One Aerospike server is running in ZoneA (say Aerospike-A).

  2. Another node of the same cluster is running in ZoneB (say Aerospike-B).

  3. The application using above cluster is running in ZoneA again.

  4. I am initializing AerospikeClinet like this:

    CODE: SELECT ALL hosts = new Host; hosts[0] = new Host(PUBLIC_IP_OF_AEROSPIKE-A, 3000); AerospikeClinet clinet = new AerospikeClient (policy, hosts);

With above setup I am getting below behavior:

  1. Writes are happening on both Aerospike-A and Aerospike-B.

  2. Reads are only happening on Aerospike-A (data is around 1million records, occupying 900MB of memory and 1.3 GB of disk)

    Question: Why are reads not going to both the nodes?

  3. If I take Aerospike-B down, everything works perfectly. There is no outage.

  4. If I take Aerospike-A down, all the writes and reads start failing. I’ve waited for 5 mins for other node to take traffic but it didn’t work.

    Questions:

    1. In above scenario, I would expect Aerospike-B to take all the traffic. But this is not happening. Is there anything I am doing wrong?
    2. Should I be giving both the hosts while initializing the client?
    3. I had executed “clinfo -v ‘config-set:context=service;paxos-recovery-policy=auto-dun-all’” on both the nodes. Is that creating a problem?
2

by devops01 » Thu Jul 31, 2014 3:26 pm

Could share your full code for testing. Also is the client running on same machine as Node A?

In a normal case an Aerospike 2 node Mesh configured cluster would form and handle both read and writes. Running auto-dun may have also been a factor.

best, Lucien