Adding 2 nodes to the cluster and how to check whether 2 nodes are connected or not?

1

Hello,

Can some one help me to configure aerospike.conf file.

I have 2 linux ubuntu PC’s and installed aerospike in both. that means now i have 2 nodes.

Now I’m trying add these 2-nodes(192.168.239.1 and 192.168.239.3) to 1-cluster.

Now I’m running my application on node-1(192.168.239.1) and now my question is how to check whether my 2nodes added in cluster or not? if not could some help me to add those 2 nodes into the cluster.

kindly give clear steps to approach.

network {
        service {
                address any
                port 3000
                access-address 192.168.239.3
        }

        heartbeat {
                #mode multicast
                #multicast-group 239.1.99.222
                #port 9918

                mode mesh
                address 192.168.239.1
                port 3002

                mesh-seed-address-port 192.168.239.3 3002


                # To use unicast-mesh heartbeats, remove the 3 lines above, and see
                # aerospike_mesh.conf for alternative.

                interval 150
                timeout 10
        }

        fabric {
                port 3001
        }

        info {
                port 3003
        }
}
2

Hi Deva,

You can use the asadm tool to query the Aerospike cluster. It is part of the aerospike-tools package, which should already be installed when you install the Aerospike server.

In particular, if you use the “info” command, it will show you the connected nodes in the cluster. Refer the documentation at:

http://www.aerospike.com/docs/tools/asadm/user_guide/cluster_mode_guide#info

3

Dear shakthimaan,

Thanks for the quick response.

I tried to go through the above mentioned link, but unable to execute info command. I don’t even know where exactly I need to execute that command.

next I am new to this using Aerospike, how could I connect node to the cluster, could you please help me out from issue.

pls don’t get panic for asking such silly questions… Thanks in advance.

4

1366×768 99 KB

Dear Shakthimaan,

I tried to execute the info command and others as well, but it is showing only NODE1-id. How can I confirm that 2nd node is connected or not? if not pls guide me to add the node2 into the cluster.

5

Deva,

From the asadm output, we see that the other node is not connected to the cluster yet. From the screenshot, we see that you are using Windows.

  1. How are you running Ubuntu?

  2. Are you able to ping between the two machines? Open a terminal in the 192.168.239.1 and try to ping the 192.168.239.3 machine.

    $ ping -c3 192.168.239.3

You should atleast see three replies. Please paste the output of the ping command.

6

819×460 52.8 KB

Dear Shakthimaan,

please find the snapshot.

I am using putty to connect Linux pc(192.168.239.1) and from there im trying to add. please help to add other node.

7

Deva:

The two Ubuntu systems are able to ping each other, so it is not a network issue.

  1. Where and how are the Ubuntu systems running?

  2. Is there any firewall enabled, and does the Ubuntu machine have multiple network interfaces? Post output of the following from a terminal:

$ sudo iptables -L $ ifconfig -a

If the Ubuntu system has multiple interfaces, you can explicitly specify the node interface for the 192.168.239.x network in the service section of the Aerospike configuration file:

node-id-interface eth1

You should also refer http://www.aerospike.com/docs/operations/troubleshoot/startup.

8

Are you advertising the correct address-ports on each node? I think you have these mixed up. The config file you show seems to be for 192.168.239.3 (based on access-address that you are advertising). If so, its config should be: ( I am listening on 3002 for incoming nodes trying to join my cluster and I am going to reach out to 192.168.239.1:3002 to join an existing node/cluster.)

            mode mesh               
            port 3002

            mesh-seed-address-port 192.168.239.1 3002

BTW, you can also dynamicaly add one node to an existing/node cluster using the following command where aa.bb.cc.dd is ip address of the node you want to join in to and its listening on port 3002 for incoming nodes trying to join its cluster.

$asinfo -v "tip:host=aa.bb.cc.dd;port=3002"

So, just to be clear, in your case, you will execute this command on 192.168.239.3 and it will be: $asinfo -v "tip:host=192.168.239.1;port=3002"

9

Dear Shaktimaan and PGupta,

Sorry for the late reply.

service {
                address any
                port 3000
#              access-address 192.168.239.3
#               node-id-interface eth1
        }

        heartbeat {
#               mode multicast
#               multicast-group 239.1.99.222
#               port 9918


                mode mesh
                address 192.168.239.1
                port 3002

                mesh-seed-address-port 192.168.239.1 3002.

1366×768 35.5 KB

1366×768 201 KB

I tried with above mentioned (asinfo -v “tip:host=192.168.239.1;port=3002” or asinfo -v “tip:host=192.168.239.3;port=3002”) both.

after adding through command prompt then I tried check whether the 2nodes are visible or not? but still same it showing only 1-node

kindly check once the above mentioned configuration.

currently I am running node-1(IP-address:192.168.239.1) to the cluster and now I am trying to add another node-2(IP-address:192.168.239.3).

so please don’t get hesitation, because I am not understanding how to approach.

Thanks in advance.

10

@Deva Please refer @pgupta’s reply. Your address and access-address have to match! The other node’s IP address should be added in mesh-seed-address-port.

Secondly, check in the ifconfig output if eth1 is the interface that is assigned to 192.168.239.3.

11

Dear Shaktimaan,

I refered @PGupta’s reply after that I changed my config file as below.

network {
        service {
                address any
                port 3000
               access-address 192.168.239.1
               network-interface-name lo
        }

        heartbeat {
               mode mesh
               address 192.168.239.1
               port 3002

              mesh-seed-address-port 192.168.239.3 3002

                interval 150
                timeout 10
        }

with this config file, I tried to run my application but it is throwing error. please find the below error.

host:           192.168.239.1
port:           3000
user:
namespace:      test
set name:       eg-set
key (string):   eg-key
Add node BB93A6B48E63744 192.168.239.1:3000

----Started collecting the DATA from Shared memory and writing into the Record--

  generation 0, ttl 0, 7 bins:
  appState : 0
  appState1 : 0
  appState2 : 0
  appState3 : 0
  appState4 : 0
  appState5 : 0
  appState6 : 0
aerospike_key_put() returned 11 - AEROSPIKE_ERR_CLUSTER
../../project/Makefile:161: recipe for target 'run' failed
make: *** [run] Error 255

Please suggest the solution.

12 
mode mesh
address any
port 3002

address in mode mesh should be omitted or any.

13

@dave The following interface is not correct.

 network-interface-name lo

Check the network interface name for 192.168.239.1 in the output of “ifconfig”. “lo” is the local loopback interface, and you cannot use it to network with other computers.

14

Dear Shakthimaan,

I tried to add the 2nodes to the cluster through command prompt. kindly see the below attachment.

819×460 27.7 KB

for both it returning OK.

after this I tried to check “info”, but still it is showing node-1 information only. when I trying to connect admin there also it showing found 1 node.

1366×768 205 KB
How to check whether my 2 nodes added to the cluster or not?

Thanks in advance.

15

Please share the current aerospike.conf from both the nodes.

16

Dear Shakthimaan,

please find the below configuration for both the NODEs.

Node-1-(192.168.239.1) config file.

network { service { address any port 3000 # access-address 192.168.239.1 # network-interface-name lo }

    heartbeat {
	  #  mode multicast
              # multicast-group 239.1.99.222
              # port 9918


            mode mesh
	#address 192.168.239.1
            port 3002


           
           # mesh-seed-address-port 192.168.239.3 3002


            # To use unicast-mesh heartbeats, remove the 3 lines above, and see
            # aerospike_mesh.conf for alternative.

            interval 150
            timeout 10
    }

    fabric {
            port 3001
    }

    info {
            port 3003
    }

}

Node-2-(192.168.239.3) config file.

network { service { address any port 3000

        #   access-address 192.168.239.3
    }

    heartbeat {
           # mode multicast
           # multicast-group 239.1.99.222
           # port 9918

            # To use unicast-mesh heartbeats, remove the 3 lines above, and see
            # aerospike_mesh.conf for alternative.

            mode mesh
          #  address 192.168.239.3
            port 3002

          #  mesh-seed-address-port 192.168.239.1 3002

            interval 150
            timeout 10
    }

    fabric {
            port 3001
    }

    info {
            port 3003
    }

}

17

Can you also post output of ifconfig -a from both the nodes?

18

Dear Shakthimaan,

Please find the below attachment for both Node-1 & 2 ifcong output.

1366×768 212 KB
1366×768 127 KB

19

with these config files, if you uncomment the mesh-seed-address-port entry on any one or both, the way you have it, cluster should form. assuming your setup is something like this…

/router/ | ---------------------------------------
node1(192.168.239.1) node2 (192.168.239.3)

make sure you are restarting the nodes everytime you change the config files. the config files are only read once when the node starts.

Without mesh-seed-address-port entry, tip command should work and form the cluster dynamically.

20

Dear PGupta,

Thanq very much for helping.

I have one more doubt, How to see or check resultant values on the screen whether both NODE 1 & 2 working or not ? I mean checking the heartbeat for both node1 & 2?