Cluster setup issue: 2 nodes: community editions: one node is ubuntu machine and other node is vagrant box on windows machine


#1

Hi,

I am trying to setup a cluster with the following section in the config file, but there seems to be a problem I could not find a solution for:

Node 1

network {
        service {
                address any
                port 3000
                #access-address 192.168.120.118
                access-address 172.28.128.4 virtual
        }

        heartbeat {
                #mode multicast
                #address 239.1.99.222
                #port 9918

                # To use unicast-mesh heartbeats, comment out the 3 lines above and
                # use the following 4 lines instead.
                mode mesh
                address 172.28.128.4
                port 3002
                mesh-seed-address-port 172.28.128.4 3002
                mesh-seed-address-port 192.168.0.20 3002

                interval 250
                timeout 20
        }

Node 2

network {
        service {
                address any
                port 3000
                access-address 192.168.0.20
        }

        heartbeat {
                #mode multicast
                #address 239.1.99.222
                #port 9918

                # To use unicast-mesh heartbeats, remove the 3 lines above, and see
                # aerospike_mesh.conf for alternative.

                mode  mesh
                address 192.168.0.20
                port 3002
                mesh-seed-address-port 192.168.0.20 3002
                mesh-seed-address-port 172.28.128.4 3002

                interval 150
                timeout 10
        }

Problem: I am not able to setup the cluster and when I tail the log file I get the below warnings on the first node

Jun 07 2016 17:46:55 GMT: WARNING (cf:socket): (socket.c:371) Error in delayed connect() to 192.168.0.20:3002: timed out
Jun 07 2016 17:47:11 GMT: WARNING (cf:socket): (socket.c:371) Error in delayed connect() to 192.168.0.20:3002: timed out
Jun 07 2016 17:47:27 GMT: WARNING (cf:socket): (socket.c:371) Error in delayed connect() to 192.168.0.20:3002: timed out
Jun 07 2016 17:47:43 GMT: WARNING (cf:socket): (socket.c:371) Error in delayed connect() to 192.168.0.20:3002: timed out
Jun 07 2016 17:47:59 GMT: WARNING (cf:socket): (socket.c:371) Error in delayed connect() to 192.168.0.20:3002: timed out
Jun 07 2016 17:48:15 GMT: WARNING (cf:socket): (socket.c:371) Error in delayed connect() to 192.168.0.20:3002: timed out
Jun 07 2016 17:48:30 GMT: WARNING (cf:socket): (socket.c:371) Error in delayed connect() to 192.168.0.20:3002: timed out
Jun 07 2016 18:27:28 GMT: WARNING (as): (signal.c:193) SIGTERM received, shutting down
Jun 07 2016 18:27:44 GMT: WARNING (hb): (hb.c:1335) rejected tip for self: 172.28.128.4:3002
Jun 07 2016 18:27:44 GMT: WARNING (hb): (hb.c:2131) couldn't add remote heartbeat service 172.28.128.4:3002 to mesh host list

Please advice!

Regards


Cluster Mismatch AMC cannot monitor this node as it belongs to a different cluster
#2

Hi,

I had a namespace mismatch and I corrected it by ensuring that both conf files have the same namespace section and restarted the servers on both the machines

But no progress, I am still not able to form a cluster

please advice

regards


#3

Appears the first node is unable to reach 192.168.0.20.

From the first node are you able to ping the second?


#4

Yes I am able to ping the 192.168.0.20 from the first node

In fact I am able to ping both the machines from each other


#5

From our discussion here, I suspect that one of these nodes may be running a firewall and preventing access.

Try disabling the firewall:

sudo service iptables stop;
sudo chkconfig iptables off;

#6

It doesn’t work

I disabled the firewall on both the machines

:confused:


#7

just to add

I disabled the iptables/ firewalls on both the machines and restarted the servers on both the machines

but still no cluster created


#8

From your AMC issue, the Ubuntu box is unable to talk to the Vagrant box but the Vagrant box is able to talk to the Ubuntu box. So something is blocking connections originating on the Ubuntu box to the Vagrant box, it could be the firewall on the Vagrant host machine.


#9

so I verified the below

[vagrant@localhost aerospike]$ sudo service iptables status

iptables: Firewall is not running.

so, what could be the possible blockage?


#10

The status as of now is:

  • I am able to run the server and amc on both individual machines

  • I am able to view the ubuntu machine as a different cluster on my vagrant box

  • When I try to view the vagrant machine on my ubuntu I get the below message(this is after disabling the firewalls on both machines):: But before it says “this feature is available in enterprise edition it displays another message that the vagrant node is successfully added”

So is this some thing to do with the community editions that I am using on both systems, and should I be using only enterprise editions on both machines

please advice

cheers


#11

Hi, For reference the below are the config files of both the machines

Node 1(vagrant machine) Config file:

 # Aerospike database configuration file.

  # This stanza must come first.
service {

        user root
        group root
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        pidfile /var/run/aerospike/asd.pid
        service-threads 4
        transaction-queues 4
        transaction-threads-per-queue 4
        proto-fd-max 15000
}

logging {

        # Log file must be an absolute path.
        file /var/log/aerospike/aerospike.log {
                context any warning
        }
        file /var/log/aerospike/udf.log {
                context any critical
                context udf info
                context aggr info
        }
}

network {

        service {
                address any
                port 3000
                #access-address 192.168.120.118
                access-address 172.28.128.3 virtual
        }

        heartbeat {
                #mode multicast
                #address 239.1.99.222
                #port 9918

                # To use unicast-mesh heartbeats, comment out the 3 lines above and
                # use the following 4 lines instead.
                mode mesh
                address 172.28.128.3
                port 3002
                mesh-address 192.168.0.20
                mesh-port 3002

                interval 150
                timeout 10
        }

        fabric {
                port 3001
        }

        info {
                port 3003
        }
}

                #namespace test {
                #       replication-factor 2
                #       memory-size 4G
                #       default-ttl 30d # 30 days, use 0 to never expire/evict.
                #
                #       storage-engine memory
                #}

namespace test {

        replication-factor 2
        memory-size 2G
        default-ttl 5d # 5 days, use 0 to never expire/evict.

         #       storage-engine memory

        # To use file storage backing, comment out the line above and use the
        # following lines instead.
        storage-engine device {
                file /opt/aerospike/data/test.dat
                filesize 5G
                data-in-memory true # Store data in memory in addition to file.
        }
}

Node 2(ubuntu machine) config file

 # Aerospike database configuration file.

service {
	user root
	group root
	paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
	pidfile /var/run/aerospike/asd.pid
	service-threads 4
	transaction-queues 4
	transaction-threads-per-queue 4
	proto-fd-max 15000
}

logging {

	# Log file must be an absolute path.
	file /var/log/aerospike/aerospike.log {
		context any info
	}

	file /var/log/aerospike/udf.log {
		
		context any critical
		context udf debug
	}
}

network {
	service {
		address any
		port 3000
		access-address 192.168.0.20
	}

	heartbeat {
		#mode multicast
		#address 239.1.99.222
		#port 9918

		# To use unicast-mesh heartbeats, remove the 3 lines above, and see
		# aerospike_mesh.conf for alternative.

		mode  mesh
		address 192.168.0.20
		port 3002
		mesh-address 172.28.128.3 
		mesh-port 3002

		interval 150
		timeout 10
	}

	fabric {
		port 3001
	}

	info {
		port 3003
	}
}

 #namespace test {
 #       replication-factor 2
 #       memory-size 4G
 #       default-ttl 30d # 30 days, use 0 to never expire/evict.
 #
 #       storage-engine memory
 #}

namespace test {

        replication-factor 2
        memory-size 2G
        default-ttl 5d # 5 days, use 0 to never expire/evict.

 #       storage-engine memory

        # To use file storage backing, comment out the line above and use the
        # following lines instead.
        storage-engine device {
                file /opt/aerospike/data/test.dat
                filesize 5G
                data-in-memory true # Store data in memory in addition to file.
        }
}

#12

Vagrant is the guest there, windows may also have a firewall.

Also are you absolutely sure messages are able to route from your Ubuntu box to your vagrant vm?

I suspect your access address really should be 192.168.120.118 virtual, or whatever your Windows host box’s IP is at this time. You will probably also need to setup port forwarding for ports 3000 and 3002 between your vagrant host and the guest vm.


#13

okay

I will try what you mentioned ut I just want to tell you that I tried to form a cluster using 2 ubuntu machine and it was done almost in a jiffy

will now try my luck with the windows firewall and then the port forwarding

will keep posted

cheers kporter