Aerospike docker container 3.6.3 is not forming a cluster

Operations Configuration
1

Hi,

I’m trying to configure an aerospike cluster with docker image 3.6.3 , but I see the its not creating a cluster environment .

(note : its works if i use the network-interface-name as eth0 for one host and lo for the other)

here is my aerospike.conf

network {
        service {
                address any
                port 3000
                access-address host-ip  virtual
                network-interface-name eth0
        }
        heartbeat {
                mode mesh
                port 3002 # Heartbeat port for this node.
                interface-address host-ip
                mesh-seed-address-port <host-ip-of-other-node> 3002
                # List one or more other nodes, one ip-address & port per line:
                mesh-port 3002
                interval 250
                timeout 10
        }
        fabric {
                address any
                port 3001
        }
        info {
                address any
                port 3003
        }
}

Could you guys help me out please . Thanks

2

Could you let us know if you are running both containers on the same physical hosts?

Which docker network configuration are you using?

Did you use Docker composer and swarm?

Please see:

3

You could bring up a 7 node cluster for testing on a single node with a single network card using multicast.

You can use the following script and configure aerospike to use multicast network.

The following only uses docker run and does not take advantage of all the orchestration tools offered by docker:

Modify aerospike.conf to use multicast hearbeat:

heartbeat {
    mode multicast                  # Send heartbeats using Multicast
    address 239.3.9.2              # multicast address
    port 9918 


	interval 150
	timeout 10
}

Use something similar to the following script makecluster.sh to build from repo and generate a 7 node cluster

docker build -t "aerospike/aerospike-server" .
for i in {3..9} ; do
echo $i;
docker stop aerospike${i}
docker rm aerospike${i}
docker run -tid --name aerospike${i} -p ${i}000:3000 -p ${i}001:3001 -p ${i}002:3002 -p ${i}003:3003 aerospike/aerospike-server
done;

| => ./makecluster.sh

Sending build context to Docker daemon 13.82 kB
Step 1 : FROM ubuntu:xenial
 ---> 2fa927b5cdd3
Step 2 : ENV AEROSPIKE_VERSION 3.9.0.3
 ---> Using cache
 ---> e1175083852a
Step 3 : ENV AEROSPIKE_SHA256 49597f4d82da22fe3d6b10a54676a7b0a864279f90cc2f92928f1dbbc76385c8
 ---> Using cache
 ---> 8e72fe0bd0ea
Step 4 : RUN apt-get update -y   &&  apt-get install -y wget python logrotate ca-certificates   && wget "https://www.aerospike.com/artifacts/aerospike-server-community/${AEROSPIKE_VERSION}/aerospike-server-community-${AEROSPIKE_VERSION}-ubuntu16.04.tgz" -O aerospike-server.tgz   && echo "$AEROSPIKE_SHA256 *aerospike-server.tgz" | sha256sum -c -   && mkdir aerospike   && tar xzf aerospike-server.tgz --strip-components=1 -C aerospike   && dpkg -i aerospike/aerospike-server-*.deb   && mkdir -p /var/log/aerospike/   && mkdir -p /var/run/aerospike/   && rm -rf aerospike-server.tgz aerospike /var/lib/apt/lists/*
 ---> Using cache
 ---> c01ff7090834
Step 5 : COPY aerospike_multicast.conf /etc/aerospike/aerospike.conf
 ---> b56601bd7eba
Removing intermediate container fad8c3c23338
Step 6 : COPY entrypoint.sh /entrypoint.sh
 ---> 4369993bb939
Removing intermediate container 5fb4f1860c5b
Step 7 : VOLUME /opt/aerospike/data
 ---> Running in 4dc02cf018e2
 ---> 6d6d0c5dc1f8
Removing intermediate container 4dc02cf018e2
Step 8 : EXPOSE 3000 3001 3002 3003
 ---> Running in 8d91a8046ec9
 ---> 501ecc11fd83
Removing intermediate container 8d91a8046ec9
Step 9 : ENTRYPOINT /entrypoint.sh
 ---> Running in 4feb29e4acdc
 ---> 04f41da16651
Removing intermediate container 4feb29e4acdc
Step 10 : CMD asd
 ---> Running in ee6d90f75e38
 ---> f4559215ebf5
Removing intermediate container ee6d90f75e38
Successfully built f4559215ebf5
3
545eda5e2544c55ccafeeb9ba3a4a178f87b0e113505423ab0476549826aa447
4
e05c83e8e8a5c50e17e7970203000377bc104bc248e1f75225dbf4852bad255e
5
47fe5942436edc03e7463c01b0b0e94ab53396fef511301e4b6f500a007c430c
6
50fca8ec246fff2089589431c0f5d203bdf3ff1ae437ef60776045f5cb26415d
7
ba3d3e522e586443bc9d466705f173453aabc2b32344e4073c0680ae294ff320
8
bc10a242f3ae823dc78c7a758da2e6094bd6bb69091add717b21eeebf86ae33b
9
c1d68ce1ada3140377791c7a8ab8f05c0d2528f345fe2e624467bbda839a0a2d

| => docker ps

    CONTAINER ID        IMAGE                        COMMAND                CREATED             STATUS              PORTS                                                                                            NAMES
    c1d68ce1ada3        aerospike/aerospike-server   "/entrypoint.sh asd"   6 seconds ago       Up 5 seconds        0.0.0.0:9000->3000/tcp, 0.0.0.0:9001->3001/tcp, 0.0.0.0:9002->3002/tcp, 0.0.0.0:9003->3003/tcp   aerospike9
    bc10a242f3ae        aerospike/aerospike-server   "/entrypoint.sh asd"   7 seconds ago       Up 5 seconds        0.0.0.0:8000->3000/tcp, 0.0.0.0:8001->3001/tcp, 0.0.0.0:8002->3002/tcp, 0.0.0.0:8003->3003/tcp   aerospike8
    ba3d3e522e58        aerospike/aerospike-server   "/entrypoint.sh asd"   7 seconds ago       Up 6 seconds        0.0.0.0:7000->3000/tcp, 0.0.0.0:7001->3001/tcp, 0.0.0.0:7002->3002/tcp, 0.0.0.0:7003->3003/tcp   aerospike7
    50fca8ec246f        aerospike/aerospike-server   "/entrypoint.sh asd"   8 seconds ago       Up 6 seconds        0.0.0.0:6000->3000/tcp, 0.0.0.0:6001->3001/tcp, 0.0.0.0:6002->3002/tcp, 0.0.0.0:6003->3003/tcp   aerospike6
    47fe5942436e        aerospike/aerospike-server   "/entrypoint.sh asd"   8 seconds ago       Up 7 seconds        0.0.0.0:5000->3000/tcp, 0.0.0.0:5001->3001/tcp, 0.0.0.0:5002->3002/tcp, 0.0.0.0:5003->3003/tcp   aerospike5
    e05c83e8e8a5        aerospike/aerospike-server   "/entrypoint.sh asd"   9 seconds ago       Up 7 seconds        0.0.0.0:4000->3000/tcp, 0.0.0.0:4001->3001/tcp, 0.0.0.0:4002->3002/tcp, 0.0.0.0:4003->3003/tcp   aerospike4
    545eda5e2544        aerospike/aerospike-server   "/entrypoint.sh asd"   9 seconds ago       Up 8 seconds        0.0.0.0:3000-3003->3000-3003/tcp                                                                 aerospike3

________________________________________________________________________________

| => docker exec -ti aerospike9 bash
    root@c1d68ce1ada3:/# tail -f /var/log/aerospike/aerospike.log 
    Aug 01 2016 18:20:31 GMT: INFO (drv_ssd): (drv_ssd.c:2072) device /opt/aerospike/data/test.dat: used 0, contig-free 4095M (4095 wblocks), swb-free 0, w-q 0 w-tot 0 (0.0/s), defrag-q 0 defrag-tot 0 (0.0/s) defrag-w-tot 0 (0.0/s)
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:164) NODE-ID bb9080011ac4202 CLUSTER-SIZE 7
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:226)    system-memory: free-kbytes 1209640 free-pct 59
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:240)    in-progress: tsvc-q 0 info-q 0 nsup-delete-q  0 rw-hash 0 proxy-hash 0 rec-refs 0
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:262)    fds: proto (0,0,0) heartbeat (0,0,0) fabric (42,42,0)
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:271)    heartbeat-received: self 557 foreign 3396
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:328) {test} objects: all 0 master 0 prole 0
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:366) {test} migrations: complete
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:385) {test} memory-usage: total-bytes 0 index-bytes 0 sindex-bytes 0 data-bytes 0 used-pct 0.00
    Aug 01 2016 18:20:36 GMT: INFO (info): (ticker.c:415) {test} device-usage: used-bytes 0 avail-pct 99
4

If running a docker container on individual host or AWS instances and using the –network host option of docker run then a cluster can be brought up in the following way:

  1. Modify the entrypoint.sh script to auto configure access-addres, address, and interface-address and mesh-seed-address.

    #!/bin/bash

    myip=$(ip -4 addr show eth0 | grep -Po ‘inet \K[\d.]+’|head -n1) sed -i -e "/^.mesh-port 3002./a\ address $myip " /etc/aerospike/aerospike.conf sed -i -e "/^.mesh-port 3002./a\ interface-address $myip " /etc/aerospike/aerospike.conf sed -i -e “/^.address any./a\ access-address $myip” /etc/aerospike/aerospike.conf

    if [[ ! -z “${IP_SEED1}” ]]; then sed -i -e “/^.interval./a\ mesh-seed-address-port ${IP_SEED1} 3002” /etc/aerospike/aerospike.conf fi

    set -e

    if command starts with an option, prepend asd

    if [ “${1:0:1}” = ‘-’ ]; then set – asd “$@” fi

    if asd is specified for the command, start it with any given options

    if [ “$1” = ‘asd’ ]; then # asd should always run in the foreground set – “$@” --foreground

    fi

    the command isn’t asd so run the command the user specified

    exec “$@”

  2. Modify your Dockerfile to include iproute2 for the use of ip addr command, and also add aerospike tools:

    Aerospike Server Dockerfile

    GitHub - aerospike/aerospike-server.docker: Dockerfile for Aerospike Server Community Edition

    FROM ubuntu:xenial

    ENV AEROSPIKE_VERSION 3.9.0.3 ENV AEROSPIKE_SHA256 49597f4d82da22fe3d6b10a54676a7b0a864279f90cc2f92928f1dbbc76385c8

    Install Aerospike

    RUN
    apt-get update -y
    && apt-get install -y wget python logrotate ca-certificates
    && wget “https://www.aerospike.com/artifacts/aerospike-server-community/${AEROSPIKE_VERSION}/aerospike-server-community-${AEROSPIKE_VERSION}-ubuntu16.04.tgz” -O aerospike-server.tgz
    && echo "$AEROSPIKE_SHA256 aerospike-server.tgz" | sha256sum -c -
    && mkdir aerospike
    && tar xzf aerospike-server.tgz --strip-components=1 -C aerospike
    && dpkg -i aerospike/aerospike-server-    .deb \ && dpkg -i aerospike/aerospike-tools-.deb
    && mkdir -p /var/log/aerospike/
    && mkdir -p /var/run/aerospike/ \ && rm -rf aerospike-server.tgz /var/lib/apt/lists/

    RUN apt-get update -y && apt-get install -y apt-utils net-tools iproute2 iputils-ping telnet

    Add the Aerospike configuration specific to this dockerfile

    COPY aerospike.conf /etc/aerospike/aerospike.conf COPY entrypoint.sh /entrypoint.sh

    Mount the Aerospike data directory

    VOLUME [“/opt/aerospike/data”]

    VOLUME [“/etc/aerospike/”]

    Expose Aerospike ports

    3000 – service port, for client connections

    3001 – fabric port, for cluster communication

    3002 – mesh port, for cluster heartbeat

    3003 – info port

    EXPOSE 3000 3001 3002 3003

    Execute the run script in foreground mode

    ENTRYPOINT [“/entrypoint.sh”] CMD [“asd”]

  3. Build image:

sudo docker build -t testaero .

Sending build context to Docker daemon 14.34 kB
Step 1 : FROM ubuntu:xenial
 ---> 42118e3df429
Step 2 : ENV AEROSPIKE_VERSION 3.9.0.3
 ---> Using cache
 ---> 787b971e0dd2
Step 3 : ENV AEROSPIKE_SHA256 49597f4d82da22fe3d6b10a54676a7b0a864279f90cc2f92928f1dbbc76385c8
 ---> Using cache
 ---> a4ae52763c7d
Step 4 : RUN apt-get update -y   &&  apt-get install -y wget python logrotate ca-certificates   && wget "https://www.aerospike.com/artifacts/aerospike-server-community/${AEROSPIKE_VERSION}/aerospike-server-community-${AEROSPIKE_VERSION}-ubuntu16.04.tgz" -O aerospike-server.tgz   && echo "$AEROSPIKE_SHA256 *aerospike-server.tgz" | sha256sum -c -   && mkdir aerospike   && tar xzf aerospike-server.tgz --strip-components=1 -C aerospike   && dpkg -i aerospike/aerospike-server-*.deb   && dpkg -i aerospike/aerospike-tools-*.deb   && mkdir -p /var/log/aerospike/   && mkdir -p /var/run/aerospike/   && rm -rf aerospike-server.tgz /var/lib/apt/lists/*
 ---> Using cache
 ---> bdf911b58705
Step 5 : RUN apt-get update -y && apt-get install -y apt-utils net-tools iproute2 iputils-ping telnet
 ---> Using cache
 ---> c0fd3540cb02
Step 6 : COPY aerospike.conf /etc/aerospike/aerospike.conf
 ---> Using cache
 ---> 7912b2444ba5
Step 7 : COPY entrypoint.sh /entrypoint.sh
 ---> Using cache
 ---> 24322d6ddd90
Step 8 : VOLUME /opt/aerospike/data
 ---> Using cache
 ---> ef247bcc8144
Step 9 : EXPOSE 3000 3001 3002 3003
 ---> Using cache
 ---> 3e4bb79b88d9
Step 10 : ENTRYPOINT /entrypoint.sh
 ---> Using cache
 ---> dc5bc8a0a9ef
Step 11 : CMD asd
 ---> Using cache
 ---> 29d509d11288
Successfully built 29d509d11288

  1. Bring up the cluster by running docker run command on each node and passing the IP_SEED1 address as an environment variable:

    ubuntu@ip-172-31-10-41:~/aerospike-server.docker$ docker run -tid --network host -e “IP_SEED1=172.31.10.42” --name aerospike -p 3000:3000 -p 3001:3001 -p 3002:3002 -p 3003:3003 testaero e40425b30e3c572afdc6105b9b0e3b21dd5a9f299a385059fe846ab5ac79a02c

  2. Verify Cluster:

    ubuntu@ip-172-31-10-41:~/aerospike-server.docker$ docker exec -ti aerospike asadm -e info

                                               Node               Node                  Ip       Build   Cluster            Cluster     Cluster         Principal   Client     Uptime   
                                              .                 Id                   .           .      Size                Key   Integrity                 .    Conns          .   
ip-172-31-10-41.us-west-2.compute.internal:3000   BB92BD3B764D70A    172.31.10.41:3000   C-3.9.0.3         2   6A63CBCD2CAEA092   True        BB9C98AB811970A        8   00:00:48   
ip-172-31-10-42.us-west-2.compute.internal:3000   *BB9C98AB811970A   172.31.10.42:3000   C-3.9.0.3         2   6A63CBCD2CAEA092   True        BB9C98AB811970A        9   22:01:50   
Number of rows: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Information~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace                                              Node   Avail%   Evictions    Master   Replica     Repl     Stop     Pending       Disk    Disk     HWM        Mem     Mem    HWM      Stop   
        .                                                 .        .           .   Objects   Objects   Factor   Writes    Migrates       Used   Used%   Disk%       Used   Used%   Mem%   Writes%   
        .                                                 .        .           .         .         .        .        .   (tx%,rx%)          .       .       .          .       .      .         .   
test        ip-172-31-10-41.us-west-2.compute.internal:3000   99         0.000     0.000     0.000     2        false    (0,0)       0.000 B    0       50      0.000 B    0       60     90        
test        ip-172-31-10-42.us-west-2.compute.internal:3000   99         0.000     0.000     0.000     2        false    (0,0)       0.000 B    0       50      0.000 B    0       60     90        
test                                                                     0.000     0.000     0.000                       (0,0)       0.000 B                    0.000 B                             
Number of rows: 3

Note for use of overlay networks: If using an overlay network, the containers will have different interfaces and ip addresses (eth0,eth1 and eth2) . interface-address, access-address and mesh-seed-address-port would need to be adjusted accordingly. In such a case you may need to also use a discovery service to determine the IP addresses of other containers in the cluster to populate mesh-seed-address-port.

A basic discovery service could be setup using a stand alone aerospike server and populating a seed IP list as nodes come up. You could then query for the IP address of the container nodes in the cluster and auto populate mesh-seed-address-port

Using aql commands in the entrypoint.sh the mesh-seed-address-port can be generated by querying the standalone aerospike instance (ie: 10.0.0.3)

aql -c "INSERT INTO test.demo (PK, ClusterName, IpAddress) VALUES ('${myip}','A', '${myip}') " -h 10.0.0.3
sleep 10
aql -c "select * from test.demo" -h 10.0.0.3 |grep -Eo '([[:digit:]]{1,3}.[[:digit:]]{1,3}.[[:digit:]]{1,3}.[[:digit:]]{1,3})' > /tmp/ip.txt
for i in $(cat /tmp/ip.txt); do sed -i -e "/^.*interval.*/a\ mesh-seed-address-port $i 3002" /etc/aerospike/aerospike.conf ; done
sleep 10
for i in $(cat /tmp/ip.txt); do aql -c "DELETE FROM test.demo WHERE PK ='$i'" -h 10.0.0.3; done
5

The multicast configuration you have there, is that right? I can’t see to get it working with that

6

Multicast has changed a little bit.

Please see:

http://www.aerospike.com/docs/operations/configure/network/heartbeat#multicast-heartbeat

  heartbeat {
    mode multicast                  # Send heartbeats using Multicast
    multicast-group 239.1.99.2              # multicast address
    port 9918                       # multicast port
    address 192.168.1.100 # (Optional) (Default any) IP of the NIC to
                                    # use to send out heartbeat and bind
                                    # fabric ports
    interval 150                    # Number of milliseconds between heartbeats
    timeout 10                      # Number of heartbeat intervals to wait
                                    # before timing out a node
  }
7

Putting “address 192.168.1.100” doesn’t work

I get an error saying ‘Legacy heartbeat IPv4 addresses should be a subset of fabric binding addresses’