Create a cluster with docker swarm


I am working to Aerospike for a school project on NoSQL Database. First i successed to install Aerospike on an Ubuntu VM. It was functionnal, load data was ok, amc ok. I used this loader :

But now i would like create a cluster to work on sharding and replication. So i followed this blog with docker swarm fully (with deploy of yml and conf on nodes) on W7.

I successed to insert data with aql and scale database to 4 servers. Nodes ip of both nodes are and

But my problem, i can not connect to the database with loader from my windows desktop.

java -cp target/aerospike-load-2.2-jar-with-dependencies.jar com.aerospike.load.AerospikeLoad -h -p 3003 -c example/temp_state.json example/temp_state.csv

ERROR AerospikeLoad :242 - com.aerospike.client.AerospikeException$Connection: Error Code 11: Failed to connect to host(s): 3003 Error Code 11: connect timed out

How could i connect to the database from my windows desktop ? It’s possible in my case to use AMC to monitor my cluster ?

Use Swarm is very easy to scale but is it usable ?

Kindly help. Thanks!

1 Like

Does aql work on the docker container? If so, then it’s probably a networking configuration issue. Either something locally not allowing you, the ports arent forwarded correctly, or some firewall configuration.


Aql access from node leader is ok


I have installed image dockersamples/visualizer to monitor nodes on leader node.

I can see my three nodes with all containers with leader node ip + port 8080 defined

On my nodes, if i realize a docker ps : port 3000 is forwarded to 3003

If i test the load with port 3003 that’s ko :

It’s the same with 3000

I use default config of docker and i don’t have firewall on my laptop.

I think my problem is a networking configuration issue but i don’t find where i can fix it …

I did a nestat -tulpen on my leader node

docker@node-leader:~$ netsat - tulpen

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0* LISTEN -

tcp 0 0 :::2376 :::* LISTEN -

tcp 0 0 :::2377 :::* LISTEN -

tcp 0 0 :::7946 :::* LISTEN -

tcp 0 0 :::8080 :::* LISTEN -

tcp 0 0 :::8081 :::* LISTEN -

tcp 0 0 :::22 :::* LISTEN -

udp 0 0* -

udp 0 0 :::7946 :::* -

We can observe that port 3003 is not in the list.

It’s stange that port 3003 is not mapped automatically. Port 8080 corresponding to visualizer and 8081 to AMC are mapped.

If i managed to map port 3003, i think connection will be ok …

You really want to connect to port 3000 for clients. Port 3001-3003 I think are for heartbeat/fabric. I’ve never used this docker setup before, so I’m not exactly sure whats going on. If aql is working, then it must be network/firewall related. Can you post output of ‘netstat -tunap’ or try grepping for the pid of the docker container, to see if docker had any forwarding setup. If the docker container doesn’t have any kind of listener setup, then you know where to look. Maybe check this out

I dug the settings of Docker Swarm.

We use param endpoint_mode: dnsrr to scale easy. With this param we can’t open port because each container is name with an unique name.

The loadbalancer (or proxy) know service name and will transfert data to this service.

=> The port is open with the load balancer

In Aeropsike doc, we can find Aerospike’s Smart Client which have load balancer role to automatically distributes both data and traffic to all the nodes in a cluster.

In the doc It’s indicate :

When you compile your application with an Aerospike Client API library (such as the C Client or Java Client), it includes the Aerospike Smart Client™, which uses a binary wire protocol to connect to servers in a cluster.

But what server must have to be parameter …

My question, using a proxy as HAPROXY it’s the solution ?

Running proxy/haproxy is not the solution I don’t think. Are you sure your docker containers aren’t listening on any ports? Can you show a full printout of ‘netstat -tunap’ on the machine running the containers? I think you just need to publish port 3000… but it would make more sense if that was required for them to include that in the setup yml… Can you also show us the inspect output of the docker container? from

try docker service inspect… ex…

$ docker service inspect --format="{{json .Endpoint.Spec.Ports}}" my-web

You will find below the resultat of netstat running on my node leader

docker@node-leader:~$ netstat -tunap
netstat: can't scan /proc - are you root?
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
PID/Program name
tcp        0      0    *               LISTEN -
tcp        0      0     TIME_WAIT -
tcp        0      0      ESTABLISHED-
tcp        0      0     ESTABLISHED-
tcp        0      0            ESTABLISHED-
tcp        0      0     TIME_WAIT-
tcp        0      0     TIME_WAIT-
tcp        0      0     TIME_WAIT-
tcp        0      0 :::2376                 :::*                    LISTEN-
tcp        0      0 :::2377                 :::*                    LISTEN-
tcp        0      0 :::7946                 :::*                    LISTEN-
tcp        0      0 :::8080                 :::*                    LISTEN-
tcp        0      0 :::8081                 :::*                    LISTEN-
tcp        0      0 :::22                   :::*                    LISTEN-
tcp        0      0 ::ffff: ::ffff: ESTABLISHED -
tcp        0      0 ::ffff: ::ffff: ESTABLISHED -
tcp        0      0 ::ffff: ::ffff: ESTABLISHED -
tcp        0      0 ::ffff: ::ffff: TIME_WAIT   -
udp        0      0  *-
udp        0      0 :::7946                 :::*-

We can look that port 3003 is not open.

I performed a docker inspect.

We can find tha port section is not present

docker@node-leader:~$ docker service inspect aerospike_aerospikedb
        "ID": "it05qt15ut2kf50w68mp9yio5",
        "Version": {
            "Index": 327
        "CreatedAt": "2018-07-04T19:06:04.793071341Z",
        "UpdatedAt": "2018-07-06T11:04:58.73011964Z",
        "Spec": {
            "Name": "aerospike_aerospikedb",
            "Labels": {
                "com.docker.stack.image": "aerospike/aerospike-server:latest",
                "com.docker.stack.namespace": "aerospike"
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "aerospike/aerospike-server:latest@sha256:bda87b8948823a3e921688f24c15a385b351dc7855e258a5bc4a85c01a3f0075",
                    "Labels": {
                        "com.aerospike.description": "This label is for all containers for the Aerospike service",
                        "com.docker.stack.namespace": "aerospike"
                    "Args": [
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {},
                    "Secrets": [
                            "File": {
                                "Name": "aerospike.conf",
                                "UID": "0",
                                "GID": "0",
                                "Mode": 288
                            "SecretID": "xqk8ru7dvj6i9nww2ehu10qr7",
                            "SecretName": "aerospike_conffile"
                    "Isolation": "default"
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                "Placement": {
                    "Constraints": [
                        "node.role == worker"
                    "Platforms": [
                            "Architecture": "amd64",
                            "OS": "linux"
                "Networks": [
                        "Target": "u8uoqywqz5p3zjshvp8pyym6a",
                        "Aliases": [
                "ForceUpdate": 1,
                "Runtime": "container"
            "Mode": {
                "Replicated": {
                    "Replicas": 4
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            "EndpointSpec": {
                "Mode": "dnsrr"

I execute this command found in this page :

docker@node-leader:~$ docker service update --publish-add 3003:3000 aerospike_aerospikedb
Error response from daemon: rpc error: code = InvalidArgument desc = EndpointSpec: port published with ingres
 mode can't be used with dnsrr mode

But i think as indicated on this dockers’s schema when EndpointSpec with dnsrr the port transfert must be done by a proxy / LB

I found in this page the utilisation of HAPROXY in front

I’m going to test in opening 3000 port on a dedicaded node

The latest java clients (4.1.7 onwards) support having a seed node being a load balancer. You still need all of the Aerospike containers to open port 3000.

ok I try to use Aerospike loader with aerospike-client version 3.3.1 (indicate in pom.xml)

I can modify in pom.xml by the last 4.1.8

But in command i need to indicate a server ip.

If I indicate IP of Swarm manager that’s OK ?

If it works… then go for it :stuck_out_tongue:

I modify in pom.xml of aerospike-loader with latest version.

=> KO, the batch don’t compile with maven.

So i mount a VM Ubuntu to install Aerospike with python.

All my redirect host and port with virtualbox are all tests are KO. ex :


The problem is what ip indicate to access to database ?

Aerospike doc is easy to understand but does not allow to test a complete project without spend at min one week at understand :angry:

The complexity is within the docker configuration and your virtualization layer - not with Aerospike… If you are running that loader locally on your desktop pointing to the VM, then it looks like you want to point to or… Not really sure what you have setup though, but try those out…


To close this ticket, connexion is ok with a proxy in using port 80. Here is my yml file :

version: '3.2'

        image: aerospike/aerospike-server:latest
        - aerospikenetwork
          - SERVICE_PORTS=3000
          - target: 3000
            published: 3000
            protocol: tcp
            mode: host
            replicas: 4
            endpoint_mode: dnsrr
            com.aerospike.description: "This label is for all containers for the Aerospike service"
        command: [ "--config-file","/run/secrets/aerospike.conf"]
        - source: conffile
          target: aerospike.conf
          mode: 0440
            constraints: [node.role == worker]
        image: aerospike/aerospike-tools:latest
        - aerospikenetwork
        - aerospike
        - /run/secrets/discovery
        - "--servicename"
        - aerospikedb
        - "-i"
        - "5"
        - "-v"
        - source: discoveryfile
          target: discovery
          mode: 0750
            constraints: [node.role == manager]
        image: dockersamples/visualizer:stable
          - "8080:8080"
        stop_grace_period: 1m30s
          - "/var/run/docker.sock:/var/run/docker.sock"
            constraints: [node.role == manager]
        image: dockercloud/haproxy
        # Won't start until at least one of our app services is up and running.
          - aerospikedb
          # The type of load balancing strategy that will be used.
          # - leastconn sends request to the service with the least active requests.
          # - roundrobin rotates the requests around the services.
          - BALANCE=leastconn
          # Used to identify services.
          - ADDITIONAL_SERVICES=project_dir:aerospikedb
          - MODE=tcp
          - STATS_PORT=1936
          - STATS_AUTH="admin:admin"
          # Since our app services are running on the same port,
          # the HAProxy will use the docker.sock to find the
          # services that it should load balance.
          - /var/run/docker.sock:/var/run/docker.sock
          # The internal used by the HAProxy is 80,
          # but we can expose any port that we would like externally.
          # For example, if you are running something else on 80,
          # you probably don't want to expose the HAProxy on 80 as well.
          - 80:80
          - 1936:1936
          #- 8080:8083
         # - 8081:8084
          - aerospikenetwork
          # The HAProxy is assigned as the manager.
            constraints: [node.role == manager]
        image: apsops/aerospike-amc:v4.0.19
          - "8081:8081"
            constraints: [node.role == manager]

        driver: overlay
        attachable: true

        file: ./aerospike.conf
        file: ./
1 Like
© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.