Create a cluster with docker swarm


#1

Hi,

I am working to Aerospike for a school project on NoSQL Database. First i successed to install Aerospike on an Ubuntu VM. It was functionnal, load data was ok, amc ok. I used this loader : https://github.com/aerospike/aerospike-loader

But now i would like create a cluster to work on sharding and replication. So i followed this blog with docker swarm fully (with deploy of yml and conf on nodes) on W7.

I successed to insert data with aql and scale database to 4 servers. Nodes ip of both nodes are 192.168.99.101 and 192.168.99.102

But my problem, i can not connect to the database with loader from my windows desktop.

java -cp target/aerospike-load-2.2-jar-with-dependencies.jar com.aerospike.load.AerospikeLoad -h 192.168.99.101 -p 3003 -c example/temp_state.json example/temp_state.csv

ERROR AerospikeLoad :242 - com.aerospike.client.AerospikeException$Connection: Error Code 11: Failed to connect to host(s):192.168.99.101 3003 Error Code 11: java.net.SocketTimeoutException: connect timed out

How could i connect to the database from my windows desktop ? It’s possible in my case to use AMC to monitor my cluster ?

Use Swarm is very easy to scale but is it usable ?

Kindly help. Thanks!


#2

Does aql work on the docker container? If so, then it’s probably a networking configuration issue. Either something locally not allowing you, the ports arent forwarded correctly, or some firewall configuration.


#3

Hi,

Aql access from node leader is ok

aql

I have installed image dockersamples/visualizer to monitor nodes on leader node.

I can see my three nodes with all containers with leader node ip + port 8080 defined

On my nodes, if i realize a docker ps : port 3000 is forwarded to 3003

If i test the load with port 3003 that’s ko :

It’s the same with 3000

I use default config of docker and i don’t have firewall on my laptop.

I think my problem is a networking configuration issue but i don’t find where i can fix it …


#4

I did a nestat -tulpen on my leader node

docker@node-leader:~$ netsat - tulpen

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -

tcp 0 0 :::2376 :::* LISTEN -

tcp 0 0 :::2377 :::* LISTEN -

tcp 0 0 :::7946 :::* LISTEN -

tcp 0 0 :::8080 :::* LISTEN -

tcp 0 0 :::8081 :::* LISTEN -

tcp 0 0 :::22 :::* LISTEN -

udp 0 0 0.0.0.0:4789 0.0.0.0:* -

udp 0 0 :::7946 :::* -

We can observe that port 3003 is not in the list.

It’s stange that port 3003 is not mapped automatically. Port 8080 corresponding to visualizer and 8081 to AMC are mapped.

If i managed to map port 3003, i think connection will be ok …


#5

You really want to connect to port 3000 for clients. Port 3001-3003 I think are for heartbeat/fabric. I’ve never used this docker setup before, so I’m not exactly sure whats going on. If aql is working, then it must be network/firewall related. Can you post output of ‘netstat -tunap’ or try grepping for the pid of the docker container, to see if docker had any forwarding setup. If the docker container doesn’t have any kind of listener setup, then you know where to look. Maybe check this out https://forums.docker.com/t/how-to-expose-port-on-running-container/3252


#6

https://www.aerospike.com/docs/operations/configure/network


#7

I dug the settings of Docker Swarm.

We use param endpoint_mode: dnsrr to scale easy. With this param we can’t open port because each container is name with an unique name.

The loadbalancer (or proxy) know service name and will transfert data to this service.

=> The port is open with the load balancer

In Aeropsike doc, we can find Aerospike’s Smart Client which have load balancer role to automatically distributes both data and traffic to all the nodes in a cluster.

In the doc https://www.aerospike.com/docs/architecture/clients.html It’s indicate :

When you compile your application with an Aerospike Client API library (such as the C Client or Java Client), it includes the Aerospike Smart Client™, which uses a binary wire protocol to connect to servers in a cluster.

But what server must have to be parameter …

My question, using a proxy as HAPROXY it’s the solution ?


#8

Running proxy/haproxy is not the solution I don’t think. Are you sure your docker containers aren’t listening on any ports? Can you show a full printout of ‘netstat -tunap’ on the machine running the containers? I think you just need to publish port 3000… but it would make more sense if that was required for them to include that in the setup yml… Can you also show us the inspect output of the docker container? from https://docs.docker.com/engine/swarm/ingress/#publish-a-port-for-tcp-only-or-udp-only

try docker service inspect… ex…

$ docker service inspect --format="{{json .Endpoint.Spec.Ports}}" my-web
[{"Protocol":"tcp","TargetPort":80,"PublishedPort":8080}]

#9

You will find below the resultat of netstat running on my node leader

docker@node-leader:~$ netstat -tunap
netstat: can't scan /proc - are you root?
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN -
tcp        0      0 192.168.99.101:44014    192.168.99.103:7946     TIME_WAIT -
tcp        0      0 192.168.99.101:22       192.168.99.1:49858      ESTABLISHED-
tcp        0      0 192.168.99.101:45382    192.168.99.101:2377     ESTABLISHED-
tcp        0      0 10.0.2.15:22            10.0.2.2:64384          ESTABLISHED-
tcp        0      0 192.168.99.101:44650    192.168.99.102:7946     TIME_WAIT-
tcp        0      0 192.168.99.101:44648    192.168.99.102:7946     TIME_WAIT-
tcp        0      0 192.168.99.101:44010    192.168.99.103:7946     TIME_WAIT-
tcp        0      0 :::2376                 :::*                    LISTEN-
tcp        0      0 :::2377                 :::*                    LISTEN-
tcp        0      0 :::7946                 :::*                    LISTEN-
tcp        0      0 :::8080                 :::*                    LISTEN-
tcp        0      0 :::8081                 :::*                    LISTEN-
tcp        0      0 :::22                   :::*                    LISTEN-
tcp        0      0 ::ffff:192.168.99.101:2377 ::ffff:192.168.99.103:33590 ESTABLISHED -
tcp        0      0 ::ffff:192.168.99.101:2377 ::ffff:192.168.99.102:51168 ESTABLISHED -
tcp        0      0 ::ffff:192.168.99.101:2377 ::ffff:192.168.99.101:45382 ESTABLISHED -
tcp        0      0 ::ffff:192.168.99.101:7946 ::ffff:192.168.99.102:36162 TIME_WAIT   -
udp        0      0 0.0.0.0:4789            0.0.0.0:*-
udp        0      0 :::7946                 :::*-

We can look that port 3003 is not open.

I performed a docker inspect.

We can find tha port section is not present

docker@node-leader:~$ docker service inspect aerospike_aerospikedb
[
    {
        "ID": "it05qt15ut2kf50w68mp9yio5",
        "Version": {
            "Index": 327
        },
        "CreatedAt": "2018-07-04T19:06:04.793071341Z",
        "UpdatedAt": "2018-07-06T11:04:58.73011964Z",
        "Spec": {
            "Name": "aerospike_aerospikedb",
            "Labels": {
                "com.docker.stack.image": "aerospike/aerospike-server:latest",
                "com.docker.stack.namespace": "aerospike"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "aerospike/aerospike-server:latest@sha256:bda87b8948823a3e921688f24c15a385b351dc7855e258a5bc4a85c01a3f0075",
                    "Labels": {
                        "com.aerospike.description": "This label is for all containers for the Aerospike service",
                        "com.docker.stack.namespace": "aerospike"
                    },
                    "Args": [
                        "--config-file",
                        "/run/secrets/aerospike.conf"
                    ],
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {},
                    "Secrets": [
                        {
                            "File": {
                                "Name": "aerospike.conf",
                                "UID": "0",
                                "GID": "0",
                                "Mode": 288
                            },
                            "SecretID": "xqk8ru7dvj6i9nww2ehu10qr7",
                            "SecretName": "aerospike_conffile"
                        }
                    ],
                    "Isolation": "default"
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Constraints": [
                        "node.role == worker"
                    ],
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "u8uoqywqz5p3zjshvp8pyym6a",
                        "Aliases": [
                            "aerospikedb"
                        ]
                    }
                ],
                "ForceUpdate": 1,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 4
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "dnsrr"
            }
        }
	}
]

I execute this command found in this page :

docker@node-leader:~$ docker service update --publish-add 3003:3000 aerospike_aerospikedb
Error response from daemon: rpc error: code = InvalidArgument desc = EndpointSpec: port published with ingres
 mode can't be used with dnsrr mode

But i think as indicated on this dockers’s schema when EndpointSpec with dnsrr the port transfert must be done by a proxy / LB

I found in this page the utilisation of HAPROXY in front https://www.aerospike.com/docs/deploy_guides/docker/examples

I’m going to test in opening 3000 port on a dedicaded node


#10

The latest java clients (4.1.7 onwards) support having a seed node being a load balancer. You still need all of the Aerospike containers to open port 3000.


#11

ok I try to use Aerospike loader with aerospike-client version 3.3.1 (indicate in pom.xml)

I can modify in pom.xml by the last 4.1.8

But in command i need to indicate a server ip.

If I indicate IP of Swarm manager that’s OK ?


#12

If it works… then go for it :stuck_out_tongue:


#13

I modify in pom.xml of aerospike-loader with latest version.

=> KO, the batch don’t compile with maven.

So i mount a VM Ubuntu to install Aerospike with python.

All my redirect host and port with virtualbox are all tests are KO. ex :

image

The problem is what ip indicate to access to database ?

Aerospike doc is easy to understand but does not allow to test a complete project without spend at min one week at understand :angry:


#14

The complexity is within the docker configuration and your virtualization layer - not with Aerospike… If you are running that loader locally on your desktop pointing to the VM, then it looks like you want to point to 127.0.0.1 or 10.0.2.15… Not really sure what you have setup though, but try those out…


#15

Hi,

To close this ticket, connexion is ok with a proxy in using port 80. Here is my yml file :

version: '3.2'

services:
    aerospikedb:
        image: aerospike/aerospike-server:latest
        networks:
        - aerospikenetwork
        environment:
          - SERVICE_PORTS=3000
        ports:
          - target: 3000
            published: 3000
            protocol: tcp
            mode: host
        deploy:
            replicas: 4
            endpoint_mode: dnsrr
        labels:
            com.aerospike.description: "This label is for all containers for the Aerospike service"
        command: [ "--config-file","/run/secrets/aerospike.conf"]
        secrets:
        - source: conffile
          target: aerospike.conf
          mode: 0440
        deploy:
          placement:
            constraints: [node.role == worker]
    
    meshworker:
        image: aerospike/aerospike-tools:latest
        networks:
        - aerospikenetwork
        depends_on:
        - aerospike
        entrypoint:
        - /run/secrets/discovery
        - "--servicename"
        - aerospikedb
        - "-i"
        - "5"
        - "-v"
        secrets:
        - source: discoveryfile
          target: discovery
          mode: 0750
        deploy:
          placement:
            constraints: [node.role == manager]
    visualizer:
        image: dockersamples/visualizer:stable
        ports:
          - "8080:8080"
        stop_grace_period: 1m30s
        volumes:
          - "/var/run/docker.sock:/var/run/docker.sock"
        deploy:
          placement:
            constraints: [node.role == manager]
    proxy:
        image: dockercloud/haproxy
        # Won't start until at least one of our app services is up and running.
        depends_on:
          - aerospikedb
        environment:
          # The type of load balancing strategy that will be used.
          # - leastconn sends request to the service with the least active requests.
          # - roundrobin rotates the requests around the services.
          - BALANCE=leastconn
          # Used to identify services.
          - ADDITIONAL_SERVICES=project_dir:aerospikedb
          - MODE=tcp
          - STATS_PORT=1936
          - STATS_AUTH="admin:admin"
        volumes:
          # Since our app services are running on the same port,
          # the HAProxy will use the docker.sock to find the
          # services that it should load balance.
          - /var/run/docker.sock:/var/run/docker.sock
        ports:
          # The internal used by the HAProxy is 80,
          # but we can expose any port that we would like externally.
          # For example, if you are running something else on 80,
          # you probably don't want to expose the HAProxy on 80 as well.
          - 80:80
          - 1936:1936
          #- 8080:8083
         # - 8081:8084
        networks:
          - aerospikenetwork
        deploy:
          # The HAProxy is assigned as the manager.
          placement:
            constraints: [node.role == manager]
    amc:
        image: apsops/aerospike-amc:v4.0.19
        ports:
          - "8081:8081"
        deploy:
          placement:
            constraints: [node.role == manager]

networks:
    aerospikenetwork:
        driver: overlay
        attachable: true

secrets:
    conffile:
        file: ./aerospike.conf
    discoveryfile:
        file: ./discovery.py

Swarm and Stop writes