Timeout options are not respected

ricky.kwan.ix · January 30, 2020, 10:26pm

Hi,

We have clusters in different parts of the world. Among them, one is in Singapore (I’m in Canada). As such, when trying to connect with the client, it times out. Trying to increase the default timeout doesn’t seem to work.

Aerospike client installed via pip install aerospike which installed 3.10.0.

import aerospike

client_config = {
    "hosts": [("sg-1.example.com", 4333, "sg.example.com")],
    "policies": {
        "login_timeout_ms": 10000
    },
    "tls": {
        "enable": True,
        "cafile": "/path/to/ca.pem"
    }
}

client = aerospike.client(client_config)
client.connect("user", "password")

It would result in the following error (IPs scrubbed)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
exception.TimeoutError: (9L, ' from 10.0.0.1:4333', None, None, False)

Based on the error code, it is a timeout. The error message doesn’t help much. And the file that generated the error is None . Further increasing login_timeout_ms doesn’t help. I also tried adding policies.connect_timeout, and policies.{read,operate,admin,query}.total_timeout individually to no avail.

If I switch the host to point to the new york cluster, it works.

I further decided to test the connection using aql.

$ aql -h sg-1.example.com:4333 -U"$USER" -P"$PASS" --tls-enable --tls-cafile=/path/to/ca.pem --tls-name sg.example.com -v
Seed:         sg-1.example.com:4333
User:         test
Config File:  /etc/aerospike/astools.conf /root/.aerospike/astools.conf
2020-01-30 17:13:31 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:13:31 INFO Add node BB9AABBCCDDEEFF 10.0.0.1:4333
2020-01-30 17:13:32 DEBUG TLS name 'sg.example.com' matches
Error 9:  from 10.0.0.1:4333

If I add -T 5000 to the aql command, we get:

$ aql -h sg-1.example.com:4333 -U"$USER" -P"$PASS" --tls-enable --tls-cafile=/path/to/ca.pem --tls-name sg.example.com -v -T 5000
Seed:         sg-1.example.com:4333
User:         test
Config File:  /etc/aerospike/astools.conf /root/.aerospike/astools.conf
2020-01-30 17:16:12 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:12 INFO Add node BB9D80000000000 10.0.0.1:4333
2020-01-30 17:16:13 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:14 DEBUG Node BB9D80000000000 peers generation changed: 30
2020-01-30 17:16:14 DEBUG Node BB9D80000000000 partition generation changed: 2058
2020-01-30 17:16:14 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:15 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:16 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:18 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:19 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:20 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:21 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:22 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:23 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:24 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:25 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:27 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:28 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:29 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:30 DEBUG TLS name 'sg.example.com' matches
2020-01-30 17:16:31 INFO Add node BB9110000000000 10.0.0.2:4333
2020-01-30 17:16:31 INFO Add node BB9220000000000 10.0.0.3:4333
2020-01-30 17:16:31 INFO Add node BB9330000000000 10.0.0.4:4333
2020-01-30 17:16:31 INFO Add node BB9440000000000 10.0.0.5:4333
2020-01-30 17:16:31 INFO Add node BB9550000000000 10.0.0.6:4333
2020-01-30 17:16:31 INFO Add node BB9660000000000 10.0.0.7:4333
2020-01-30 17:16:31 INFO Add node BB9770000000000 10.0.0.8:4333
2020-01-30 17:16:31 INFO Add node BB9880000000000 10.0.0.9:4333
2020-01-30 17:16:31 INFO Add node BB9990000000000 10.0.0.10:4333
2020-01-30 17:16:31 INFO Add node BB9100000000000 10.0.0.11:4333
2020-01-30 17:16:31 INFO Add node BB9110000000000 10.0.0.12:4333
2020-01-30 17:16:31 INFO Add node BB9120000000000 10.0.0.13:4333
2020-01-30 17:16:31 INFO Add node BB9130000000000 10.0.0.14:4333
2020-01-30 17:16:31 INFO Add node BB9140000000000 10.0.0.15:4333
2020-01-30 17:16:31 INFO Add node BB9150000000000 10.0.0.16:4333
2020-01-30 17:16:31 DEBUG Node BB9110000000000 peers generation changed: 42
2020-01-30 17:16:31 DEBUG Node BB9110000000000 partition generation changed: 2786
2020-01-30 17:16:32 DEBUG Node BB9220000000000 peers generation changed: 39
2020-01-30 17:16:32 DEBUG Node BB9220000000000 partition generation changed: 2706
2020-01-30 17:16:32 DEBUG Node BB9330000000000 peers generation changed: 36
2020-01-30 17:16:32 DEBUG Node BB9330000000000 partition generation changed: 2447
2020-01-30 17:16:32 DEBUG Node BB9440000000000 peers generation changed: 54
2020-01-30 17:16:32 DEBUG Node BB9440000000000 partition generation changed: 3498
2020-01-30 17:16:32 DEBUG Node BB9550000000000 peers generation changed: 48
2020-01-30 17:16:32 DEBUG Node BB9550000000000 partition generation changed: 3153
2020-01-30 17:16:32 DEBUG Node BB9660000000000 peers generation changed: 51
2020-01-30 17:16:32 DEBUG Node BB9660000000000 partition generation changed: 3234
2020-01-30 17:16:33 DEBUG Node BB9770000000000 peers generation changed: 45
2020-01-30 17:16:33 DEBUG Node BB9770000000000 partition generation changed: 3028
2020-01-30 17:16:33 DEBUG Node BB9880000000000 peers generation changed: 57
2020-01-30 17:16:33 DEBUG Node BB9880000000000 partition generation changed: 3696
2020-01-30 17:16:33 DEBUG Node BB9990000000000 peers generation changed: 72
2020-01-30 17:16:33 DEBUG Node BB9990000000000 partition generation changed: 4448
2020-01-30 17:16:33 DEBUG Node BB9100000000000 peers generation changed: 66
2020-01-30 17:16:33 DEBUG Node BB9100000000000 partition generation changed: 4100
2020-01-30 17:16:34 DEBUG Node BB9110000000000 peers generation changed: 60
2020-01-30 17:16:34 DEBUG Node BB9110000000000 partition generation changed: 3695
2020-01-30 17:16:34 DEBUG Node BB9120000000000 peers generation changed: 69
2020-01-30 17:16:34 DEBUG Node BB9120000000000 partition generation changed: 4101
2020-01-30 17:16:34 DEBUG Node BB9130000000000 peers generation changed: 33
2020-01-30 17:16:34 DEBUG Node BB9130000000000 partition generation changed: 2207
2020-01-30 17:16:34 DEBUG Node BB9140000000000 peers generation changed: 75
2020-01-30 17:16:34 DEBUG Node BB9140000000000 partition generation changed: 5629
2020-01-30 17:16:34 DEBUG Node BB9150000000000 peers generation changed: 63
2020-01-30 17:16:34 DEBUG Node BB9150000000000 partition generation changed: 3666
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.1 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.2 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.3 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.4 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.5 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.6 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.7 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.8 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.9 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.10 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.11 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.12 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.13 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.14 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.15 4333
2020-01-30 17:16:41 DEBUG Add seed 10.0.0.16 4333
Aerospike Query Client
Version 3.23.0
C Client Version 4.6.9
Copyright 2012-2019 Aerospike. All rights reserved.
aql>

So how can apply the equivalent to that timeout in the python client?

pgupta · January 31, 2020, 12:30am

Are you using Enterprise Edition? If so, please open a ticket with Aerospike Support.

lucien · January 31, 2020, 9:22pm

Issue seems to be with missing connect_timeout setting. You should be able to use both login_timeout_ms (needed for auth) and connect_timeout

client_config = {
    "hosts": [("sg-1.example.com", 4333, "sg.example.com")],
    "policies": {
        "login_timeout_ms": 10000
    },
    "connect_timeout": 10000,
    "tls": {
        "enable": True,
        "cafile": "/path/to/ca.pem"
    }
}

We will update our api doc with info on connect_timeout

system · February 6, 2020, 9:23pm

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Seeing errors during connect	10	1406	July 9, 2017
Python Client - Connecting to db	3	2261	December 19, 2019
Increasing amount of i/o timeouts with last Aerospike Go client Go Client	5	3103	August 17, 2018
TLS authentication using Python client Python Client python	0	1714	May 3, 2017
Client timeout error on Ubuntu 14.04 AQL	4	3453	April 9, 2015

Timeout options are not respected

Related topics