Aerospike - error while fetching build version: EOF (invalid memory address or nil pointer dereference)

Hey! I am using Aerospike Server Community Edition on Kubernetes, app version: 5.5.0.7

about 20h after deployment I get the following error from the pod:

Error while fetching build version: EOF. Retrying. panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x6ad1ab]

goroutine 1 [running]: github.com/aerospike/aerospike-client-go.(*Connection).updateDeadline(0xc000123040, 0xc000096150, 0x8a05b8) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/connection.go:262 +0xab github.com/aerospike/aerospike-client-go.(*Connection).Write(0xc000123040, 0xc00008e900, 0xe, 0x16, 0xc000090020, 0xc000159c50, 0xa9ca20) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/connection.go:158 +0x48 github.com/aerospike/aerospike-client-go.(*info).sendCommand(0xc000090020, 0xc000123040, 0x6, 0xc00009a070) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/info.go:68 +0x46 github.com/aerospike/aerospike-client-go.newInfo(0xc000123040, 0xc000159e10, 0x1, 0x1, 0x1, 0xc000020190, 0xc0000aa000) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/info.go:48 +0x1c7 github.com/aerospike/aerospike-client-go.RequestInfo(0xc000123040, 0xc000159e10, 0x1, 0x1, 0xc000159df0, 0x1, 0x1) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/info.go:57 +0x4d main.performLivenessProbeOp(0x81923b, 0x9, 0x817d9c, 0x5, 0x817d9c, 0x5) /go/src/github.com/aerospike/aerospike-kubernetes-init/aerospike-utility/aku-adm.go:179 +0x12b main.performOperation(0x7fffc5a59ce9, 0x8, 0x81923b, 0x9) /go/src/github.com/aerospike/aerospike-kubernetes-init/aerospike-utility/aku-adm.go:97 +0x1b8 main.main() /go/src/github.com/aerospike/aerospike-kubernetes-init/aerospike-utility/aku-adm.go:54 +0x245

when I restart the pods they become healthy again for an about 20 hours

my namespace configs are:

    namespace test {
    replication-factor 2
    memory-size 1G
    default-ttl 2000
    nsup-period 1000
    high-water-memory-pct 60

}

and all the data in memory

I am still not sure what causing this and be happy to get some advice

tnx

Engineering has requested logs from Kubernetes Operator to figure out what is happening. You may also try upgrading the Go client as some fixes there may be relevant.

the logs I see before the problem occur are:

Jul 24 2021 04:23:25 GMT: WARNING (service): (service.c:429) (repeated:4021) refusing client connection - proto-fd-max 15000 Jul 24 2021 04:23:25 GMT: WARNING (hb): (hb.c:4903) (repeated:7) could not create heartbeat connection to node - {172.31.216.163:3002} Jul 24 2021 04:23:25 GMT: WARNING (socket): (socket.c:869) (repeated:7) Error while connecting: 111 (Connection refused) Jul 24 2021 04:23:25 GMT: WARNING (socket): (socket.c:928) (repeated:7) Error while connecting socket to 172.31.216.163:3002

Looks like you have hit the maximum number of client connections allowed (see the proto-fd-max configuration). This typically means that either the server is not configured correctly for the expected number of client connections, or that there is a problem slowing down the communication between client and server (could be connectivity issue on a server node itself) causing the client apps to start churning connections and overwhelming the server. Digging deeper would require the full Aerospike server logs and potentially do a latency analysis.

© 2021 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.