Aerospike - error while fetching build version: EOF (invalid memory address or nil pointer dereference)

1

Hey! I am using Aerospike Server Community Edition on Kubernetes, app version: 5.5.0.7

about 20h after deployment I get the following error from the pod:

Error while fetching build version: EOF. Retrying. panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x6ad1ab]

goroutine 1 [running]: github.com/aerospike/aerospike-client-go.(*Connection).updateDeadline(0xc000123040, 0xc000096150, 0x8a05b8) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/connection.go:262 +0xab github.com/aerospike/aerospike-client-go.(*Connection).Write(0xc000123040, 0xc00008e900, 0xe, 0x16, 0xc000090020, 0xc000159c50, 0xa9ca20) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/connection.go:158 +0x48 github.com/aerospike/aerospike-client-go.(*info).sendCommand(0xc000090020, 0xc000123040, 0x6, 0xc00009a070) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/info.go:68 +0x46 github.com/aerospike/aerospike-client-go.newInfo(0xc000123040, 0xc000159e10, 0x1, 0x1, 0x1, 0xc000020190, 0xc0000aa000) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/info.go:48 +0x1c7 github.com/aerospike/aerospike-client-go.RequestInfo(0xc000123040, 0xc000159e10, 0x1, 0x1, 0xc000159df0, 0x1, 0x1) /go/pkg/mod/github.com/aerospike/aerospike-client-go@v4.1.0+incompatible/info.go:57 +0x4d main.performLivenessProbeOp(0x81923b, 0x9, 0x817d9c, 0x5, 0x817d9c, 0x5) /go/src/github.com/aerospike/aerospike-kubernetes-init/aerospike-utility/aku-adm.go:179 +0x12b main.performOperation(0x7fffc5a59ce9, 0x8, 0x81923b, 0x9) /go/src/github.com/aerospike/aerospike-kubernetes-init/aerospike-utility/aku-adm.go:97 +0x1b8 main.main() /go/src/github.com/aerospike/aerospike-kubernetes-init/aerospike-utility/aku-adm.go:54 +0x245

when I restart the pods they become healthy again for an about 20 hours

my namespace configs are:

    namespace test {
    replication-factor 2
    memory-size 1G
    default-ttl 2000
    nsup-period 1000
    high-water-memory-pct 60

}

and all the data in memory

I am still not sure what causing this and be happy to get some advice

tnx

2

Engineering has requested logs from Kubernetes Operator to figure out what is happening. You may also try upgrading the Go client as some fixes there may be relevant.

3

the logs I see before the problem occur are:

Jul 24 2021 04:23:25 GMT: WARNING (service): (service.c:429) (repeated:4021) refusing client connection - proto-fd-max 15000 Jul 24 2021 04:23:25 GMT: WARNING (hb): (hb.c:4903) (repeated:7) could not create heartbeat connection to node - {172.31.216.163:3002} Jul 24 2021 04:23:25 GMT: WARNING (socket): (socket.c:869) (repeated:7) Error while connecting: 111 (Connection refused) Jul 24 2021 04:23:25 GMT: WARNING (socket): (socket.c:928) (repeated:7) Error while connecting socket to 172.31.216.163:3002

4

Looks like you have hit the maximum number of client connections allowed (see the proto-fd-max configuration). This typically means that either the server is not configured correctly for the expected number of client connections, or that there is a problem slowing down the communication between client and server (could be connectivity issue on a server node itself) causing the client apps to start churning connections and overwhelming the server. Digging deeper would require the full Aerospike server logs and potentially do a latency analysis.

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.