Huge amount of memory usage in “newPartitions” and “NewConnection”


#1

Aerospike go client is using huge amount of memory, especially in “newPartitions” function (I don’t know what it is). How can i reduce memory usage. I am using only one “Client Objest” in golang code, which is initialized in main function.

Our cluster size is below.

  cluster: 1
  cluster nodes: 6
  machine-type: CPU 2cores, Memory 15GB

  ConnectionQueueSize = 512

  paxos-single-replica-limit 1
  pidfile /var/run/aerospike/asd.pid
  service-threads 8
  transaction-queues 8
  transaction-threads-per-queue 4
  proto-fd-max 30000
  batch-threads 8
~ # go tool pprof main http://localhost:6000/debug/pprof/heap
Fetching profile from http://localhost:6000/debug/pprof/heap
Saved profile in /root/pprof/pprof.main.localhost:6000.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz
Entering interactive mode (type "help" for commands)
(pprof) top
11383.14MB of 11600.15MB total (98.13%)
Dropped 875 nodes (cum <= 58MB)
Showing top 10 nodes out of 52 (cum >= 135.40MB)
      flat  flat%   sum%        cum   cum%
 7680.93MB 66.21% 66.21%  7680.93MB 66.21%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newPartitions
 3095.59MB 26.69% 92.90%  3100.05MB 26.72%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.NewConnection
  321.09MB  2.77% 95.67%   321.09MB  2.77%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.clonePartitions
  120.36MB  1.04% 96.71%   120.36MB  1.04%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newSingleConnectionQueue
   67.95MB  0.59% 97.29%    84.84MB  0.73%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*info).parseMultiResponse
   40.12MB  0.35% 97.64%   375.11MB  3.23%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Cluster).tend
   17.13MB  0.15% 97.78%    78.83MB  0.68%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*info).sendCommand
   16.90MB  0.15% 97.93%   167.02MB  1.44%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.parsePeers
   13.67MB  0.12% 98.05%   103.91MB   0.9%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newInfo
    9.40MB 0.081% 98.13%   135.40MB  1.17%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newNode
(pprof) list github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newPartitions
Total: 11.33GB
ROUTINE ======================== github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newPartitions in /go/src/github.com/my-app/vendor/github.com/aerospike/aerospike-client-go/partitions.go
    7.50GB     7.50GB (flat, cum) 66.21% of Total
 Error: open /go/src/github.com/my-app/vendor/github.com/aerospike/aerospike-client-go/partitions.go: no such file or directory
(pprof) list github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newPartitionParser
Total: 11.33GB

#2

How long did your program run before taking this profile?


#3

How long did your program run before taking this profile?

Less than 10 minutes.

Now, we’re performing a load test, which send 100-400 requests per seconds (I think it’s not so heavy for our aerospike cluster). (1 to 10 read/write to aerospike cluster per one http request).


#4

Do you know how big your records are? (This question is related to the big memory consumption for connections)


#5

Record size must be less than 1MB ( write-block-size = 1MB). It is 500KB to 1MB in the largest, and less than 1KB in the smallest in our performance test.

Number of records is about 120,000 = 20,000 * 6 node ( replica-factor = 3 ).

aql> show sets
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| disable-eviction | ns           | set-enable-xdr | objects | stop-writes-count | set                    | memory_data_bytes | truncate_lut | tombstones |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| "false"          | "ephemeral"  | "use-default"  | "4372"  | "0"               | "request"              | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "26"    | "0"               | "consumption"          | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "11"    | "0"               | "goal"                 | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "141"   | "0"               | "click_frequency"      | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "105"   | "0"               | "click"                | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5592"  | "0"               | "impression_frequency" | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5268"  | "0"               | "impression"           | "0"               | "0"          | "0"        |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
[127.0.0.1:3000] 7 rows in set (0.002 secs)

+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| disable-eviction | ns           | set-enable-xdr | objects | stop-writes-count | set                    | memory_data_bytes | truncate_lut | tombstones |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| "false"          | "ephemeral"  | "use-default"  | "26"    | "0"               | "consumption"          | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "9"     | "0"               | "goal"                 | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "4426"  | "0"               | "request"              | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5720"  | "0"               | "impression_frequency" | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5447"  | "0"               | "impression"           | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "150"   | "0"               | "click_frequency"      | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "106"   | "0"               | "click"                | "0"               | "0"          | "0"        |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
[10.52.82.3:3000] 7 rows in set (0.004 secs)

+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| disable-eviction | ns           | set-enable-xdr | objects | stop-writes-count | set                    | memory_data_bytes | truncate_lut | tombstones |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| "false"          | "ephemeral"  | "use-default"  | "4208"  | "0"               | "request"              | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "23"    | "0"               | "consumption"          | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "10"    | "0"               | "goal"                 | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5533"  | "0"               | "impression_frequency" | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "162"   | "0"               | "click_frequency"      | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "92"    | "0"               | "click"                | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5117"  | "0"               | "impression"           | "0"               | "0"          | "0"        |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
[10.52.81.3:3000] 7 rows in set (0.006 secs)

+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| disable-eviction | ns           | set-enable-xdr | objects | stop-writes-count | set                    | memory_data_bytes | truncate_lut | tombstones |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| "false"          | "ephemeral"  | "use-default"  | "22"    | "0"               | "consumption"          | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "4"     | "0"               | "goal"                 | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "4255"  | "0"               | "request"              | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5234"  | "0"               | "impression"           | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "110"   | "0"               | "click"                | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "154"   | "0"               | "click_frequency"      | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5585"  | "0"               | "impression_frequency" | "0"               | "0"          | "0"        |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
[10.52.84.3:3000] 7 rows in set (0.007 secs)

+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| disable-eviction | ns           | set-enable-xdr | objects | stop-writes-count | set                    | memory_data_bytes | truncate_lut | tombstones |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| "false"          | "ephemeral"  | "use-default"  | "27"    | "0"               | "consumption"          | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "10"    | "0"               | "goal"                 | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "4324"  | "0"               | "request"              | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5322"  | "0"               | "impression"           | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "157"   | "0"               | "click_frequency"      | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "110"   | "0"               | "click"                | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5695"  | "0"               | "impression_frequency" | "0"               | "0"          | "0"        |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
[10.52.86.3:3000] 7 rows in set (0.008 secs)

+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| disable-eviction | ns           | set-enable-xdr | objects | stop-writes-count | set                    | memory_data_bytes | truncate_lut | tombstones |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
| "false"          | "ephemeral"  | "use-default"  | "4490"  | "0"               | "request"              | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "20"    | "0"               | "consumption"          | "0"               | "0"          | "0"        |
| "false"          | "ephemeral"  | "use-default"  | "4"     | "0"               | "goal"                 | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5787"  | "0"               | "impression_frequency" | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "5439"  | "0"               | "impression"           | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "163"   | "0"               | "click_frequency"      | "0"               | "0"          | "0"        |
| "false"          | "persistent" | "use-default"  | "113"   | "0"               | "click"                | "0"               | "0"          | "0"        |
+------------------+--------------+----------------+---------+-------------------+------------------------+-------------------+--------------+------------+
[10.52.85.3:3000] 7 rows in set (0.010 secs)

#6

Which version of the client are you using?


#7

Aerospike’s version is 4.2.0.4 and client-go version is below.

[[constraint]]
  name = "github.com/aerospike/aerospike-client-go"                                                                                                                                                             
  version = "1.31.0"

#8

I found the issue comment on memory leak of aerospike-client-go below.

Plase tell me the detail of memory leak and versions. @khosrow


#9

I have switched go-client version to “1.24.0”, but the problem’s not solved yet.

I found most suspicious code which is consuming large memory. I guess “dataBuffer” has been growing. How can i handle this?

(the profile below is not in performance test enviroment, but local enviroment)

Total: 9212.04MB
ROUTINE ======================== github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.NewConnection in /Users/shuuky/src/github.com/my-app/vendor/github.com/aerospike/aerospike-client-go/connection.go
   3512.12MB    3512.23MB (flat, cum) 48.27% of Total
     .          .     80:// NewConnection creates a connection on the network and returns the pointer
     .          .     81:// A minimum timeout of 2 seconds will always be applied.
     .          .     82:// If the connection is not established in the specified timeout,
     .          .     83:// an error will be returned
     .          .     84:func NewConnection(address string, timeout time.Duration) (*Connection, error) {
   3512.12MB    3512.12MB     85:	newConn := &Connection{dataBuffer: make([]byte, DefaultBufferSize)}

And I was logging connection stats by “Client.Stat()”, below. ( “open-connections”:6 which is number of nodes. I guess it’s no problem about number of connections )

2018-09-21T14:25:40.872Z	INFO	my-app/main.go:50	6, <nil>, map[string]interface {}{"10.52.81.3:3000":map[string]interface {}{"node-removed-count":0, "tends-total":2, "tends-successful":2, "tends-failed":0, "partition-map-updates":1, "open-connections":1, "node-added-count":1, "connections-attempts":1, "connections-successful":1, "connections-failed":0, "connections-pool-empty":0}, "10.52.82.3:3000":map[string]interface {}{"connections-failed":0, "tends-total":2, "partition-map-updates":1, "node-added-count":1, "node-removed-count":0, "tends-failed":0, "connections-attempts":1, "connections-successful":1, "connections-pool-empty":0, "open-connections":1, "tends-successful":2}, "10.52.83.3:3000":map[string]interface {}{"connections-successful":1, "tends-total":2, "tends-failed":0, "partition-map-updates":1, "node-added-count":1, "connections-attempts":1, "connections-failed":0, "connections-pool-empty":0, "open-connections":1, "tends-successful":2, "node-removed-count":0}, "10.52.84.3:3000":map[string]interface {}{"connections-pool-empty":0, "open-connections":1, "tends-total":2, "node-added-count":1, "node-removed-count":0, "connections-attempts":1, "connections-successful":1, "connections-failed":0, "tends-successful":2, "tends-failed":0, "partition-map-updates":1}, "10.52.85.3:3000":map[string]interface {}{"node-added-count":1, "connections-attempts":1, "connections-successful":1, "connections-failed":0, "connections-pool-empty":0, "tends-successful":2, "tends-failed":0, "partition-map-updates":1, "open-connections":1, "tends-total":2, "node-removed-count":0}, "10.52.86.3:3000":map[string]interface {}{"connections-attempts":1, "connections-pool-empty":0, "open-connections":1, "tends-total":2, "partition-map-updates":1, "node-removed-count":0, "connections-successful":1, "connections-failed":0, "tends-successful":2, "tends-failed":0, "node-added-count":1}, "cluster-aggregated-stats":map[string]interface {}{"tends-failed":0, "node-removed-count":0, "connections-attempts":6, "connections-successful":6, "connections-pool-empty":0, "open-connections":6, "tends-total":12, "connections-failed":0, "tends-successful":12, "partition-map-updates":6, "node-added-count":6}, "open-connections":6}

Some more information please. @khosrow


#10

Can you try the latest code in master on Github and let me know how it goes? (it is not versioned, just use the latest commit)

That line of course will allocate a buffer, but the default size is 64KB. It does not look suspicious unless something else is going on. You can potentially have 512 * 6 of these connections, with each buffer potentially growing to up to 1 MB. That means 3GB of RAM right there.


#11

Thank you for your reply. @khosrow

I’ve switched client version to latest commit (“6b0509afa61”)

It seems the memory usage is better than before, but it’s still leaking memory and the amount of memory usage is not small. At the end of load test, the server process (k8s pod) was killed because of memory usage.

And it looks that the latency is higher (worse) than before, i guess.

profile is below.

.1. before the load test

$ go tool pprof -top http://localhost:6000/debug/pprof/heap
Fetching profile from http://localhost:6000/debug/pprof/heap
Saved profile in /root/pprof/pprof.main.localhost:6000.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
5.21MB of 5.23MB total (99.64%)
Dropped 574 nodes (cum <= 0.03MB)
      flat  flat%   sum%        cum   cum%
    2.76MB 52.73% 52.73%     2.76MB 52.73%  github.com/my-app/goal.init
    1.23MB 23.45% 76.19%     1.23MB 23.46%  github.com/my-app/kvs/async.init
    1.23MB 23.45% 99.64%     1.24MB 23.81%  github.com/my-app/vendor/github.com/my-app/lib/streaming.init
         0     0% 99.64%     5.23MB   100%  main.init
         0     0% 99.64%     5.23MB   100%  runtime.main

.2. in the middle of load test

$ go tool pprof -top http://localhost:6000/debug/pprof/heap
Fetching profile from http://localhost:6000/debug/pprof/heap
Saved profile in /root/pprof/pprof.main.localhost:6000.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz
3852706.60kB of 4017857.84kB total (95.89%)
Dropped 730 nodes (cum <= 20089.29kB)
     flat  flat%   sum%        cum   cum%
2741475.69kB 68.23% 68.23% 2741475.69kB 68.23%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newPartitions
766760.38kB 19.08% 87.32% 772758.55kB 19.23%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.NewConnection
79026.32kB  1.97% 89.28% 94148.16kB  2.34%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*info).parseMultiResponse
74885.24kB  1.86% 91.15% 74885.24kB  1.86%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newSingleConnectionHeap
34427.65kB  0.86% 92.00% 52499.54kB  1.31%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Cluster).tend
24009.87kB   0.6% 92.60% 114513.55kB  2.85%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newNode
22319.87kB  0.56% 93.16% 22319.87kB  0.56%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go/types.(*Message).Resize
18205.36kB  0.45% 93.61% 42636.24kB  1.06%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*peerListParser).readPeer
18150.29kB  0.45% 94.06% 36433.29kB  0.91%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go/types.(*Message).Serialize
17956.33kB  0.45% 94.51% 155866.06kB  3.88%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.parsePeers
15618.44kB  0.39% 94.90% 90503.68kB  2.25%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newConnectionHeap
15551.88kB  0.39% 95.28% 78069.96kB  1.94%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*info).sendCommand
12495.83kB  0.31% 95.60% 101176.94kB  2.52%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newInfo
6829.49kB  0.17% 95.77% 49465.72kB  1.23%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*peerListParser).readNodeList
2992.95kB 0.074% 95.84% 2752679.77kB 68.51%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*partitionParser).parseReplicasAll
1623.63kB  0.04% 95.88% 228485.14kB  5.69%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Cluster).seedNodes.func2
 101.99kB 0.0025% 95.88% 97528.91kB  2.43%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*nodeValidator).validateAlias
  92.28kB 0.0023% 95.89% 43580.37kB  1.08%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Cluster).clusterBoss
  58.35kB 0.0015% 95.89% 2756669.49kB 68.61%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Node).refreshPartitions
  45.55kB 0.0011% 95.89% 57923.19kB  1.44%  github.com/my-app/budget.add
  44.16kB 0.0011% 95.89% 2756598.77kB 68.61%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newPartitionParser
  22.49kB 0.00056% 95.89% 48632.71kB  1.21%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Client).AddBins

.3. near the end of load test (the process is killed)

$go tool pprof -top -base /root/pprof/pprof.main.localhost:6000.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz main  http://localhost:6000/debug/pprof/heap
Fetching profile from http://localhost:6000/debug/pprof/heap
Saved profile in /root/pprof/pprof.main.localhost:6000.alloc_objects.alloc_space.inuse_objects.inuse_space.008.pb.gz
9172.03MB of 9549.57MB total (96.05%)
Dropped 786 nodes (cum <= 47.75MB)
      flat  flat%   sum%        cum   cum%
 7458.48MB 78.10% 78.10%  7458.48MB 78.10%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newPartitions
  904.08MB  9.47% 87.57%   910.10MB  9.53%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.NewConnection
  202.17MB  2.12% 89.69%   202.17MB  2.12%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newSingleConnectionHeap
  193.93MB  2.03% 91.72%   229.65MB  2.40%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*info).parseMultiResponse
   83.35MB  0.87% 92.59%   129.11MB  1.35%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Cluster).tend
   64.94MB  0.68% 93.27%   309.30MB  3.24%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newNode
   43.75MB  0.46% 93.73%   102.32MB  1.07%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*peerListParser).readPeer
   43.20MB  0.45% 94.18%   375.21MB  3.93%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.parsePeers
   43.17MB  0.45% 94.63%    86.41MB   0.9%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go/types.(*Message).Serialize
   42.18MB  0.44% 95.07%   244.36MB  2.56%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newConnectionHeap
   36.86MB  0.39% 95.46%   179.75MB  1.88%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*info).sendCommand
   29.65MB  0.31% 95.77%   234.53MB  2.46%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.newInfo
   16.44MB  0.17% 95.94%   118.75MB  1.24%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*peerListParser).readNodeList
    8.14MB 0.085% 96.03%  7472.58MB 78.25%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*partitionParser).parseReplicasAll
    1.05MB 0.011% 96.04%   437.75MB  4.58%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Cluster).seedNodes.func2
    0.52MB 0.0055% 96.05%   107.21MB  1.12%  github.com/my-app/vendor/github.com/aerospike/aerospike-client-go.(*Cluster).clusterBoss


#12

This is interesting. Unfortunately I cannot trigger the leak myself so I’m working in the dark a bit. Do you think you can provide me with a snippet of code that can reproduce this? Something like a simple web app like your own that I can use to reproduce the problem?


#13

Did you make any progress?

I have confirmation from other users that the latest commit fixed the high memory consumption problem for them. To make sure you got it right, can you check if you in fact have the latest commit, and your vendor version is the same as that commit. It would help to compile and build your app via -a flag to make sure go recompiles all dependencies from scratch.


#14

We are still waiting for your confirmation to move forward on this issue. Do you have anything new for us?


#15

Thank you for your reply. @khosrow

I’m sorry, I’m too busy to confirm more detail now.

I’ve decided (as a product manager) to use memcached sharding by go-client library instead of aerospike for the production release. though I’d like to use aerospike in a long term.

So I will confirm this issue in a few month (because I’m busy on my buisiness now).

P.S.

The profile above must be the build by latest commit (“6b0509afa61”).


#16

I have confirmed the latest commit (“6b0509afa61”) is working very good. @khosrow

Before, we build aerospike’s servers on three availability zones (zone a to c) on GCP k8s. There were two nodes on each zone and the cluster was separeted on multi AZ. And we would often got connection (tcp) timeout and huge memory useage.

Now we’ve build aerospike’s server on ComputeEngine on GCP on the same single zone. So that we’ve got very low latency with no connection timeout nor huge memory usage too.

I guess that we cannot build the cluster on multi AZ and that the increase of connection was causing unexpected amounts of memory usage.


#17

Good to hear. Thanks for letting us know.

This confirms our own conclusions that connection timeouts (especially on cloud platforms) are extremely expensive, and should be avoided as much as possible.