Adding a new node does not decrease the system memory usage

1

I added one new node to a cluster for the purpose of lowering the memory used per node, however, after the node is added, the free namespace memory for the existing nodes did increase, but the free system memory for the existing nodes are staying the same, I am wondering what is going on within the Aerospike? Is Aerospike holding this memory for future data?

This is the system memory info:

              total        used        free      shared  buff/cache   available
Mem:      259999656   209928584    12126304    36248184    37944768     8061072
Swap:             0           0           0

And the namespace memory info:

Namespace|            Node|    Total|Expirations|Evictions|  Stop|~Disk~|~~~~~~~~~~~Memory~~~~~~~~~~|~Primary~
         |                |  Records|           |         |Writes|  HWM%|      Used|Used%|HWM%|Stop%|~~Index~~
         |                |         |           |         |      |      |          |     |    |     |     Type
xxx      |aaa.bbb.ccc.ddd |488.466 M|  604.144 M|  0.000  |False |     0|177.643 GB|   81| 100|  100|shmem
xxx      |aaa.bbb.ccc.ddd |488.737 M|    4.664 G|  0.000  |False |     0|177.173 GB|   81| 100|  100|shmem
xxx      |aaa.bbb.ccc.ddd |488.785 M|    4.665 G|  0.000  |False |     0|177.188 GB|   81| 100|  100|shmem
xxx      |aaa.bbb.ccc.ddd |488.336 M|    4.666 G|  0.000  |False |     0|177.041 GB|   81| 100|  100|shmem
xxx      |aaa.bbb.ccc.ddd |488.539 M|    4.662 G|  0.000  |False |     0|177.078 GB|   81| 100|  100|shmem
xxx      |                |  2.443 G|   19.261 G|  0.000  |      |      |886.124 GB|     |    |     |

And this is the namespace configuration:

namespace xxx {
	replication-factor 2
	memory-size 220G

	nsup-period 5m
	default-ttl 1825D

	high-water-memory-pct 100

	storage-engine memory

	stop-writes-pct 100

	transaction-pending-limit 500

	single-scan-threads 8

	conflict-resolution-policy last-update-time
}

I also have the metrics from AMS:

image
image1920Ă—493 38.8 KB

2

Adding a node will not decrease the memory usage by Aerospike on the remaining nodes. The parameter “memory-size” determines the maximum memory allocated by Aerospike. Aerospike requests memory as required in chunks, but any “freed up” namespace memory (in this case, by adding a node) is not returned to the system. This is the behavior you are observing.

1 Like
3

But why is the system memory usage increase at a faster rate than the namespace memory usage? Is Aerospike gradually trying to reserve all memory set in the namespace configuration even if the namespace memory usage is below 70%?

4

And this time it went OOM but the namespace memory usage stayed quite constant, is there a potential memory leak issue? This behavior occurred after we added a new node

5

To understand and explain the behavior you are seeing, more details need to be examined. I believe you are using the Enterprise version (the primary index type being shmem), and so I would suggest opening a Support case to resolve this issue.

6

It could be due to some fragmentation as you are using a storage-engine memory configuration. The heap memory line in the logs may give you some hints. My response on this topic: Aerospike in-memory DB uses much memory than expected - #23 by helper_bro may help. The memory allocated for the primary index would indeed not be freed up without a restart but the memory allocated for the data (since you are data in memory) should be freed up but you could get some fragmentation…

1 Like
7

This topic was automatically closed 84 days after the last reply. New replies are no longer allowed.