Large memory usage by Python driver on big batch insert`

Thomas_Browne · February 2, 2017, 3:58pm

Okay I have raised an issue, which also includes some simplified code:

github.com/aerospike/aerospike-client-python

Possible memory leak in Python driver ordered map insert

opened 03:12PM - 02 Feb 17 UTC

closed 12:54AM - 11 Feb 17 UTC

Refers to this Community board entry: https://discuss.aerospike.com/t/large-me…mory-usage-by-python-driver-on-big-batch-insert/3819 There appears to be growing memory usage, an order of magnitude greater than the space occupied by the data itself, by the Python driver when inserting into an ordered map. Here is a simplified version of the code from the Community forum link above. This will insert 100 million floats, batched by a ticker code and an integer, into ordered maps. The memory used by this piece of code grows enormously as the inserts progress. After 10 million inserts, AMC shows Aerospike test db using only 208MB, whereas this piece of code's memory footprint has grown to 1.5GB. And it continues upwards from there. ### UPDATE: This appears only to affect the python 3 driver. Tested with version 2.0.6. This code will work with both python 2.7 and 3.x. Just need to install numpy and psutil. ``` from __future__ import print_function import aerospike import numpy as np from datetime import datetime import os import psutil if __name__ == "__main__": thisprocess = psutil.Process(os.getpid()) config = { 'hosts': [ ('localhost', 3000) ] } client = aerospike.client(config).connect() map_policy={'map_order':aerospike.MAP_KEY_VALUE_ORDERED} num_tickers = 1000 # number of individual securities num_periods = 24*30 # total_entries = int(100e6) entries_per_period = int(total_entries / num_periods) this_period = 0 timer = datetime.utcnow() counter = 0 # set the map policies for period in range(num_periods): for ticker in range(num_tickers): key = ("test", "tickermap", str(ticker) + ":" + str(period)) client.map_set_policy(key, "ticks", map_policy) # insert for period in range(num_periods): for entry in range(entries_per_period): ticker = str(np.random.choice(num_tickers)) + ":" + str(period) key = ("test", "tickermap", ticker) client.map_put(key, "ticks", this_period, np.random.uniform()) this_period += 1 counter += 1 if(datetime.utcnow() - timer).seconds > 0: print(counter, "per second, total insertions:", this_period, " GB used:",\ thisprocess.memory_info().rss / 1e9) counter = 0 timer = datetime.utcnow()

Topic		Replies	Views
Bucketing / splitting data	3	1812	October 27, 2021
Aerospike learning - writing many keys slow performance Python Client	10	1793	October 28, 2021
Aerospike requires a big virtual memory, where is the memory used? Tuning sizing	6	2838	January 13, 2016
Aerospike in-memory DB uses much memory than expected	23	7393	July 11, 2022
Aerospike using double memory	6	1448	May 15, 2020

Large memory usage by Python driver on big batch insert`

Related topics