Large memory usage by Python driver on big batch insert`

Thomas_Browne · February 2, 2017, 3:58pm

Okay I have raised an issue, which also includes some simplified code:

github.com/aerospike/aerospike-client-python

Possible memory leak in Python driver ordered map insert

opened 03:12PM - 02 Feb 17 UTC

closed 12:54AM - 11 Feb 17 UTC

Refers to this Community board entry: https://discuss.aerospike.com/t/large-me…mory-usage-by-python-driver-on-big-batch-insert/3819 There appears to be growing memory usage, an order of magnitude greater than the space occupied by the data itself, by the Python driver when inserting into an ordered map. Here is a simplified version of the code from the Community forum link above. This will insert 100 million floats, batched by a ticker code and an integer, into ordered maps. The memory used by this piece of code grows enormously as the inserts progress. After 10 million inserts, AMC shows Aerospike test db using only 208MB, whereas this piece of code's memory footprint has grown to 1.5GB. And it continues upwards from there. ### UPDATE: This appears only to affect the python 3 driver. Tested with version 2.0.6. This code will work with both python 2.7 and 3.x. Just need to install numpy and psutil. ``` from __future__ import print_function import aerospike import numpy as np from datetime import datetime import os import psutil if __name__ == "__main__": thisprocess = psutil.Process(os.getpid()) config = { 'hosts': [ ('localhost', 3000) ] } client = aerospike.client(config).connect() map_policy={'map_order':aerospike.MAP_KEY_VALUE_ORDERED} num_tickers = 1000 # number of individual securities num_periods = 24*30 # total_entries = int(100e6) entries_per_period = int(total_entries / num_periods) this_period = 0 timer = datetime.utcnow() counter = 0 # set the map policies for period in range(num_periods): for ticker in range(num_tickers): key = ("test", "tickermap", str(ticker) + ":" + str(period)) client.map_set_policy(key, "ticks", map_policy) # insert for period in range(num_periods): for entry in range(entries_per_period): ticker = str(np.random.choice(num_tickers)) + ":" + str(period) key = ("test", "tickermap", ticker) client.map_put(key, "ticks", this_period, np.random.uniform()) this_period += 1 counter += 1 if(datetime.utcnow() - timer).seconds > 0: print(counter, "per second, total insertions:", this_period, " GB used:",\ thisprocess.memory_info().rss / 1e9) counter = 0 timer = datetime.utcnow()

Topic		Replies	Views
Aerospike learning - writing many keys slow performance Python Client	10	1843	October 28, 2021
Memory usage keeps increasing without increase in keys	13	2933	April 24, 2019
Bucketing / splitting data	3	1855	October 27, 2021
Bug leak memory and lost data on Aerospike	8	1358	March 4, 2020
Aerospike memory usage spikes up linearly Operations shared-memory	7	2134	October 5, 2022

Large memory usage by Python driver on big batch insert`

Related topics