Bad memory using in Lists

1

Hello!

I heard a lot of good words by aerospike. That Aerospike eat much less memory than Redis. But I have another remarks.

Redis 2Gb Ram vs Aero 2Gb Ram. In redis I use lists, z operations, set-get and much more. In aero I use just lists.

So Redis with 2Gb RAM with 80000 elements in list eat 700MB. Ans Aero with 2Gb RAM with 4500 elements in list eat 1800MB. Aero namespace: write-block 8M, data-in-memory false. 3 dbs with repl factor 3.

Why so?

2

That sounds odd, and I’m no redis expert - but I feel like we’re missing a lot of data here. Things that would be helpful are; Data Model, code snippets, versions of software (client/server), how you’re measuring this usage, and example data snippet if possible.

1 Like
3

Particularly, since data-in-memory is false, are you comparing Redis memory usage to Aerospike disk usage?

If you are comparing memory then since these aren’t in memory, you must be comparing Aerospike’s various static allocations that support clustering/fabric/etc and the 64 bytes per record in the primary index. Aerospike does pre-allocate a fair amount of memory for various internal tasks but this isn’t an issue for most deployments since we don’t really target tiny data-sets like Redis; if the amount of pre-allocated RAM is an issue we can help you configure some of that down.

4

Aerospike 4.5.0.3, 4.5.0.3, 4.5.1.5 I use php-aerospike module. Just add in list simple records from 40 to 105 symbols. Measure - [free -m] in ubuntu Memory increase after adding records. So if I dont add records - memory stops increase.

Code:

$a_key = 'xxxyyyy';
$aerokey_push = $dbaero->initKey(NAMEDB, 'abc', $a_key);
$options = [Aerospike::OPT_POLICY_EXISTS => Aerospike::POLICY_EXISTS_IGNORE, Aerospike::OPT_TTL => 5184000];

$operations_push = array(
	array(
		"op" 	=> Aerospike::OPERATOR_WRITE, 
		"bin" 	=> "key", 
		"val" 	=> $a_key
	),
	array(
		"op" 	=> Aerospike::OP_LIST_APPEND, 
		"bin" 	=> "queue", 
		"val" 	=> 'text from 20 to 128 symbols'
	)
);

$aero_push = $dbaero->operateOrdered($aerokey_push, $operations_push, $ans, 
$options);

Yes I compare exactly RAM! And in Aerospike I use data-in-memory false. I need to store in SSD. But exactly RAM is higher than in Redis.

So whats wrong with it?

5

Could you provide the output of free -m as well as asadm -e "show stat like memory_used" before and after adding these lists.

6

I cant provide, because memory was 100% and aerospike was down with not available memory.

In free -m I measure this one metric. free -m | awk ‘/^Mem:/{print $3}’

                 total        used        free      shared  buff/cache   available
Mem:            985         240         132           0         612         584

There is 240 this metric. It is not possible to measure asadm, but I think it is not necessary because aerospike was down, and logs shows that exactly aerospike havent got enough memory.

Is it possible this problem is because OPT_TTL ?

7

By default the primary index incrementally allocates 1 GiB slabs.

You can configure this down (minimum is 128 MiB) with the index-stage-size namespace configuration option. The configuration is required to be a power of 2. For 128 MiB it would look something like:

namespace test {
    ...
    index-stage-size 128M
    ...
    storage-engine ssd {
        ...
    }
}