Index-type flash - Error Code 1: Server error - No space left on device

Operations Configuration
1

Hi! I am trying to configure an index-type flash, and it’s complicated. Could you please explain what’s wrong?

namespace test {
 replication-factor 1
 default-ttl 0d
 nsup-period 600

 partition-tree-sprigs 8K

    storage-engine device {
        file /opt/aerospike/data/test.dat
        filesize 8G
    }

    index-type flash {
	mount /mnt/flash
	mounts-budget 8G
    }
}

lsblk:

nvme0n1     259:0    0 953.9G  0 disk 
├─nvme0n1p1 259:1    0 232.8G  0 part /var/snap/firefox/common/host-hunspell
│                                     /
├─nvme0n1p2 259:2    0     1G  0 part /boot/efi
└─nvme0n1p3 259:3    0   9.3G  0 part /mnt/flash

After few seconds inserting I got Error while writing key xxx: com.aerospike.client.AerospikeException: Error Code 1: Server error

Here is no space

/mnt/flash# ll
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001100
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001101
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001102
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001103
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001104
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001105
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001106
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001107
-rw------- 1 root root 1073741824 Oct 28 08:27 ae001108

But used just few percent of spaces

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Usage Information (2024-10-28 08:28:12 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|     Node|Evictions|  Stop|~System~|~~~~~Primary Index~~~~~|~~Secondary~~~|~~~~~~~~~~~~~~~~~~~Storage Engine~~~~~~~~~~~~~~~~~~
         |         |         |Writes|~Memory~| Type|      Used|Evict%|~~~~Index~~~~~|  Type|      Used| Used%|Evict%|  Used|Avail%|Avail
         |         |         |      |  Avail%|     |          |      | Type|    Used|      |          |      |      | Stop%|      |Stop%
test     |host:3000|  0.000  |False |      79|flash|149.299 MB| 0.0 %|shmem|0.000 B |device|410.572 MB|5.01 %| 0.0 %|70.0 %|94.0 %|5.0 %
test     |         |  0.000  |      |        |     |149.299 MB|      |     |0.000 B |      |410.572 MB|5.01 %|      |      |      |     
Number of rows: 1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2024-10-28 08:28:12 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace|     Node|Rack|  Repl|Expirations|  Total|~~~~~~~~~~Objects~~~~~~~~~~|~~~~~~~~~Tombstones~~~~~~~~|~~~~Pending~~~~
         |         |  ID|Factor|           |Records| Master|  Prole|Non-Replica| Master|  Prole|Non-Replica|~~~~Migrates~~~
         |         |    |      |           |       |       |       |           |       |       |           |     Tx|     Rx
test     |host:3000|   0|     1|    0.000  |2.446 M|2.446 M|0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  
test     |         |    |      |    0.000  |2.446 M|2.446 M|0.000  |    0.000  |0.000  |0.000  |    0.000  |0.000  |0.000  
Number of rows: 1

In log

 WARNING (os): (os.c:208) failed min-free-kbytes check - min-free-kbytes should be at least 1150976
 WARNING (os): (os.c:205) failed swappiness check - swappiness not set to 0
 WARNING (os): (os.c:208) failed rmem-max check - rmem-max should be at least 15728640
 WARNING (os): (os.c:208) failed wmem-max check - wmem-max should be at least 5242880
 WARNING (config): (cfg.c:4806) failed best-practices checks - see 'https://docs.aerospike.com/docs/operations/install/linux/bestpractices/index.html'
 WARNING (arenax): (arenax_ee.c:437) could not allocate 1073741824-byte arena stage 9: No space left on device
 WARNING (index): (index.c:659) arenax alloc failed
 WARNING (arenax): (arenax_ee.c:437) (repeated:101) could not allocate 1073741824-byte arena stage 9: No space left on device
 WARNING (index): (index.c:659) (repeated:100) arenax alloc failed
2

It is behaving as if the mount is either full.:

df -h /mnt/flash

Or possibly a permission issues between asd and /mnt/flash.

ls -lah /mnt/flash
ps -ef | grep $(pgrep asd)
3

Yes, /mnt/flash becomes full within a few seconds after starting the insertion. This is a problem because the amount of inserted data is very small - 2.446 M records * 64 byte = ~156 MB, but it takes 10 GB.

I am trying to keep data in

/opt/aerospike/data/test.dat

and the index here

/mnt/flash

df -h /mnt/flash

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p3  9.3G  214M  9.1G   3% /mnt/flash

ps -ef | grep $(pgrep asd)

root     1593173       1  0 20:55 ?        00:00:01 /usr/bin/asd --config-file /etc/aerospike/aerospike.conf --fgdaemon
4

In all flash, each PI belongs to an intial 4K sprig allocation. So, as you add records, you have to not see it as 64 byte allocation, but as if you are allocating all the sprigs, as records will be spread across all the sprigs. Once all sprigs have their first record, you could add 63 additional records in that sprig and there will be no usage increase. Check sizing per this: Capacity planning guide | Aerospike Documentation

5

Ah, you have configured 8K sprigs per partition - please see: Capacity planning guide | Aerospike Documentation

Assuming 1 nodes cluster:

There are 4K partitions * 8K sprigs per partition * 4K page size per sprig = 128G.

The most sprigs that you could configure for this index size is 512.