Can reads occur concurrently in Aerospike?


#1

It seems like Aerospike reads block each other. What’s the reason for this? And are read-only Scans the same as normal reads, but occur without affecting baseline database transactions?


What should I know before going with UDF?
#2

Do you mean reads of the same record (that is, identified by the same key)? Or do you mean concurrent reads of different records?

Aerospike is a multi-node, multi-core, multi-threaded system. Service threads pick up incoming operations, placing them on one of several transaction queues (often one of those is defined per-core). Those operations are then handled by transaction threads running on different cores, in parallel.

At the record level, operations are serialized if they happen concurrently. Read locks are slightly lighter than write latches. The number of operations queued up on a single record is controlled by the transaction-pending-limit config parameter, and has a default depth of 20. If that is exceeded, your client will get an exception with error code 14 (key busy). If this queue is frequently exceeded you have a “hot key” situation.

The way you handle a hot key depends on your business logic, and operational limits. You can temporarily raise the config param’s value, but it applies to the entire server node. You can change the read.replica policy to read from any of the replicas, instead of just the master, distributing the read load. You can also partition the key on the application side.


#3

I meant reads of the same record.

So what exactly are read-only Scans? And what is meant by they “occur without affecting baseline database transactions”? Does it mean they read without locking the record?


#4

Scans are handled by scan threads, which are typically configured to a value matching the number of cores. If you go higher, you will be affecting the regular CRUD transactions.

Still, a scan will get a record lock in order to avoid reading it as it’s being updated.


#5

Yes, but will scans occur concurrently if a record is locked form a read?


#6

Scans are happening concurrently on all the scan threads. Scans walk one partition at a time, reading the records within. Each read operation obtains the read lock, which is lighter than obtaining the write latch.


#7

So in aerospike, only the read requests with the same key will still block each other? Are they placed in the same transaction queue and processed by the same transaction thread? Can you show more design details about the fact:“At the record level, operations are serialized if they happen concurrently” ?


#8

Operations get pulled off the NIC by service threads, which then either try to directly access the record (if it’s stored in an in-memory namespace) or get placed on one of the transaction queues (if the data is in a namespace that stores its record on disk). There should be a transaction queue per-core, and each of these transaction queues has a small pool of transaction threads that go to the SSD.

The record queue is something different. When two or more threads try to access the same record (meaning they all have the same key), those operations will queue up for that specific record. This queue is per-record, and its size is controlled by the transaction-pending-limit config param.

Because Aerospike is multi-node / multi-core / multi-threaded there could be two or more threads trying to access the record concurrently. The read operations obtain a lock, the write operations a latch, to ensure isolation. Any operation that doesn’t have the lock or latch queues up, and will in turn be able to get exclusive access while it’s executing.