Is there a 'Many' functionality for writes etc?


#1

Hi.

So I’m aware we have the $db->getMany() and $db->existsMany() functions but is there anything like $db->writeMany() or anything like that? We are currently writing a number of keys using a foreach loop and when I’m profiling I don’t like the fact that we’re making 100+ calls to Aerospike when I’d hope we could just make one and that would be it.

in Redis we use the pipeline functionality so I’m wondering if Aerospike has something similar so my profiling will show 1 call to Aerospike (and reduce any possible issues)

Many thanks


#2

Hey Craig,

There is no batch-write capability, at the moment. At the same time, pipelining only works in Redis as long as it’s a single node, or in a sharded situation, as long as all the keys are on the same shard. Redis isn’t designed as a distributed database, so all that ad-hoc clustering comes with serious limitations.

That said, what are you doing here? Why do you have 100 writes in row? I’m trying to figure out if this is a maintenance kind of operation. The PHP client is limited to a single write at a time, because of the single execution thread nature of PHP. In clients such as the C, Java, and Go ones, you can do those in parallel with multiple threads.


#3

Hey @rbotzer, thanks for the reply.

AFAIK the issue with pipelining in Clustering has been resolved? If not, do you have anywhere I can look this stuff up. We currently don’t use clustering but are looking at it immediately due to platform instability in certain areas.

With regards to what we’re doing, currently it’s writing a data set based on analytical tracking as opposed to maintenance, so basically bean counting a number of keys. I don’t mind the number of writes per se, that doesn’t bother me, but just whether there was a way to throw an array of $key => $values at the Aerospike client and have the system level library do the foreach loop instead of PHP. So let’s say at the moment I have something like this:

// This array will have data in it like [$id => ['key' => $key, 'bin' => $bin, 'data' => $data]] x 100 etc just for demo purposes
$dataArray = [<array of stuff>];
$ttl = '<whatever>';
$options = [];

foreach($dataArray as $id => $dataToWrite) {
    $status = $this->aerospike->put($dataToWrite['key'], [$dataToWrite['bin'] => $dataToWrite['data']], $ttl, $options);
}

As you can see the code above is calling put() however many times (use 100 as an example). Ideally, I’d love to be able to do something like:

// This array will have data in it like [$id => ['key' => $key, 'bin' => $bin, 'data' => $data]] x 100 etc just for demo purposes
$dataArray = ['<array of stuff>'];
$ttl = '<whatever>';
$options = [];

// $status could return an array matching the items that were sent indicating their success or failure
// Obviously putMany() would expect an array in a specific format
$status = $this->aerospike->putMany($dataArray);

// Check status and do something on failure / success etc...

Ignore the coding, it’s just to clarify a point really.

That way, in profiling PHP would only make 1 call to the putMany() function and the library would handle the multiple writes instead of PHP calling it over and over. I hope that makes sense?

Ultimately it would avoid this:


#4

Hello,

I’m running into the same kind of problem. I have to write many records at the same time and I’d like a solution to avoid calling the API hundreds of times in a row…

Thanks in advance.