Parallel reads gives "Operation not allowed at this time"

Operations
, ,
#1

I am trying to make 50 concurrent read requests. But it fails with the following error stack trace:

AerospikeError: Operation not allowed at this time.
    at Function.fromASError (/Users/.../node_modules/aerospike/lib/error.js:113:21)
    at QueryCommand.convertError (/Users/.../node_modules/aerospike/lib/commands/command.js:91:27)
    at QueryCommand.convertResponse (/Users/.../node_modules/aerospike/lib/commands/command.js:101:24)
    at asCallback (/Users/.../node_modules/aerospike/lib/commands/command.js:163:24)

I have put a detailed question here.

How can I check what are my limits of concurrent reads and fix this?

#2

On running asinfo -v "get-config:context=service;scan-threads-limit" I get:

paxos-single-replica-limit=1; pidfile=/var/run/aerospike/asd.pid; proto-fd-max=15000; advertise-ipv6=false; auto-pin=none; batch-index-threads=6; batch-max-buffers-per-queue=255; batch-max-requests=5000; batch-max-unused-buffers=256; batch-without-digests=false; cluster-name=null; disable-udf-execution=false; downgrading=false; enable-benchmarks-fabric=false; enable-health-check=false; enable-hist-info=false; info-threads=16; keep-caps-ssd-health=false; log-local-time=false; log-millis=false; microsecond-histograms=false; migrate-fill-delay=0; migrate-max-num-incoming=4; migrate-threads=1; min-cluster-size=1; node-id=BB9020011AC4202; node-id-interface=null; proto-fd-idle-ms=0; proto-slow-netio-sleep-ms=1; query-batch-size=100; query-buf-size=2097152; query-bufpool-size=256; query-in-transaction-thread=false; query-long-q-max-size=500; query-microbenchmark=false; query-pre-reserve-partitions=false; query-priority=10; query-priority-sleep-us=1; query-rec-count-bound=18446744073709551615; query-req-in-query-thread=false; query-req-max-inflight=100; query-short-q-max-size=500; query-threads=6; query-threshold=10; query-untracked-time-ms=1000; query-worker-threads=15; run-as-daemon=true; scan-max-done=100; scan-threads-limit=64; service-threads=6; sindex-builder-threads=4; sindex-gc-max-rate=50000; sindex-gc-period=10; stay-quiesced=false; ticker-interval=10; transaction-max-ms=1000; transaction-retry-ms=1002; vault-ca=null; vault-path=null; vault-token-file=null; vault-url=null; work-directory=/opt/aerospike; debug-allocations=none; indent-allocations=false

If this helps.

#3

Have you tried increasing the scan-threads-limit? Per the details on stackoverflow, those are not read requests… but full scans…

#4

Thanks for replying @meher. I have set the config to scan-threads-limit=64. But I am not able to run more than 20 parallel requests on a set which only has 60 records.

Also, could you please guide me how this is a full scan? And how I can fix it??

Much Thanks!

#5

Are you waiting for the 20 parallel ones to complete prior to submitting more? When scanning a set, no matter how small or large the set is, the whole namespace needs to be checked. You may want to put small sets in a smaller namespace. Also, Aerospike may release a new feature in the next release in Q2 that would have a new mechanism to very efficiently scan small sets even in large namespaces.

#6

Are you waiting for the 20 parallel ones to complete prior to submitting more?

No, but if I try to make more than 20 requests. App crashes with the error mentioned on top.

You may want to put small sets in a smaller namespace.

In future I will have around 10K records where I would need to filter results based on predicates mentioned. How should I approach that? Also, will Aerospike give good throughput in that?
Note: Right now I have just 60 records in a set and my queries are taking 0.3sec/fetch.

#7

When scanning a set, no matter how small or large the set is, the whole namespace needs to be checked.

Agreed, but indexing will not help at all here?

#8

Hi @meher, could you suggest here? Thanks.

#9

You could certainly create a secondary index on a specific bin that you add for this set and issue a secondary index query rather than a scan. This should definitely be faster than a scan as you would directly get to the relevant record without wasting time checking for whether a record belongs to the set that needs to be processed.

#10

Sure Meher. But I have to filter records based on 3 bins.

To give more context:
Using Aerospike.filter.contains(myBin, binValue, Aerospike.indexType.none); gives me good performance but it only allows me to add only one filter.

To add more filters I used predexp but it brings down the performance significantly. Secondary indexes present on all required bins. As I understand this is happening because this is a full scan.

How can I achieve performance here given I cannot create composite keys using append method? I will be querying on integer ranges as well.

#11

Currently you may use only one secondary index in a query. For better performance, combine a query filter with a (predicate) expression - with the filter on the most selective bin condition, and the expression for the remaining condition.

#12

Hi @neelp, I am not sure if I fully understand your suggestion.

For better performance, combine a query filter with a (predicate) expression

Here, do you want me to use Aerospike.filter and add one more predexp(on most selective bin)?
I have tried this and does not give good performance.

#13

Take the most selective clause in the overall condition, and use the secondary index (through query filter) for that if you can, and use expressions (or predexp, which are deprecated) for the remaining clauses.

#14

Thanks for replying @neelp. I tried out the approach you mentioned. But the throughput is extremely low. It takes around 300ms per fetch. With same configs, I achieve around 13K inserts/sec.

Just want to confirm if this is expected or I can make some changes in how I am interacting with Aerospike.

#15

The response time you are seeing is not expected. The throughput bottleneck would be on the client end - async requests or multiple threads will help in that case. Without looking into the details of your config, data, and query, it is difficult to pinpoint the issue.

#17

Thanks for the reply @neelp. Apologies for late reply. I have been extremely occupied.

Regarding low throughput, I have tried to add all the config and code in this thread. Please let me know if more info is required.

#18

Thanks - can you share how many records are in the namespace (I may have missed it), and if you run the query with just the filter (without predexp), how long it takes. I am guessing it should not take much longer than that if you add predexp.

#19

Namespace has 132032 records. But the set that I am querying has only 30 records.

#20

if you run the query with just the filter (without predexp), how long it takes. I am guessing it should not take much longer than that if you add predexp.

I don’t have the exact times but performance is not an issue with just the filter. As you mentioned, the performance degrades after adding predexp filters.

© 2021 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.