How to throttle scan UDFs

The Aerospike Knowledge Base has moved to https://support.aerospike.com. Content on https://discuss.aerospike.com is being migrated to either https://support.aerospike.com or https://docs.aerospike.com. Maintenance on articles stored in this repository ceased on December 31st 2022 and this article may be stale. If you have any questions, please do not hesitate to raise a case via https://support.aerospike.com.

How to throttle scan UDFs

Context

Running scan UDFs can impact system resources and transaction queues and may require throttling their executions. The scan-max-udf-transactions setting can be tuned for such a purpose.

Method

The scan-max-udf-transactions setting can be tuned to globally throttle UDF executions. This setting is dynamic and controls the number of transactions that is sent per running UDFs to the transaction-queues.

A throttling will occur when the number of active transactions for a particular UDF job is greater or equal to scan-max-udf-transactions.

This throttling will continue until the transaction has finished processing and the udf-active (number of active transactions per scan UDF) count for that job has decremented below scan-max-udf-transactions.

The udf-active count can be checked by running an jobs info call while the scan UDF is running:

Command to run:

asinfo -v 'jobs:module=scan'

Example:

$ asinfo -v 'jobs:module=scan'
module=scan:trid=14506333510584022897:job-type=background-udf:ns=test:set=testset:priority=2:status=done(ok):job-progress=100.00:run-time=38:time-since-done=20095:recs-read=20:net-io-bytes=30:udf-filename=bg_scan_udf:udf-function=test_bin_add_1000:udf-active=0:udf-success=20:udf-failed=0

Monitor the udf-active number and set scan-max-udf-transactions accordingly:

asinfo -v "set-config:context=service;scan-max-udf-transactions=24"

Notes

  • The throttling is independent of the number of scan udfs.

  • udf-active: refers to the number of transactions that were sent to the transaction-queues but have not yet finished processing.

Keywords

UDF THROTTLING SCAN LUA SCAN-UDF

Timestamp

07/16/2018