How to throttle scan UDFs


#1

How to throttle scan UDFs

Context

Running scan UDFs can impact system resources and transaction queues and may require throttling their executions. The scan-max-udf-transactions setting can be tuned for such a purpose.

Method

The scan-max-udf-transactions setting can be tuned to globally throttle UDF executions. This setting is dynamic and controls the number of transactions that is sent per running UDFs to the transaction-queues.

A throttling will occur when the number of active transactions for a particular UDF job is greater or equal to scan-max-udf-transactions.

This throttling will continue until the transaction has finished processing and the udf-active (number of active transactions per scan UDF) count for that job has decremented below scan-max-udf-transactions.

The udf-active count can be checked by running an jobs info call while the scan UDF is running:

Command to run:

asinfo -v 'jobs:module=scan'

Example:

$ asinfo -v 'jobs:module=scan'
module=scan:trid=14506333510584022897:job-type=background-udf:ns=test:set=testset:priority=2:status=done(ok):job-progress=100.00:run-time=38:time-since-done=20095:recs-read=20:net-io-bytes=30:udf-filename=bg_scan_udf:udf-function=test_bin_add_1000:udf-active=0:udf-success=20:udf-failed=0

Monitor the udf-active number and set scan-max-udf-transactions accordingly:

asinfo -v "set-config:context=service;scan-max-udf-transactions=24"

Notes

The throttling is independent of the number of scan udfs.

udf-active: refers to the number of transactions that were sent to the transaction-queues but have not yet finished processing.

Throttling can also be implemeted at the lua level by implementing a sleep function:

http://lua-users.org/wiki/SleepFunction (os.sleep() )

https://stackoverflow.com/questions/17987618/how-to-add-a-sleep-or-wait-to-my-lua-script (luasocket sleep)

Keywords

UDF THROTTLING SCAN LUA SCAN-UDF

Timestamp

07/16/2018