Aerospike getmany failed, error 22

error

#1

when i backup my aerospike database i found this error i use this command asbackup -n push -d $LOCALPATH/asbackup_$datestamp is there something wrong or how i can find other clues to find the reason


#2

Is migration is going on in cluster?

Recommending you to use latest aerospike tools. http://www.aerospike.com/download/tools/3.7.3/


#3

nope, i tried asmonitor and info ,it showed migration(0,0)


#4

i tried newest version of aerospike tools but it showed this: 2016-02-12 09:10:41 GMT [ERR] [11269] Error while running node scan for BB93A938D01B95C - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/aerospike_scan.c:192 2016-02-12 09:10:41 GMT [ERR] [11270] Error while running node scan for BB99E7F8D01B95C - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/aerospike_scan.c:192 2016-02-12 09:10:41 GMT [ERR] [11271] Error while running node scan for BB932938D01B95C - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/aerospike_scan.c:192


#5

Looks like you have lots of scan job running in background. Asbackup issues a new scan for which its seeing max running job reached. Can you provide information about how many scan jobs running?

Run following command using aql: aql> show scans


#6

you are right , i found many scan jobs on node2


#7

Thats why you are seeing scan jobs for asbackup is rejected. Wait for those scan jobs to complete, or if you dont need them anymore kill them .

Issue following command for the same:

Kill one scan job :- asinfo -v ‘scan-abort:id=jobid’

Kill all scan job :- asinfo -v ‘scan-abort-all:’


#8

is it because that i have killed a asbackup progress that caused this problem?


#9

Maximum scan job active in a system is configured to 100. So Apart from killing one asbackup there is something else which stopped many scans, or may be during long period of time it happens.


#10

This is configurable. See the configuration parameters scan-max-active. You should also check that your scan-thread value is set correctly. It should be equal to the number of cores on your machine.


#11

i think i knew where the problem is , i wrote a shell script to backup everyday ,but with large amount of data , it cannot be finished in one day then another process of that shell runs , so it may cause too many scan operations