when i backup my aerospike database i found this error
i use this command
asbackup -n push -d $LOCALPATH/asbackup_$datestamp
is there something wrong or how i can find other clues to find the reason
Is migration is going on in cluster?
Recommending you to use latest aerospike tools. Aerospike Downloads | Aerospike
nope, i tried asmonitor and info ,it showed migration(0,0)
i tried newest version of aerospike tools but it showed this: 2016-02-12 09:10:41 GMT [ERR] [11269] Error while running node scan for BB93A938D01B95C - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/aerospike_scan.c:192 2016-02-12 09:10:41 GMT [ERR] [11270] Error while running node scan for BB99E7F8D01B95C - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/aerospike_scan.c:192 2016-02-12 09:10:41 GMT [ERR] [11271] Error while running node scan for BB932938D01B95C - code 22: AEROSPIKE_ERR_FAIL_FORBIDDEN at src/main/aerospike/aerospike_scan.c:192
Looks like you have lots of scan job running in background. Asbackup issues a new scan for which its seeing max running job reached. Can you provide information about how many scan jobs running?
Run following command using aql: aql> show scans
you are right , i found many scan jobs on node2
Thats why you are seeing scan jobs for asbackup is rejected. Wait for those scan jobs to complete, or if you dont need them anymore kill them .
Issue following command for the same:
Kill one scan job :- asinfo -v ‘scan-abort:id=jobid’
Kill all scan job :- asinfo -v ‘scan-abort-all:’
is it because that i have killed a asbackup progress that caused this problem?
Maximum scan job active in a system is configured to 100. So Apart from killing one asbackup there is something else which stopped many scans, or may be during long period of time it happens.
This is configurable. See the configuration parameters scan-max-active. You should also check that your scan-thread value is set correctly. It should be equal to the number of cores on your machine.
i think i knew where the problem is , i wrote a shell script to backup everyday ,but with large amount of data , it cannot be finished in one day then another process of that shell runs , so it may cause too many scan operations