Error while storing record - code 4

Tools asbackup & asrestore
1

Hi Guys,

Need your help ,im tried to take the backup db : root@ip-10-163-21-121:/# asbackup --host 10.163.21.85 --namespace risk --output-file /etc/asdb_new 2023-07-04 11:04:09 GMT [INF] [ 9777] Starting backup of 10.163.21.85 (namespace: risk, set: [all], bins: [all], after: 1970-01-01 00:00:00 UTC, before: 1970-01-01 00:00:00 UTC, no ttl only: false, limit: 0) to /etc/asdb_new 2023-07-04 11:04:09 GMT [INF] [ 9777] [src/main/aerospike/as_lookup.c:190][as_set_node_address] Switch node address from 10.163.21.85 to node’s access-address 10.163.21.85 2023-07-04 11:04:09 GMT [INF] [ 9777] [src/main/aerospike/as_cluster.c:202][as_cluster_add_nodes_copy] Add node BB93EBD033ABC0A 10.163.21.85:3000 2023-07-04 11:04:09 GMT [INF] [ 9777] Processing 1 node(s) 2023-07-04 11:04:09 GMT [INF] [ 9777] Node ID Objects Replication 2023-07-04 11:04:09 GMT [INF] [ 9777] BB93EBD033ABC0A 16747 2 2023-07-04 11:04:09 GMT [INF] [ 9777] Namespace risk contains 8373 record(s) 2023-07-04 11:04:12 GMT [INF] [ 9815] No secondary indexes 2023-07-04 11:04:12 GMT [INF] [ 9815] Backing up 0 UDF file(s) 2023-07-04 11:04:12 GMT [INF] [ 9815] Starting backup for 4096 partitions from 0 to 4095 2023-07-04 11:04:13 GMT [INF] [ 9814] 36% complete (~6372 KiB/s, ~3028 rec/s, ~2154 B/rec) 2023-07-04 11:04:13 GMT [INF] [ 9814] ~1s remaining 2023-07-04 11:04:17 GMT [INF] [ 9815] Completed backup for 4096 partitions from 0 to 4095, records: 16747, size: 36532224 (~2181 B/rec) 2023-07-04 11:04:17 GMT [INF] [ 9777] Backed up 16747 record(s), 0 secondary index(es), 0 UDF file(s), 36535646 byte(s) in total (~2181 B/rec)

backup is completed but while storing got error while storing record - code 4: root@ip-10-163-21-121:/# asrestore --host 127.0.0.1 --namespace risk --input-file /etc/asdb_new 2023-07-04 11:05:10 GMT [INF] [ 9837] [src/main/aerospike/as_cluster.c:202][as_cluster_add_nodes_copy] Add node BB91C9613DAC40A 127.0.0.1:3000 2023-07-04 11:05:10 GMT [INF] [ 9837] [src/main/aerospike/as_cluster.c:202][as_cluster_add_nodes_copy] Add node BB91C9613DAC40A 127.0.0.1:3000 2023-07-04 11:05:10 GMT [INF] [ 9837] Starting restore to 127.0.0.1 (bins: [all], sets: [all]) from /etc/asdb_new 2023-07-04 11:05:10 GMT [INF] [ 9837] Processing 1 node(s) 2023-07-04 11:05:10 GMT [INF] [ 9837] Restoring /etc/asdb_new 2023-07-04 11:05:10 GMT [INF] [ 9837] Opened backup file /etc/asdb_new 2023-07-04 11:05:10 GMT [INF] [ 9837] Restoring records 2023-07-04 11:05:11 GMT [INF] [ 9873] 0 UDF file(s), 0 secondary index(es), 2960 record(s) (2916 rec/s, 6175 KiB/s, 2168 B/rec, retries: 0) 2023-07-04 11:05:11 GMT [INF] [ 9873] Expired 0 : skipped 0 : err_ignored 0 : inserted 2128: failed 0 (existed 0 , fresher 0) 2023-07-04 11:05:11 GMT [INF] [ 9873] 17% complete, ~4s remaining 2023-07-04 11:05:11 GMT [ERR] [ 9855] Error while storing record - code 4: AEROSPIKE_ERR_REQUEST_INVALID at src/main/aerospike/as_event.c:1278 2023-07-04 11:05:11 GMT [INF] [ 9873] 0 UDF file(s), 0 secondary index(es), 4163 record(s) (2430 rec/s, 6488 KiB/s, 2734 B/rec, retries: 0) 2023-07-04 11:05:11 GMT [INF] [ 9873] Expired 0 : skipped 0 : err_ignored 0 : inserted 4015: failed 0 (existed 0 , fresher 0) 2023-07-04 11:05:11 GMT [INF] [ 9873] 26% complete, ~4s remaining

2

Have you done any troubleshooting? Also in the future could you paste inside triple backticks to preserve newlines ? There is also a web macro for it called “Preformatted text”

for my own sanity:
$ cat f|sed 's/2023/%2023/g' |tr '%' '\n'
root@ip-10-163-21-121:/# asbackup --host 10.163.21.85 --namespace risk --output-file /etc/asdb_new
2023-07-04 11:04:09 GMT [INF] [ 9777] Starting backup of 10.163.21.85 (namespace: risk, set: [all], bins: [all], after: 1970-01-01 00:00:00 UTC, before: 1970-01-01 00:00:00 UTC, no ttl only: false, limit: 0) to /etc/asdb_new
2023-07-04 11:04:09 GMT [INF] [ 9777] [src/main/aerospike/as_lookup.c:190][as_set_node_address] Switch node address from 10.163.21.85 to node’s access-address 10.163.21.85
2023-07-04 11:04:09 GMT [INF] [ 9777] [src/main/aerospike/as_cluster.c:202][as_cluster_add_nodes_copy] Add node BB93EBD033ABC0A 10.163.21.85:3000
2023-07-04 11:04:09 GMT [INF] [ 9777] Processing 1 node(s)
2023-07-04 11:04:09 GMT [INF] [ 9777] Node ID Objects Replication
2023-07-04 11:04:09 GMT [INF] [ 9777] BB93EBD033ABC0A 16747 2
2023-07-04 11:04:09 GMT [INF] [ 9777] Namespace risk contains 8373 record(s)
2023-07-04 11:04:12 GMT [INF] [ 9815] No secondary indexes
2023-07-04 11:04:12 GMT [INF] [ 9815] Backing up 0 UDF file(s)
2023-07-04 11:04:12 GMT [INF] [ 9815] Starting backup for 4096 partitions from 0 to 4095
2023-07-04 11:04:13 GMT [INF] [ 9814] 36
 complete (~6372 KiB/s, ~3028 rec/s, ~2154 B/rec)
2023-07-04 11:04:13 GMT [INF] [ 9814] ~1s remaining
2023-07-04 11:04:17 GMT [INF] [ 9815] Completed backup for 4096 partitions from 0 to 4095, records: 16747, size: 36532224 (~2181 B/rec)
2023-07-04 11:04:17 GMT [INF] [ 9777] Backed up 16747 record(s), 0 secondary index(es), 0 UDF file(s), 36535646 byte(s) in total (~2181 B/rec)

backup is completed but while storing got error while storing record - code 4:

root@ip-10-163-21-121:/# asrestore --host 127.0.0.1 --namespace risk --input-file /etc/asdb_new
2023-07-04 11:05:10 GMT [INF] [ 9837] [src/main/aerospike/as_cluster.c:202][as_cluster_add_nodes_copy] Add node BB91C9613DAC40A 127.0.0.1:3000
2023-07-04 11:05:10 GMT [INF] [ 9837] [src/main/aerospike/as_cluster.c:202][as_cluster_add_nodes_copy] Add node BB91C9613DAC40A 127.0.0.1:3000
2023-07-04 11:05:10 GMT [INF] [ 9837] Starting restore to 127.0.0.1 (bins: [all], sets: [all]) from /etc/asdb_new
2023-07-04 11:05:10 GMT [INF] [ 9837] Processing 1 node(s)
2023-07-04 11:05:10 GMT [INF] [ 9837] Restoring /etc/asdb_new
2023-07-04 11:05:10 GMT [INF] [ 9837] Opened backup file /etc/asdb_new
2023-07-04 11:05:10 GMT [INF] [ 9837] Restoring records
2023-07-04 11:05:11 GMT [INF] [ 9873] 0 UDF file(s), 0 secondary index(es), 2960 record(s) (2916 rec/s, 6175 KiB/s, 2168 B/rec, retries: 0)
2023-07-04 11:05:11 GMT [INF] [ 9873] Expired 0 : skipped 0 : err_ignored 0 : inserted 2128: failed 0 (existed 0 , fresher 0)
2023-07-04 11:05:11 GMT [INF] [ 9873] 17
 complete, ~4s remaining
2023-07-04 11:05:11 GMT [ERR] [ 9855] Error while storing record - code 4: AEROSPIKE_ERR_REQUEST_INVALID at src/main/aerospike/as_event.c:1278
2023-07-04 11:05:11 GMT [INF] [ 9873] 0 UDF file(s), 0 secondary index(es), 4163 record(s) (2430 rec/s, 6488 KiB/s, 2734 B/rec, retries: 0)
2023-07-04 11:05:11 GMT [INF] [ 9873] Expired 0 : skipped 0 : err_ignored 0 : inserted 4015: failed 0 (existed 0 , fresher 0)
2023-07-04 11:05:11 GMT [INF] [ 9873] 26
 complete, ~4s remaining

Looking at the src/dst the ips are different. Are both the source and destination running on the same aerospike-server version? Was the same tools package used for asbackup as well as asrestore? That error is pretty generic and is referenced as “Bad Parameter” because the server sends back a generic failure. It’s usually caused because something invalid was sent like a non-existent namespace was trying to be written to, a bad scan expression, or who knows what.

From: https://github.com/aerospike/aerospike-client-csharp/blob/master/AerospikeClient/Main/ResultCode.cs#L112
|||/// <summary>|
|---|---|---|
|||/// Bad parameter(s) were passed in database operation call.|
|||/// </summary>|
|||public const int PARAMETER_ERROR = 4;|

Now, the odd part is that asbackup/asrestore shouldn’t make those kinds of mistakes mostly. Can you confirm source and target cluster versions, the aerospike-tools version on src/dst, that the namespace exists, and review the logs on the servers for the time of the failure?