Invalid key type: 0 error during backups

Tools asbackup & asrestore
1

Hello,

asbackup is logging the below errors, although the backup still completes. I’m planning to do a test restore to see if it succeeds and how much data is restored. What is causing this error and how can it be fixed? Invalid key type sounds critical

Errors:

2017-08-10 07:34:32 GMT [ERR] [20119] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0
2017-08-10 07:34:32 GMT [ERR] [20118] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0
2017-08-10 07:34:32 GMT [ERR] [20118] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0
2017-08-10 07:34:32 GMT [ERR] [20120] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0
2017-08-10 07:34:32 GMT [ERR] [20118] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0
2017-08-10 07:34:32 GMT [ERR] [20120] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0
2017-08-10 07:34:32 GMT [ERR] [20120] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0
2017-08-10 07:34:32 GMT [ERR] [20118] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0
2017-08-10 07:34:32 GMT [ERR] [20119] [src/main/aerospike/as_command.c:729][as_command_parse_key] Invalid key type: 0

command:

/usr/bin/asbackup --host xxx.xxx.xxx.xxx --namespace sports --verbose --nice 1 --priority 1 --directory /var/aerospike/2017-08-10-06-32-59_sports_backup
2

weeeird. no idea never seen it

3

out of curiosity… did the number of records backed up match what you’d expect? What was the final output? Did it have any number of “Failed” records?

4

It is seeing a key type = 0 which is AS_BYTES_UNDEF. Its looking for AS_BYTES_INTEGER =1, _STRING, _DOUBLE, or _BLOB for key type. Don’t know why that field is zero in your case.

5

Oh, wow. Interesting. May I ask a few further questions?

  • Which version of the Aerospike server and asbackup does this happen with?

  • Which Aerospike client (i.e., C, Java, etc.) and which client version does your application use to access the Aerospike server?

  • Could you share the namespace configuration section for sports from your aerospike.conf? I’m mostly interested in seeing, whether the namespace is SSD-backed and whether reads come from memory (“data in memory”) or from SSD.

  • Do you use sets? If you use sets, then does the issue affect the data in all sets or just a specific set? If it’s just a specific set, is there anything special about the keys used in this set?

Thank you!

6

@tlo Sorry for delayed reply. Here are some answers to your questions.

Which version of the Aerospike server and asbackup does this happen with?

$ asinfo -v version
Aerospike Community Edition build 3.9.1.1

asbackup - whatever came as standard with this version

Which Aerospike client (i.e., C, Java, etc.) and which client version does your application use to access the Aerospike server?

GO and Ruby. Multiple different versions.

Could you share the namespace configuration section for sports from your aerospike.conf? I’m mostly interested in seeing, whether the namespace is SSD-backed and whether reads come from memory (“data in memory”) or from SSD.

...
namespace sport_a {
        conflict-resolution-policy last-update-time
        replication-factor 2
        memory-size 64M
        default-ttl 24H
        #high-water-disk-pct 50
        high-water-memory-pct 60
        stop-writes-pct 90
        #ldt-enabled true
        storage-engine memory
}
namespace sport_b {
        conflict-resolution-policy last-update-time
        replication-factor 2
        memory-size 1G
        default-ttl 180d
        high-water-disk-pct 50
        high-water-memory-pct 60
        stop-writes-pct 90
        ldt-enabled true
        storage-engine device {
                file /opt/aerospike/data/sport_b.dat
                filesize 1G
                data-in-memory true
                write-block-size 512K
                defrag-lwm-pct 60
        }
}
namespace sport_c {
        conflict-resolution-policy last-update-time
        replication-factor 2
        memory-size 256M
        default-ttl 0
        high-water-disk-pct 50
        high-water-memory-pct 60
        stop-writes-pct 90
        ldt-enabled true
        storage-engine device {
                file /opt/aerospike/data/sport_c.dat
                filesize 1G
                data-in-memory true
                write-block-size 128K
                defrag-lwm-pct 60
                #cold-start-empty true
        }
}
namespace sport_d {
        conflict-resolution-policy last-update-time
        replication-factor 2
        memory-size 4G
        default-ttl 0
        high-water-disk-pct 50
        high-water-memory-pct 60
        stop-writes-pct 90
                
        migrate-sleep 2
        ldt-enabled true
        storage-engine device {
                file /opt/aerospike/data/sport_d.dat
                filesize 60G
                #data-in-memory true
                write-block-size 512K
                defrag-lwm-pct 60
                #cold-start-empty true
        }
}
namespace sport_e {
        conflict-resolution-policy last-update-time
        replication-factor 2
        memory-size 512M
        default-ttl 0
        high-water-disk-pct 50
        high-water-memory-pct 60
        stop-writes-pct 90
        #ldt-enabled true
        storage-engine device {
                file /opt/aerospike/data/sport_e.dat
                filesize 4G
                data-in-memory true
                write-block-size 128K
                defrag-lwm-pct 60
                #cold-start-empty true
        }
}
namespace sport_f {
        conflict-resolution-policy last-update-time
        replication-factor 2
        memory-size 512M
        default-ttl 0
        high-water-disk-pct 50
        high-water-memory-pct 60
        stop-writes-pct 90
        storage-engine device {
                file /opt/aerospike/data/sport_f.dat
                filesize 4G
                data-in-memory true
                write-block-size 128K
                defrag-lwm-pct 60
        }
}

Namespace sport_d is giving these errors.

Do you use sets? If you use sets, then does the issue affect the data in all sets or just a specific set? If it’s just a specific set, is there anything special about the keys used in this set?

Yes, we use sets. I would have to back them up individually to see if error occurs in all of them.