UseCompression flag in Aerospike Go client results in unexpected Server Error

I am using Aerospike Go Client 3.0.5 (& Aerospike Server build version: 4.2.0.4)

Recently, I was experimenting with Go client to optimize the network usage of Aerospike Server.

Without compression each node is sending around 2-3 Gb/s per node, and CPU/RAM/Disk usage are way below the thresholds to plan for adding extra node(s).

Because of such high usage of network I don’t want to scale to use more before I optimize the network usage of the server.

My most reads happen through Batch-Get.

I thought to use the compression (UseCompression flag in BatchPolicy)

The documentation of UseCompression is as

// UseCompression tells the server to compress its response using zlib.
// Use zlib compression on write or batch read commands when the command buffer size is greater
// than 128 bytes. In addition, it directs the server to compress its response on read commands.
// The server response compression threshold is also 128 bytes.
//
// This option will increase CPU and memory usage (for extra compressed buffers), but
// decrease the size of data sent over the network.
//
// Default: false

From this documentation it seemed very obvious to me that I don’t need to do any changes in Aerospike-Server’s config.

But as soon as I started using this flag (with true value) I started getting Server Error errors on the client.

Actual log:

11:20AM ERR Aerospike Batch Get Failed error="Server error\nServer error\nServer error\nServer error\n"

For further debugging I went to one of the Aerospike Server node and checked the logs if any corresponding errors are there and I found

decompression failed! (rv -3)
Sep 10 11:22:04 rtb-fr-dmm-as-1.srv.harsh.com asd[2306]: Sep 10 2020 11:22:04 GMT: WARNING (demarshal): (thr_demarshal.c:720) compressed proto_p<HexSpaced>:02 04 b1 00 00 00 00 00 00 00 00 00 00 00 00 b6 78 9c 62 62 66 00 81 75 62 dd 0c 28 80 f9 39 03 23 98 31 45 93 81 81 81 15 c4 66 4c f3 5c f0 de 77 c2 74 d7 49 2a ef a5 9b ac 57 2e bd af 18 21 cb c0 0c 55 c7 c1 90 92 9b 1b 5f 5c 9c c2 c0 c0 c0 a6 97 57 6b b4 ec 7e 0c d3 fd e8 cf ff 8f 36 59 14 84 e7 b4 77 81 94 09 4c 9b f5 54 60 f6 01 4e 67 e7 47 7a 47 17 32 cb cf 3b ad ee a4 06 92 10 5e e6 97 2b b3 da 34 de a7 bc 8a 2d 53 fc c7 24 ed 33 4b 99 38 41 12 a2 09 b9 17 8e 6a 09 17 6b fa df 59 b4 5f 58 67 a7 6f 78 61 ff 74 46 40 00 00 00 ff ff d3 be 32 d6
Sep 10 11:22:04 rtb-fr-dmm-as-1.srv.harsh.com asd[2306]: Sep 10 2020 11:22:04 GMT: WARNING (demarshal): (thr_demarshal.c:719) as_proto decompression failed! (rv -3)

This looks like that Aerospike Client is sending the zlib compressed request to Server & Server is not able to decompress the request. Further explanation could be that zlib library is not present on the server, cross-checked the presence of zlib & it is present.

dpkg -l | grep zlib
ii  zlib1g:amd64                               1:1.2.8.dfsg-5                              amd64        compression library - runtime

Searching on the internet related to compression in Aerospike, all I could find was the Storage Compression which needs namespace config changes, my intention is not to use Storage Compression (will be good to use, but not a goal right now, also no where it is mentioned that for UseCompression storage compression is also needed)

Current Namespace config for reference:

namespace dmm_ssd {
        replication-factor 2
        memory-size 52G
        default-ttl 30d
        high-water-disk-pct 80
        high-water-memory-pct 90
        stop-writes-pct 100

        storage-engine device {
                device /dev/nvme1n1
                write-block-size 1M
                defrag-lwm-pct 50
        }
}

Please help me to understand what I am missing, and how I can fix the issue and start using UseCompression successfully.

2 Likes

Hi. There are several reasons why this is not working. For one, this is an Enterprise Edition feature, and requires the compression feature key in the license file (see the Product Matrix).

The other issue, this feature was added in EE 4.8, and you’re on (likely CE) version 4.2. This is mentioned in the 4.8.0.1 release notes under New Features. Granted, reading release notes is for true connoisseurs, so the easier way to keep track of features in new releases is subscribing to The Aerospike Standup.

Thanks for the response, can these things be mentioned in the documentation of client as well? I think at least version related information can be added in the documentation (for some specific features like this one, I’d expect that UseCompression flag in policy can have been better documented with version info) and EE/CE can be checked out from Product Matrix.

1 Like

I agree, including version information would have saved you time here. We’ll take this suggestion into account.

Thanks @rbotzer

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.