I am using Aerospike Go Client 3.0.5 (& Aerospike Server build version: 4.2.0.4)
Recently, I was experimenting with Go client to optimize the network usage of Aerospike Server.
Without compression each node is sending around 2-3 Gb/s per node, and CPU/RAM/Disk usage are way below the thresholds to plan for adding extra node(s).
Because of such high usage of network I don’t want to scale to use more before I optimize the network usage of the server.
My most reads happen through Batch-Get.
I thought to use the compression (UseCompression flag in BatchPolicy)
The documentation of UseCompression is as
// UseCompression tells the server to compress its response using zlib.
// Use zlib compression on write or batch read commands when the command buffer size is greater
// than 128 bytes. In addition, it directs the server to compress its response on read commands.
// The server response compression threshold is also 128 bytes.
//
// This option will increase CPU and memory usage (for extra compressed buffers), but
// decrease the size of data sent over the network.
//
// Default: false
From this documentation it seemed very obvious to me that I don’t need to do any changes in Aerospike-Server’s config.
But as soon as I started using this flag (with true value) I started getting Server Error errors on the client.
Actual log:
11:20AM ERR Aerospike Batch Get Failed error="Server error\nServer error\nServer error\nServer error\n"
For further debugging I went to one of the Aerospike Server node and checked the logs if any corresponding errors are there and I found
decompression failed! (rv -3)
Sep 10 11:22:04 rtb-fr-dmm-as-1.srv.harsh.com asd[2306]: Sep 10 2020 11:22:04 GMT: WARNING (demarshal): (thr_demarshal.c:720) compressed proto_p<HexSpaced>:02 04 b1 00 00 00 00 00 00 00 00 00 00 00 00 b6 78 9c 62 62 66 00 81 75 62 dd 0c 28 80 f9 39 03 23 98 31 45 93 81 81 81 15 c4 66 4c f3 5c f0 de 77 c2 74 d7 49 2a ef a5 9b ac 57 2e bd af 18 21 cb c0 0c 55 c7 c1 90 92 9b 1b 5f 5c 9c c2 c0 c0 c0 a6 97 57 6b b4 ec 7e 0c d3 fd e8 cf ff 8f 36 59 14 84 e7 b4 77 81 94 09 4c 9b f5 54 60 f6 01 4e 67 e7 47 7a 47 17 32 cb cf 3b ad ee a4 06 92 10 5e e6 97 2b b3 da 34 de a7 bc 8a 2d 53 fc c7 24 ed 33 4b 99 38 41 12 a2 09 b9 17 8e 6a 09 17 6b fa df 59 b4 5f 58 67 a7 6f 78 61 ff 74 46 40 00 00 00 ff ff d3 be 32 d6
Sep 10 11:22:04 rtb-fr-dmm-as-1.srv.harsh.com asd[2306]: Sep 10 2020 11:22:04 GMT: WARNING (demarshal): (thr_demarshal.c:719) as_proto decompression failed! (rv -3)
This looks like that Aerospike Client is sending the zlib compressed request to Server & Server is not able to decompress the request.
Further explanation could be that zlib library is not present on the server, cross-checked the presence of zlib & it is present.
dpkg -l | grep zlib
ii zlib1g:amd64 1:1.2.8.dfsg-5 amd64 compression library - runtime
Searching on the internet related to compression in Aerospike, all I could find was the Storage Compression which needs namespace config changes, my intention is not to use Storage Compression (will be good to use, but not a goal right now, also no where it is mentioned that for UseCompression storage compression is also needed)
Current Namespace config for reference:
namespace dmm_ssd {
replication-factor 2
memory-size 52G
default-ttl 30d
high-water-disk-pct 80
high-water-memory-pct 90
stop-writes-pct 100
storage-engine device {
device /dev/nvme1n1
write-block-size 1M
defrag-lwm-pct 50
}
}
Please help me to understand what I am missing, and how I can fix the issue and start using UseCompression successfully.