I’m working on porting one of our Python projects to Rust. I use Tokio as an async runtime and the GitHub - aerospike/aerospike-client-rust at async client library.
The application is a thrift CRUD wrapper around aerospike. At this moment rust app acts as a thrift proxy - forwarding all incoming requests (except one, implemented in rust) to the python app. The implemented one is just a single batch_get + parsing response.
The issue I found is a high batch_get p99 response delay - ~300ms compared to the sync python client (26ms).
Testing environment: rust and python apps are deployed to EKS, same namespace, same nodes(arm64, 4core). Aerospike Enterprise Edition build 7.1.0.3, 2 nodes in the same EKS. Rust and python apps are talking to the same aerospike cluster.
Test method: I use istio to split incoming traffic in a ratio 10:1 rust:python. I need some traffic going to python to see the difference in response time.
I did try my best to find out the reason for such poor rust performance changing number of connections in the pool, pool per node - no visible effect. I tried to install a tokio-console in the hope of seeing some locks/mutexes waiting - nothing interesting there. I tried to add tracing to the aerospike client and found it spends most of the time in parse_result method.
Also I have found couple of panic conditions (and hopefully some speed improvements) fixed here.
Lookng for any help/advice. Thanks!
Aerospike support ticket: #00038190
