thanks for help i have tried with one docker machine as shown here
i can insert 50000 record in 4692ms
Top and namespace config doesn’t show too much out of the ordinary. The log shows you’ve experienced significant timeouts and proxies:
For 130 records, you’ve experiences 95 proxies, another 85 which timed out, and 16 write timeouts.
The only time proxies would occur is when the client is talking to the wrong node for data access. This can occur if there’s a load balancer or virtual IP in front of the server cluster and access is through this load balancer. For your docker swarm config, are you using compose v3.2 syntax and enable dnsrr for endpoint discovery? This can also occur if the cluster just started migrations and the client has not been updated yet.
None of the above would have your original performance issue: 20mins for 100 records.
Your namespace config shows you’re already data-in-memory and write-block-size is 1K. The first is good, the latter, write-block-size, should be 1M for HDDs.
thanks for help i’am using the same steps as here
only by creating two Aerospike Containers on the prod network i use the following lines to be able to connect my client:
docker service create --network prod --replicas 2 --name aerospike -p 3000:3000 -p 3001:3001 -p 3002:3002 -p 3003:3003 aerospike/aerospike-server
if i dont use -p 3000 i get connection refused!
That is correct, because your clients are outside of the docker environment.
Proper client/server networking when crossing the docker environment is very complicated. It would be much easier if you can have your client run within a docker container in the same docker environment.
i figure out how to run jar file in docker and try it!
Please keep track of 2 time periods:
- The time it takes to establish the connection
- The
putloop.
I’d like to focus on just the DB operations. Client timeouts would also affect your final time since you’re doing these sequentially. You have 3 timeout events, with 5 iterations each, so 15 timeout intervals added into your time calculations.
It seems to me DB operations are going along just fine, except for client timeouts that are vastly inflating your timed number.
Hi Richard ps could you tell me why i have much timeout?
Based on your previous log, you have some really high write latencies.
Around half of your write operations takes longer than 1s (2^10 ms). See the lines:
histogram dump: {test}-write (…) msec
(00: …) (10:…)
And an explanation of our histogram logline: http://www.aerospike.com/docs/reference/serverlogmessages
At this point, I’m concluding your storage device is unable to keep up with your workload. Have you tied to increase your write-block-size to 1M?
In-memory only storage engine would skip your storage device completely, but be aware that data does not persist if you stop your containers.
ps could you tell me how i increase write-block-size to 1M thanks
my storage-engine.write-block-size : 1048576
That’s already 1M. My mistake on an earlier post.
i have no idea how can i explain these timeout!!
You have writes that are taking longer than the timeout you’ve set. Your writes are taking so long because you’re doing lots of IO on a single HDD.
but why with only one node i can insert 50000 record in 4692ms on the same HDD
I cannot answer this for you. Your Fusion drive introduces uncontrolled variability and would take extensive testing to accurately account for its behavior.
It could be because you have 1 node being SSD accelerated, but with 2 nodes you’re limited by the HDD. Again, because you have no control over what’s SSD accelerated on a Fusion drive, it’s hard to accurately troubleshoot storage performance issues.