I find out from log4j log that there are 100x of this exception every day
com.aerospike.client.AerospikeException: Error Code 9: Timeout
at com.aerospike.client.command.ReadCommand.parseResult(ReadCommand.java:121)
at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:60)
at com.aerospike.client.AerospikeClient.get(AerospikeClient.java:508)
at org.springframework.cache.aerospike.AerospikeCache.get(AerospikeCache.java:58)
at org.springframework.cache.interceptor.AbstractCacheInvoker.doGet(AbstractCacheInvoker.java:68)
at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:533)
at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:499)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:389)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:327)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
com.aerospike.client.AerospikeException: java.io.EOFException
at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:93)
at com.aerospike.client.AerospikeClient.get(AerospikeClient.java:508)
com.aerospike.client.AerospikeException: Buffer pool take interrupted.
at com.aerospike.client.async.AsyncCluster$BlockBufferQueue.getByteBuffer(AsyncCluster.java:124)
at com.aerospike.client.async.AsyncCluster.getByteBuffer(AsyncCluster.java:75)
at com.aerospike.client.async.AsyncCommand.execute(AsyncCommand.java:67)
at com.aerospike.client.async.AsyncClient.put(AsyncClient.java:228)
I am using
CentOS 7.3
aerospike-tools-3.12.0-1.el7.x86_64
aerospike-amc-community-4.0.12-1.x86_64
aerospike-server-community-3.12.0-1.el7.x86_64
aerospike-client-3.3.2-jar-with-dependencies.jar
resin-4.0.49-1.x86_64
jdk1.8.0_121-1.8.0_121-fcs.x86_64
The VM is having
8 cores
40GB RAM
Kingston SSD HDD
The following is aerospike.conf
service {
user root
group root
paxos-single-replica-limit 0 # Number of nodes where the replica count is automatically reduced to 1.
pidfile /var/run/aerospike/asd.pid
service-threads 4
transaction-queues 4
transaction-threads-per-queue 4
proto-fd-max 100000
proto-fd-idle-ms 15000
}
logging {
# Log file must be an absolute path.
file /var/log/aerospike/aerospike.log {
context any info
}
}
network {
service {
address any
port 3000
}
heartbeat {
# To use unicast-mesh heartbeats, remove the 3 lines above, and see
# aerospike_mesh.conf for alternative.
mode mesh
address 192.168.10.168
port 3002
mesh-seed-address-port 192.168.10.168 3002
interval 150
timeout 10
}
fabric {
port 3001
}
info {
port 3003
}
}
namespace test {
replication-factor 1
memory-size 4G
default-ttl 30d # 30 days, use 0 to never expire/evict.
ldt-enabled true
storage-engine memory
}
namespace bar {
replication-factor 1
memory-size 4G
default-ttl 30d # 30 days, use 0 to never expire/evict.
storage-engine memory
}
namespace opentravel {
replication-factor 1
memory-size 10G
default-ttl 30d # 30 days, use 0 to never expire/evict.
ldt-enabled true
storage-engine device {
file /cache/aerospike/data/opentravel.dat
filesize 4G
write-block-size 1048576
max-write-cache 128M
data-in-memory false # Store data in memory in addition to file.
}
}
I need to find out if my configuration is incorrect, the SSD is not good enough or something else?
I am using the default value of aerospike java client.
The following is the histogram
Admin> show distribution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~hotelbeds - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 1890924 1890924 1890924 2521232 26472936 29939630 29939630 29939630 29939630 31515400
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~bar - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~giata - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 310752 854568 1217112 1268904 1760928 2045784 2356536 2356536 2408328 2589600
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~gallop - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~qunar - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 206416 1290100 1419110 1419110 1754536 1935150 2064160 2347982 2451190 2580200
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~abreu_online - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 7182297 7182297 7722900 7722900 7722900 7722900 7722900 7722900 7722900 7722900
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~mikitravel - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 309948 826528 1239792 1265621 1291450 1368937 1833859 1833859 1833859 2582900
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~uapi - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~sunseries - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~test - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~hanatour - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 1995000 1995000 1995000 1995000 1995000 1995000 1995000 1995000 1995000 1995000
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ctrip - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~opentravel - TTL Distribution in Seconds~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having ttl less than or equal to value measured in Seconds
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 855 855 855 855 912 1254 1767 3534 5700 5700
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~hotelbeds - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 2 2 11 59 99 99 99 99 99 99
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~hanatour - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 1 1 2 2 2 2 2 2 4 99
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~giata - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 1 1 2 2 2 5 6 19 20 99
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~gallop - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~qunar - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 5 5 5 5 5 5 5 5 7 42
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~abreu_online - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 1 2 99 99 99 99 99 99 99 99
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~mikitravel - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 1 1 1 1 1 1 1 2 2 94
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~sunseries - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~~~ctrip - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 0 0 0 0 0 0 0 0 0 0
Number of rows: 1
~~~~~~~~~~~~~~~~~~~~~~~opentravel - Object Size Distribution in Record Blocks~~~~~~~~~~~~~~~~~~~~~~
Percentage of records having objsz less than or equal to value measured in Record Blocks
Node 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
roomstays-a localhost.localdomain:3000 1 1 1 1 2 2 2 2 2 2
Number of rows: 1