I am running benchmark tool on the same machine at which aerospike is installed. Platform is ubuntu 14.04. Everything is working fine and I am able to connect to database but while running benchmark it is saying
2015-04-01 19:08:09 ERROR Server is currently in readonly mode. Shutting down…
After I restart aerospike service, it runs for next 2-3 minutes and after that the same error.
Running this command for benchmark test:
./target/benchmarks -h 127.0.0.1 -p 3000 -n test -k 100000000 -o S:50 -w RU,90 -z 20
Can you please share your server config (aerospike.conf) and machine details like RAM, disk etc. Also share your aerospike.log file (/var/log/aerospike/aerospike.log)
This is most probably caused due to some problem in configuration around replication factor and available RAM.
In some cases, the replication factor can't be satisfied. The cluster can be configured to either decrease the replication factor and retain all data, or begin evicting the oldest data that is marked as disposable. If the cluster can't accept any more data, it will begin operating in a read-only mode until new capacity becomes available - at which point it will automatically being accepting application writes.
Server Specs : Version: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, Max Speed: 4000 MHz, Current Speed: 2100 MHz; 96GB RAM and has two HDDs (One SAS : 900G, SSD: 120G)
OS: Ubuntu 14.04 64bit
Everything is kept same as default file after installation except following modifications.
namespace test {
replication-factor 1
memory-size 1K
default-ttl 30d # 30 days, use 0 to never expire/evict.
# Warning - legacy data in defined raw partition devices will be erased.
# These partitions must not be mounted by the file system.
storage-engine device {
# Use one or more lines like those below with actual device paths.
# device /dev/sdb
# device /dev/sdc
device /dev/sda
# The 2 lines below optimize for SSD.
scheduler-mode noop
write-block-size 128K
# Use the line below to store data in memory in addition to devices.
# data-in-memory true
}
}
Sorry for the late reply on this. Looks like you need to allocate more memory… With 1K memory-size you will not be able to hold many records… Only 16 actually (each record uses 64bytes of RAM for the index). You will likely be able to go beyond (based on periodic checks for this) but the system will fairly quickly put you in stop-writes because you would be above the stop-writes-pct (default 90% of memory allocated). At that point, the database is putting itself in read-only mode to avoid using too much more memory then allocated. You can verify this in the logs (you would need to share more of the logs for me to point out where).
You can find more details on stop writes scenarios here.
Also, here is our capacity planning guide where you can find details on primary index sizing and more.
Thanks, and let us know if you have any further questions.