Hi,all
Recently,I met the problem after writing data to aerospike with expiration=86400*3 ,but a few hours later,the data decreased.There are 4 nodes in the cluster but somehow 3 nodes use disk and memory store while the last one store only in memory,does it matters?
You’re not really giving that much information here.
Did you restart these nodes, or the server on them (asd)?
How are you setting that TTL? Which language client?
Once you create a record, do you ever update it? If you do, does that update have a TTL?
What is the default TTL of the namespace?
Quoting the configuration here, and adding a code snippet from the client would help.
And no, you shouldn’t have namespace definition that doesn’t agree like that, but mostly for operational and performance reasons. Do a diff on the config files to see if they’re the same.
Let’s just say this is not normal behavior. You probably want to look in the logs for things marked ‘warning’ and ‘error’.
Evictions start when you breach the high watermark for memory or disk. You will see a warning in your log that hwm_breached has happened. The operations manual discusses the key metrics you should be monitoring, and this is one of them.
i have the same problem and i dont know how to fix it.
I use the default config after install:
namespace test {
replication-factor 2
memory-size 4G
default-ttl 30d # 30 days, use 0 to never expire/evict.
storage-engine memory
}
there are 256.408 objects saved (24% memory usage), some minutes later
there are only 133.294 objects (13% memory usage).
13,229 objects expired (seems to be correct)
0 objects evicted
What happend with my data?
My statistics:
1 : node
BB9E091CB7AC40C
2 : statistics
cluster_size=1;cluster_key=DCF17F353E9E73A7;cluster_integrity=true;uptime=545;system_free_mem_pct=79;system_swapping=false;heap_allocated_kbytes=1654377;heap_active_kbytes=1863852;heap_mapped_kbytes=2336768;heap_efficiency_pct=71;objects=133295;sub_objects=0;tombstones=0;tsvc_queue=0;info_queue=0;delete_queue=0;rw_in_progress=0;proxy_in_progress=0;tree_gc_queue=0;client_connections=31;heartbeat_connections=0;fabric_connections=0;heartbeat_received_self=3635;heartbeat_received_foreign=0;reaped_fds=0;info_complete=9935;proxy_retry=0;demarshal_error=0;early_tsvc_client_error=0;early_tsvc_batch_sub_error=0;early_tsvc_udf_sub_error=0;batch_index_initiate=0;batch_index_queue=0:0,0:0,0:0,0:0;batch_index_complete=0;batch_index_error=0;batch_index_timeout=0;batch_index_unused_buffers=0;batch_index_huge_buffers=0;batch_index_created_buffers=0;batch_index_destroyed_buffers=0;batch_initiate=0;batch_queue=0;batch_error=0;batch_timeout=0;scans_active=0;query_short_running=0;query_long_running=0;sindex_ucgarbage_found=0;sindex_gc_locktimedout=0;sindex_gc_inactivity_dur=0;sindex_gc_activity_dur=0;sindex_gc_list_creation_time=0;sindex_gc_list_deletion_time=0;sindex_gc_objects_validated=0;sindex_gc_garbage_found=0;sindex_gc_garbage_cleaned=0;paxos_principal=BB9E091CB7AC40C;migrate_allowed=true;migrate_partitions_remaining=0;fabric_bulk_send_rate=0;fabric_bulk_recv_rate=0;fabric_ctrl_send_rate=0;fabric_ctrl_recv_rate=0;fabric_meta_send_rate=0;fabric_meta_recv_rate=0;fabric_rw_send_rate=0;fabric_rw_recv_rate=0
3 : features
peers;cdt-list;cdt-map;pipelining;geo;float;batch-index;replicas-all;replicas-master;replicas-prole;udf
4 : cluster-generation
0
5 : partition-generation
1
6 : build_time
Wed Feb 15 21:57:43 UTC 2017
7 : edition
Aerospike Community Edition
8 : version
Aerospike Community Edition build 3.11.1.1
9 : build
3.11.1.1
10 : services
11 : services-alumni
12 : build_os
debian8
1 - How are you inserting the objects? Are you modifying the default TTL when you are inserting objects? With 30d default ttl, you should not be expiring objects - assuming you dropped from 256K to 133K fairly quickly. What is the size of your objects?
2 - can you $grep thr_nsup /var/log/aerospike/aerospike.log and post output?
3 - can you post output of $asinfo -v "namespace/test"
@pgupta Hi is there any restriction for client ttl over default-ttl of namespace.
for an example ,can we set ttl while writing record to higher then default-ttl of namespace.