Aerospike data expire


#1

Hi,all Recently,I met the problem after writing data to aerospike with expiration=86400*3 ,but a few hours later,the data decreased.There are 4 nodes in the cluster but somehow 3 nodes use disk and memory store while the last one store only in memory,does it matters?

Thanks ,


#2

You’re not really giving that much information here.

  1. Did you restart these nodes, or the server on them (asd)?
  2. How are you setting that TTL? Which language client?
  3. Once you create a record, do you ever update it? If you do, does that update have a TTL?
  4. What is the default TTL of the namespace?

Quoting the configuration here, and adding a code snippet from the client would help.

And no, you shouldn’t have namespace definition that doesn’t agree like that, but mostly for operational and performance reasons. Do a diff on the config files to see if they’re the same.

Let’s just say this is not normal behavior. You probably want to look in the logs for things marked ‘warning’ and ‘error’.


#3

Thanks sir.May be I find the answer.Watching the UsedMem,all the nodes exceed the upper limit of hwm mem.It that evicting data makes data loss?


#4

There’s a good knowledge base article in the community forum, FAQ What are Expiration, Eviction and Stop-Writes?

Evictions start when you breach the high watermark for memory or disk. You will see a warning in your log that hwm_breached has happened. The operations manual discusses the key metrics you should be monitoring, and this is one of them.


#5

Hi,

i have the same problem and i dont know how to fix it.

I use the default config after install:

namespace test {
	replication-factor 2
	memory-size 4G
	default-ttl 30d # 30 days, use 0 to never expire/evict.
	storage-engine memory
}

there are 256.408 objects saved (24% memory usage), some minutes later there are only 133.294 objects (13% memory usage). 13,229 objects expired (seems to be correct) 0 objects evicted

What happend with my data?

My statistics:

1 :  node
     BB9E091CB7AC40C
2 :  statistics
     cluster_size=1;cluster_key=DCF17F353E9E73A7;cluster_integrity=true;uptime=545;system_free_mem_pct=79;system_swapping=false;heap_allocated_kbytes=1654377;heap_active_kbytes=1863852;heap_mapped_kbytes=2336768;heap_efficiency_pct=71;objects=133295;sub_objects=0;tombstones=0;tsvc_queue=0;info_queue=0;delete_queue=0;rw_in_progress=0;proxy_in_progress=0;tree_gc_queue=0;client_connections=31;heartbeat_connections=0;fabric_connections=0;heartbeat_received_self=3635;heartbeat_received_foreign=0;reaped_fds=0;info_complete=9935;proxy_retry=0;demarshal_error=0;early_tsvc_client_error=0;early_tsvc_batch_sub_error=0;early_tsvc_udf_sub_error=0;batch_index_initiate=0;batch_index_queue=0:0,0:0,0:0,0:0;batch_index_complete=0;batch_index_error=0;batch_index_timeout=0;batch_index_unused_buffers=0;batch_index_huge_buffers=0;batch_index_created_buffers=0;batch_index_destroyed_buffers=0;batch_initiate=0;batch_queue=0;batch_error=0;batch_timeout=0;scans_active=0;query_short_running=0;query_long_running=0;sindex_ucgarbage_found=0;sindex_gc_locktimedout=0;sindex_gc_inactivity_dur=0;sindex_gc_activity_dur=0;sindex_gc_list_creation_time=0;sindex_gc_list_deletion_time=0;sindex_gc_objects_validated=0;sindex_gc_garbage_found=0;sindex_gc_garbage_cleaned=0;paxos_principal=BB9E091CB7AC40C;migrate_allowed=true;migrate_partitions_remaining=0;fabric_bulk_send_rate=0;fabric_bulk_recv_rate=0;fabric_ctrl_send_rate=0;fabric_ctrl_recv_rate=0;fabric_meta_send_rate=0;fabric_meta_recv_rate=0;fabric_rw_send_rate=0;fabric_rw_recv_rate=0
3 :  features
     peers;cdt-list;cdt-map;pipelining;geo;float;batch-index;replicas-all;replicas-master;replicas-prole;udf
4 :  cluster-generation
     0
5 :  partition-generation
     1
6 :  build_time
     Wed Feb 15 21:57:43 UTC 2017
7 :  edition
     Aerospike Community Edition
8 :  version
     Aerospike Community Edition build 3.11.1.1
9 :  build
     3.11.1.1
10 :  services

11 :  services-alumni

12 :  build_os
     debian8

#6

1 - How are you inserting the objects? Are you modifying the default TTL when you are inserting objects? With 30d default ttl, you should not be expiring objects - assuming you dropped from 256K to 133K fairly quickly. What is the size of your objects?

2 - can you $grep thr_nsup /var/log/aerospike/aerospike.log and post output? 3 - can you post output of $asinfo -v "namespace/test"


#7

Hi, thx for your help. I found the issue in my code. Aerospike is working fine.


#8

@dercoder was the observed issue with your code overwriting the default-ttl to a lower value?