by ikochergin » Thu Aug 07, 2014 12:09 am
When we add a new server to Aerospike cluster it begins to synchronize data with other servers in cluster. But while this synchronization is going we have a strange problem - Aerospike clients can’t find a part of keys that have been stored in the aerospike cluster earlier and re-write this keys on new server.
This is normal behavior for Aerospike? Is it possible to restrict the access of clients to the new server while synchronization?
This is definitely not a normal behavior. All records are fully available to the clients during node addition/departure to the cluster.
Which version of the Client and server are you testing?
Could we have more details on your configuration and possibly post your server configuration file and client code for retrieving and testing data.
Thanks
Lucien
by ikochergin » Fri Aug 08, 2014 12:47 am
Lucien, thanks for the quick response. Here is collected data:
We use 2 servers (nodes) identical configuration
Platform: DELL PowerEdge™ R720 DX150
CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
RAM: 128 GB
NETWORK:
Intel(R) GbE 4P I350-t rNDC
eth0 - client connection, info
bond0 (eth1 eth2 eth3) - fabric, migration, mesh, info
STORAGE:
PERC H710 Mini (Embedded)
/dev/sdb1 INTEL SSDSA2BW60 raid0, No Read Ahead, Write Through, 128K
/dev/sdb2 INTEL SSDSA2BW60 raid0, No Read Ahead, Write Through, 128K
Prepared according to the instructions Over-provision in http://www.aerospike.com/docs/operations/plan/ssd/ssd_setup.html
Operating system: Ubuntu 12.04.5 3.11.0-26-generic #45~precise1-Ubuntu SMP
Aerospike:
Server CE 3.3.9
Client Java 3.0.26, requests use AsyncClient
by ikochergin » Fri Aug 08, 2014 12:48 am
Server configuration:
Monitor> compareconfig
IP mesh-address
node1:3000 192.168.1.1
node2:3000 192.168.1.2
Monitor> printconfig
====node1:3000====
allow-inline-transactions true
auto-dun false
auto-undun false
batch-max-requests 5000
batch-priority 200
batch-threads 4
defrag-queue-escape 10
defrag-queue-hwm 20
defrag-queue-lwm 5
defrag-queue-priority 10
dump-message-above-size 134217728
enable-fastpath true
enable-xdr false
fabric-port 3001
fb-health-bad-pct 0
fb-health-good-pct 50
fb-health-msg-per-burst 0
fb-health-msg-timeout 200
heartbeat-address 192.168.1.2
heartbeat-interval 500
heartbeat-mode mesh
heartbeat-port 3002
heartbeat-protocol v2
heartbeat-timeout 30
info-threads 16
max-msgs-per-type -1
memory-accounting false
mesh-address 192.168.1.1
mesh-port 3002
microbenchmarks false
migrate-max-num-incoming 6
migrate-priority 10
migrate-read-priority 10
migrate-read-sleep 500
migrate-rx-lifetime-ms 60000
migrate-threads 6
migrate-xmit-hwm 10
migrate-xmit-lwm 5
migrate-xmit-priority 10
migrate-xmit-sleep 500
network-info-port 3003
nsup-auto-hwm-pct 15
nsup-period 120
nsup-queue-escape 10
nsup-queue-hwm 500
nsup-queue-lwm 1
nsup-startup-evict false
paxos-max-cluster-size 32
paxos-protocol v3
paxos-recovery-policy auto-dun-all
paxos-retransmit-period 5
paxos-single-replica-limit 1
pidfile /var/run/aerospike/asd.pid
prole-extra-ttl 0
proto-fd-idle-ms 60000
proto-fd-max 30000
proxy-hist-track-back 1800
proxy-hist-track-slice 10
proxy-hist-track-thresholds 1,8,64
query-batch-size 100
query-bufpool-size 256
query-hist-track-back 1800
query-hist-track-slice 10
query-hist-track-thresholds 1,8,64
query-in-transaction-thread 0
query-job-tracking false
query-long-q-max-size 500
query-priority 10
query-rec-count-bound 4294967295
query-req-in-query-thread 0
query-req-max-inflight 100
query-short-q-max-size 500
query-sleep 1
query-threads 6
query-threshold 10
query-worker-threads 15
query_rec_count-hist-track-back 1800
query_rec_count-hist-track-slice 10
query_rec_count-hist-track-thresholds 1,8,64
reads-hist-track-back 1800
reads-hist-track-slice 10
reads-hist-track-thresholds 1,8,64
replication-fire-and-forget false
respond-client-on-master-completion false
reuse-address true
scan-priority 200
scan-sleep 1
service-address 0.0.0.0
service-port 3000
sindex-data-max-memory 18446744073709551616
sindex-populator-scan-priority 3
snub-nodes false
stop-writes-noxdr false
storage-benchmarks false
ticker-interval 10
transaction-duplicate-threads 0
transaction-max-ms 1000
transaction-pending-limit 30
transaction-queues 12
transaction-repeatable-read false
transaction-retry-ms 1000
transaction-threads-per-queue 3
udf-hist-track-back 1800
udf-hist-track-slice 10
udf-hist-track-thresholds 1,8,64
udf-runtime-gmax-memory 18446744073709551616
udf-runtime-max-memory 18446744073709551616
use-queue-per-device false
write-duplicate-resolution-disable false
writes_master-hist-track-back 1800
writes_master-hist-track-slice 10
writes_master-hist-track-thresholds 1,8,64
writes_reply-hist-track-back 1800
writes_reply-hist-track-slice 10
writes_reply-hist-track-thresholds 1,8,64
xdr-delete-shipping-enabled true
xdr-nsup-deletes-enabled false
====node2:3000====
allow-inline-transactions true
auto-dun false
auto-undun false
batch-max-requests 5000
batch-priority 200
batch-threads 4
defrag-queue-escape 10
defrag-queue-hwm 20
defrag-queue-lwm 5
defrag-queue-priority 10
dump-message-above-size 134217728
enable-fastpath true
enable-xdr false
fabric-port 3001
fb-health-bad-pct 0
fb-health-good-pct 50
fb-health-msg-per-burst 0
fb-health-msg-timeout 200
heartbeat-address 192.168.1.1
heartbeat-interval 500
heartbeat-mode mesh
heartbeat-port 3002
heartbeat-protocol v2
heartbeat-timeout 30
info-threads 16
max-msgs-per-type -1
memory-accounting false
mesh-address 192.168.1.2
mesh-port 3002
microbenchmarks false
migrate-max-num-incoming 6
migrate-priority 10
migrate-read-priority 10
migrate-read-sleep 500
migrate-rx-lifetime-ms 60000
migrate-threads 6
migrate-xmit-hwm 10
migrate-xmit-lwm 5
migrate-xmit-priority 10
migrate-xmit-sleep 500
network-info-port 3003
nsup-auto-hwm-pct 15
nsup-period 120
nsup-queue-escape 10
nsup-queue-hwm 500
nsup-queue-lwm 1
nsup-startup-evict false
paxos-max-cluster-size 32
paxos-protocol v3
paxos-recovery-policy auto-dun-all
paxos-retransmit-period 5
paxos-single-replica-limit 1
pidfile /var/run/aerospike/asd.pid
prole-extra-ttl 0
proto-fd-idle-ms 60000
proto-fd-max 30000
proxy-hist-track-back 1800
proxy-hist-track-slice 10
proxy-hist-track-thresholds 1,8,64
query-batch-size 100
query-bufpool-size 256
query-hist-track-back 1800
query-hist-track-slice 10
query-hist-track-thresholds 1,8,64
query-in-transaction-thread 0
query-job-tracking false
query-long-q-max-size 500
query-priority 10
query-rec-count-bound 4294967295
query-req-in-query-thread 0
query-req-max-inflight 100
query-short-q-max-size 500
query-sleep 1
query-threads 6
query-threshold 10
query-worker-threads 15
query_rec_count-hist-track-back 1800
query_rec_count-hist-track-slice 10
query_rec_count-hist-track-thresholds 1,8,64
reads-hist-track-back 1800
reads-hist-track-slice 10
reads-hist-track-thresholds 1,8,64
replication-fire-and-forget false
respond-client-on-master-completion false
reuse-address true
scan-priority 200
scan-sleep 1
service-address 0.0.0.0
service-port 3000
sindex-data-max-memory 18446744073709551616
sindex-populator-scan-priority 3
snub-nodes false
stop-writes-noxdr false
storage-benchmarks false
ticker-interval 10
transaction-duplicate-threads 0
transaction-max-ms 1000
transaction-pending-limit 30
transaction-queues 12
transaction-repeatable-read false
transaction-retry-ms 1000
transaction-threads-per-queue 3
udf-hist-track-back 1800
udf-hist-track-slice 10
udf-hist-track-thresholds 1,8,64
udf-runtime-gmax-memory 18446744073709551616
udf-runtime-max-memory 18446744073709551616
use-queue-per-device false
write-duplicate-resolution-disable false
writes_master-hist-track-back 1800
writes_master-hist-track-slice 10
writes_master-hist-track-thresholds 1,8,64
writes_reply-hist-track-back 1800
writes_reply-hist-track-slice 10
writes_reply-hist-track-thresholds 1,8,64
xdr-delete-shipping-enabled true
xdr-nsup-deletes-enabled false
by ikochergin » Fri Aug 08, 2014 12:49 am
Stats:
Used RAM in cluster 125 GB
Used SSD Disk 780 GB
Master objects 838,622,686
Replicated objects 838,629,567
Writes ~ 3000-10000
Reads ~ 300-800
Client connections ~ 100-3000
Monitor> stat
====node1:3000====
batch_errors 0
batch_initiate 2008820799
batch_queue 0
batch_timeout 0
batch_tree_count 1
client_connections 1496
cluster_integrity true
cluster_key C8924A8AB4F3FE1A
cluster_size 2
data-used-bytes-memory 0
err_duplicate_proxy_request 0
err_out_of_space 0
err_replica_non_null_node 0
err_replica_null_node 0
err_rw_cant_put_unique 0
err_rw_pending_limit 75
err_rw_request_not_found 5120
err_storage_queue_full 0
err_sync_copy_null_master 0
err_sync_copy_null_node 0
err_tsvc_requests 2774919
err_write_fail_bin_exists 0
err_write_fail_generation 2226283
err_write_fail_generation_xdr 0
err_write_fail_incompatible_type 0
err_write_fail_key_exists 120442
err_write_fail_key_mismatch 0
err_write_fail_not_found 0
err_write_fail_noxdr 0
err_write_fail_parameter 482105
err_write_fail_prole_delete 1156014
err_write_fail_prole_generation 0
err_write_fail_prole_unknown 0
err_write_fail_unknown 0
fabric_msgs_rcvd 7571909944
fabric_msgs_sent 7571920391
free-pct-disk 55
free-pct-memory 46
heartbeat_received_foreign 2969924
heartbeat_received_self 0
index-used-bytes-memory 53666090240
info_queue 0
migrate_msgs_recv 2426664700
migrate_msgs_sent 2425190422
migrate_num_incoming_accepted 8192
migrate_num_incoming_refused 1338085
migrate_progress_recv 0
migrate_progress_send 0
migrate_rx_objs 0
migrate_tx_objs 0
objects 838532660
ongoing_write_reqs 1
partition_absent 0
partition_actual 2052
partition_desync 0
partition_object_count 838532659
partition_ref_count 4099
partition_replica 2044
paxos_principal BB970D95F3ACAB8
proxy_action 4705
proxy_initiate 9271
proxy_retry 0
proxy_retry_new_dest 0
proxy_retry_q_full 0
proxy_retry_same_dest 0
proxy_unproxy 0
query_abort 2
query_agg 0
query_agg_abort 0
query_agg_avg_rec_count 0
query_agg_err 0
query_agg_success 0
query_avg_rec_count 1
query_fail 3
query_long_queue_full 0
query_long_running 5571
query_lookup_abort 2
query_lookup_avg_rec_count 1
query_lookup_err 0
query_lookup_success 165766227
query_lookups 165766229
query_reqs 165766232
query_short_queue_full 0
query_short_running 165760658
query_success 165766227
query_tracked 20
queue 0
read_dup_prole 287419376
reaped_fds 3334
record_locks 2
record_refs 838614445
rw_err_ack_badnode 0
rw_err_ack_internal 0
rw_err_ack_nomatch 3802
rw_err_dup_cluster_key 751
rw_err_dup_internal 0
rw_err_dup_send 0
rw_err_write_cluster_key 0
rw_err_write_internal 0
rw_err_write_send 0
sindex-used-bytes-memory 13691030399
sindex_gc_locktimedout 6
sindex_ucgarbage_found 1091
stat_cluster_key_err_ack_dup_trans_reenqueue 0
stat_cluster_key_err_ack_rw_trans_reenqueue 0
stat_cluster_key_partition_transaction_queue_count 0
stat_cluster_key_prole_retry 0
stat_cluster_key_regular_processed 0
stat_cluster_key_trans_to_proxy_retry 474
stat_cluster_key_transaction_reenqueue 0
stat_delete_success 457822055
stat_deleted_set_objects 0
stat_duplicate_operation 122993086
stat_evicted_objects 0
stat_evicted_objects_time 0
stat_evicted_set_objects 0
stat_expired_objects 226214511
stat_nsup_deletes_not_shipped 446283288
stat_proxy_errs 0
stat_proxy_reqs 9271
stat_proxy_reqs_xdr 0
stat_proxy_success 9271
stat_read_errs_notfound 180892490
stat_read_errs_other 0
stat_read_reqs 619687446
stat_read_reqs_xdr 0
stat_read_success 438794505
stat_rw_timeout 19080
stat_slow_trans_queue_batch_pop 34
stat_slow_trans_queue_pop 1227
stat_slow_trans_queue_push 1227
stat_write_errs 2774432
stat_write_errs_notfound 0
stat_write_errs_other 2774432
stat_write_reqs 2534815576
stat_write_reqs_xdr 0
stat_write_success 2532022515
stat_xdr_pipe_miss 0
stat_xdr_pipe_writes 0
stat_zero_bin_records 0
storage_defrag_corrupt_record 0
storage_defrag_wait 0
system_free_mem_pct 40
system_sindex_data_memory_used 13691030399
system_swapping false
total-bytes-disk 947268878336
total-bytes-memory 126701535232
transactions 5176019957
tree_count 0
tscan_aborted 0
tscan_initiate 12
tscan_pending 0
tscan_succeeded 13
udf_delete_err_others 0
udf_delete_reqs 0
udf_delete_success 0
udf_lua_errs 0
udf_query_rec_reqs 0
udf_read_errs_other 0
udf_read_reqs 0
udf_read_success 0
udf_replica_writes 0
udf_scan_rec_reqs 0
udf_write_err_others 0
udf_write_reqs 0
udf_write_success 0
uptime 782763
used-bytes-disk 418403531520
used-bytes-memory 67357120639
waiting_transactions 0
write_master 2640886596
write_prole 4772590044
====node2:3000====
batch_errors 0
batch_initiate 1415953601
batch_queue 0
batch_timeout 0
batch_tree_count 0
client_connections 1052
cluster_integrity true
cluster_key C8924A8AB4F3FE1A
cluster_size 2
data-used-bytes-memory 0
err_duplicate_proxy_request 0
err_out_of_space 0
err_replica_non_null_node 0
err_replica_null_node 0
err_rw_cant_put_unique 0
err_rw_pending_limit 0
err_rw_request_not_found 4206
err_storage_queue_full 0
err_sync_copy_null_master 0
err_sync_copy_null_node 0
err_tsvc_requests 3145827
err_write_fail_bin_exists 0
err_write_fail_generation 1894073
err_write_fail_generation_xdr 0
err_write_fail_incompatible_type 0
err_write_fail_key_exists 101745
err_write_fail_key_mismatch 0
err_write_fail_not_found 0
err_write_fail_noxdr 0
err_write_fail_parameter 1652858
err_write_fail_prole_delete 1840997
err_write_fail_prole_generation 0
err_write_fail_prole_unknown 0
err_write_fail_unknown 0
fabric_msgs_rcvd 5233414591
fabric_msgs_sent 5233414600
free-pct-disk 55
free-pct-memory 46
heartbeat_received_foreign 2296751
heartbeat_received_self 0
index-used-bytes-memory 53666090240
info_queue 0
migrate_msgs_recv 1257214060
migrate_msgs_sent 1256497541
migrate_num_incoming_accepted 4096
migrate_num_incoming_refused 0
migrate_progress_recv 0
migrate_progress_send 0
migrate_rx_objs 0
migrate_tx_objs 0
objects 838532660
ongoing_write_reqs 2
partition_absent 0
partition_actual 2044
partition_desync 0
partition_object_count 838532660
partition_ref_count 4098
partition_replica 2052
paxos_principal BB970D95F3ACAB8
proxy_action 9271
proxy_initiate 0
proxy_retry 0
proxy_retry_new_dest 0
proxy_retry_q_full 0
proxy_retry_same_dest 0
proxy_unproxy 0
query_abort 2
query_agg 0
query_agg_abort 0
query_agg_avg_rec_count 0
query_agg_err 0
query_agg_success 0
query_avg_rec_count 1
query_fail 4
query_long_queue_full 0
query_long_running 4106
query_lookup_abort 2
query_lookup_avg_rec_count 1
query_lookup_err 1
query_lookup_success 165741356
query_lookups 165741359
query_reqs 165741362
query_short_queue_full 0
query_short_running 165737253
query_success 165741356
query_tracked 5
queue 0
read_dup_prole 9942874
reaped_fds 1201
record_locks 2
record_refs 838577542
rw_err_ack_badnode 0
rw_err_ack_internal 0
rw_err_ack_nomatch 296
rw_err_dup_cluster_key 0
rw_err_dup_internal 0
rw_err_dup_send 0
rw_err_write_cluster_key 0
rw_err_write_internal 0
rw_err_write_send 0
sindex-used-bytes-memory 13660413310
sindex_gc_locktimedout 7
sindex_ucgarbage_found 3828
stat_cluster_key_err_ack_dup_trans_reenqueue 0
stat_cluster_key_err_ack_rw_trans_reenqueue 0
stat_cluster_key_partition_transaction_queue_count 0
stat_cluster_key_prole_retry 0
stat_cluster_key_regular_processed 0
stat_cluster_key_trans_to_proxy_retry 0
stat_cluster_key_transaction_reenqueue 0
stat_delete_success 367190681
stat_deleted_set_objects 0
stat_duplicate_operation 143653055
stat_evicted_objects 0
stat_evicted_objects_time 0
stat_evicted_set_objects 0
stat_expired_objects 176401542
stat_nsup_deletes_not_shipped 360725544
stat_proxy_errs 0
stat_proxy_reqs 0
stat_proxy_reqs_xdr 0
stat_proxy_success 0
stat_read_errs_notfound 148204722
stat_read_errs_other 0
stat_read_reqs 500391766
stat_read_reqs_xdr 0
stat_read_success 352187040
stat_rw_timeout 302
stat_slow_trans_queue_batch_pop 0
stat_slow_trans_queue_pop 0
stat_slow_trans_queue_push 0
stat_write_errs 3145632
stat_write_errs_notfound 0
stat_write_errs_other 3145632
stat_write_reqs 1846347067
stat_write_reqs_xdr 0
stat_write_success 1843201142
stat_xdr_pipe_miss 0
stat_xdr_pipe_writes 0
stat_zero_bin_records 0
storage_defrag_corrupt_record 0
storage_defrag_wait 4
system_free_mem_pct 41
system_sindex_data_memory_used 13660413310
system_swapping false
total-bytes-disk 947268878336
total-bytes-memory 126701535232
transactions 3808157259
tree_count 0
tscan_aborted 0
tscan_initiate 7
tscan_pending 0
tscan_succeeded 6
udf_delete_err_others 0
udf_delete_reqs 0
udf_delete_success 0
udf_lua_errs 0
udf_query_rec_reqs 0
udf_read_errs_other 0
udf_read_reqs 0
udf_read_success 0
udf_replica_writes 0
udf_scan_rec_reqs 0
udf_write_err_others 0
udf_write_reqs 0
udf_write_success 0
uptime 600854
used-bytes-disk 418403486592
used-bytes-memory 67326503550
waiting_transactions 0
write_master 1966759777
write_prole 3969736736
by ikochergin » Fri Aug 08, 2014 12:50 am
Code:
Client Java 3.0.26, requests use AsyncClient
Loot of RW
READ: ower get(Policy policy, RecordArrayListener listener, Key[] keys)
WRITE: ower put(WritePolicy policy, WriteListener listener, Key key, Bin... bins), or operate(WritePolicy policy, RecordListener listener, Key key, Operation... operations)
Additional description of the code i will write later
Thanks ikochergin
Posts: 12 Joined: Wed Aug 06, 2014 11:39 pm Location: Moscow
by devops01 » Fri Aug 08, 2014 11:23 am
Quick follow up, are you doing batch reads? Data would not be available for a batch read until migrations are completed. Also you seem to have a large Interval of 500 ms configured.
–Lucien
by ikochergin » Tue Aug 12, 2014 7:10 am
Yes our software mainly uses Batch Read. The documentation contains a description of limitation on migration?
We also have delays occur, very similar to the periodic.
Thanks