Cluster synchronization: re-write keys


#1

by ikochergin » Thu Aug 07, 2014 12:09 am

When we add a new server to Aerospike cluster it begins to synchronize data with other servers in cluster. But while this synchronization is going we have a strange problem - Aerospike clients can’t find a part of keys that have been stored in the aerospike cluster earlier and re-write this keys on new server.

This is normal behavior for Aerospike? Is it possible to restrict the access of clients to the new server while synchronization?


#2

This is definitely not a normal behavior. All records are fully available to the clients during node addition/departure to the cluster.

Which version of the Client and server are you testing?

Could we have more details on your configuration and possibly post your server configuration file and client code for retrieving and testing data.

Thanks

Lucien


#3

by ikochergin » Fri Aug 08, 2014 12:47 am

Lucien, thanks for the quick response. Here is collected data:

We use 2 servers (nodes) identical configuration

Platform: DELL PowerEdge™ R720 DX150
CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
RAM: 128 GB
NETWORK:
    Intel(R) GbE 4P I350-t rNDC
    eth0 - client connection, info
    bond0 (eth1 eth2 eth3) - fabric, migration, mesh, info
STORAGE: 
    PERC H710 Mini (Embedded)
    /dev/sdb1 INTEL SSDSA2BW60 raid0, No Read Ahead, Write Through, 128K
    /dev/sdb2 INTEL SSDSA2BW60 raid0, No Read Ahead, Write Through, 128K
    Prepared according to the instructions Over-provision in http://www.aerospike.com/docs/operations/plan/ssd/ssd_setup.html 

Operating system: Ubuntu 12.04.5 3.11.0-26-generic #45~precise1-Ubuntu SMP

Aerospike:
    Server CE 3.3.9
    Client Java 3.0.26, requests use AsyncClient

Server errors after system disk full
Unused data on the disk - Error Code 8: Server memory error
Aerospike hangs when s-index deletes
#4

by ikochergin » Fri Aug 08, 2014 12:48 am

Server configuration:

Monitor> compareconfig
IP                   mesh-address
node1:3000     192.168.1.1
node2:3000     192.168.1.2

Monitor> printconfig
   ====node1:3000====
allow-inline-transactions                                      true
auto-dun                                                      false
auto-undun                                                    false
batch-max-requests                                             5000
batch-priority                                                  200
batch-threads                                                     4
defrag-queue-escape                                              10
defrag-queue-hwm                                                 20
defrag-queue-lwm                                                  5
defrag-queue-priority                                            10
dump-message-above-size                                   134217728
enable-fastpath                                                true
enable-xdr                                                    false
fabric-port                                                    3001
fb-health-bad-pct                                                 0
fb-health-good-pct                                               50
fb-health-msg-per-burst                                           0
fb-health-msg-timeout                                           200
heartbeat-address                                       192.168.1.2
heartbeat-interval                                              500
heartbeat-mode                                                 mesh
heartbeat-port                                                 3002
heartbeat-protocol                                               v2
heartbeat-timeout                                                30
info-threads                                                     16
max-msgs-per-type                                                -1
memory-accounting                                             false
mesh-address                                            192.168.1.1
mesh-port                                                      3002
microbenchmarks                                               false
migrate-max-num-incoming                                          6
migrate-priority                                                 10
migrate-read-priority                                            10
migrate-read-sleep                                              500
migrate-rx-lifetime-ms                                        60000
migrate-threads                                                   6
migrate-xmit-hwm                                                 10
migrate-xmit-lwm                                                  5
migrate-xmit-priority                                            10
migrate-xmit-sleep                                              500
network-info-port                                              3003
nsup-auto-hwm-pct                                                15
nsup-period                                                     120
nsup-queue-escape                                                10
nsup-queue-hwm                                                  500
nsup-queue-lwm                                                    1
nsup-startup-evict                                            false
paxos-max-cluster-size                                           32
paxos-protocol                                                   v3
paxos-recovery-policy                                  auto-dun-all
paxos-retransmit-period                                           5
paxos-single-replica-limit                                        1
pidfile                                  /var/run/aerospike/asd.pid
prole-extra-ttl                                                   0
proto-fd-idle-ms                                              60000
proto-fd-max                                                  30000
proxy-hist-track-back                                          1800
proxy-hist-track-slice                                           10
proxy-hist-track-thresholds                                  1,8,64
query-batch-size                                                100
query-bufpool-size                                              256
query-hist-track-back                                          1800
query-hist-track-slice                                           10
query-hist-track-thresholds                                  1,8,64
query-in-transaction-thread                                       0
query-job-tracking                                            false
query-long-q-max-size                                           500
query-priority                                                   10
query-rec-count-bound                                    4294967295
query-req-in-query-thread                                         0
query-req-max-inflight                                          100
query-short-q-max-size                                          500
query-sleep                                                       1
query-threads                                                     6
query-threshold                                                  10
query-worker-threads                                             15
query_rec_count-hist-track-back                                1800
query_rec_count-hist-track-slice                                 10
query_rec_count-hist-track-thresholds                        1,8,64
reads-hist-track-back                                          1800
reads-hist-track-slice                                           10
reads-hist-track-thresholds                                  1,8,64
replication-fire-and-forget                                   false
respond-client-on-master-completion                           false
reuse-address                                                  true
scan-priority                                                   200
scan-sleep                                                        1
service-address                                             0.0.0.0
service-port                                                   3000
sindex-data-max-memory                         18446744073709551616
sindex-populator-scan-priority                                    3
snub-nodes                                                    false
stop-writes-noxdr                                             false
storage-benchmarks                                            false
ticker-interval                                                  10
transaction-duplicate-threads                                     0
transaction-max-ms                                             1000
transaction-pending-limit                                        30
transaction-queues                                               12
transaction-repeatable-read                                   false
transaction-retry-ms                                           1000
transaction-threads-per-queue                                     3
udf-hist-track-back                                            1800
udf-hist-track-slice                                             10
udf-hist-track-thresholds                                    1,8,64
udf-runtime-gmax-memory                        18446744073709551616
udf-runtime-max-memory                         18446744073709551616
use-queue-per-device                                          false
write-duplicate-resolution-disable                            false
writes_master-hist-track-back                                  1800
writes_master-hist-track-slice                                   10
writes_master-hist-track-thresholds                          1,8,64
writes_reply-hist-track-back                                   1800
writes_reply-hist-track-slice                                    10
writes_reply-hist-track-thresholds                           1,8,64
xdr-delete-shipping-enabled                                    true
xdr-nsup-deletes-enabled                                      false
   ====node2:3000====
allow-inline-transactions                                      true
auto-dun                                                      false
auto-undun                                                    false
batch-max-requests                                             5000
batch-priority                                                  200
batch-threads                                                     4
defrag-queue-escape                                              10
defrag-queue-hwm                                                 20
defrag-queue-lwm                                                  5
defrag-queue-priority                                            10
dump-message-above-size                                   134217728
enable-fastpath                                                true
enable-xdr                                                    false
fabric-port                                                    3001
fb-health-bad-pct                                                 0
fb-health-good-pct                                               50
fb-health-msg-per-burst                                           0
fb-health-msg-timeout                                           200
heartbeat-address                                       192.168.1.1
heartbeat-interval                                              500
heartbeat-mode                                                 mesh
heartbeat-port                                                 3002
heartbeat-protocol                                               v2
heartbeat-timeout                                                30
info-threads                                                     16
max-msgs-per-type                                                -1
memory-accounting                                             false
mesh-address                                            192.168.1.2
mesh-port                                                      3002
microbenchmarks                                               false
migrate-max-num-incoming                                          6
migrate-priority                                                 10
migrate-read-priority                                            10
migrate-read-sleep                                              500
migrate-rx-lifetime-ms                                        60000
migrate-threads                                                   6
migrate-xmit-hwm                                                 10
migrate-xmit-lwm                                                  5
migrate-xmit-priority                                            10
migrate-xmit-sleep                                              500
network-info-port                                              3003
nsup-auto-hwm-pct                                                15
nsup-period                                                     120
nsup-queue-escape                                                10
nsup-queue-hwm                                                  500
nsup-queue-lwm                                                    1
nsup-startup-evict                                            false
paxos-max-cluster-size                                           32
paxos-protocol                                                   v3
paxos-recovery-policy                                  auto-dun-all
paxos-retransmit-period                                           5
paxos-single-replica-limit                                        1
pidfile                                  /var/run/aerospike/asd.pid
prole-extra-ttl                                                   0
proto-fd-idle-ms                                              60000
proto-fd-max                                                  30000
proxy-hist-track-back                                          1800
proxy-hist-track-slice                                           10
proxy-hist-track-thresholds                                  1,8,64
query-batch-size                                                100
query-bufpool-size                                              256
query-hist-track-back                                          1800
query-hist-track-slice                                           10
query-hist-track-thresholds                                  1,8,64
query-in-transaction-thread                                       0
query-job-tracking                                            false
query-long-q-max-size                                           500
query-priority                                                   10
query-rec-count-bound                                    4294967295
query-req-in-query-thread                                         0
query-req-max-inflight                                          100
query-short-q-max-size                                          500
query-sleep                                                       1
query-threads                                                     6
query-threshold                                                  10
query-worker-threads                                             15
query_rec_count-hist-track-back                                1800
query_rec_count-hist-track-slice                                 10
query_rec_count-hist-track-thresholds                        1,8,64
reads-hist-track-back                                          1800
reads-hist-track-slice                                           10
reads-hist-track-thresholds                                  1,8,64
replication-fire-and-forget                                   false
respond-client-on-master-completion                           false
reuse-address                                                  true
scan-priority                                                   200
scan-sleep                                                        1
service-address                                             0.0.0.0
service-port                                                   3000
sindex-data-max-memory                         18446744073709551616
sindex-populator-scan-priority                                    3
snub-nodes                                                    false
stop-writes-noxdr                                             false
storage-benchmarks                                            false
ticker-interval                                                  10
transaction-duplicate-threads                                     0
transaction-max-ms                                             1000
transaction-pending-limit                                        30
transaction-queues                                               12
transaction-repeatable-read                                   false
transaction-retry-ms                                           1000
transaction-threads-per-queue                                     3
udf-hist-track-back                                            1800
udf-hist-track-slice                                             10
udf-hist-track-thresholds                                    1,8,64
udf-runtime-gmax-memory                        18446744073709551616
udf-runtime-max-memory                         18446744073709551616
use-queue-per-device                                          false
write-duplicate-resolution-disable                            false
writes_master-hist-track-back                                  1800
writes_master-hist-track-slice                                   10
writes_master-hist-track-thresholds                          1,8,64
writes_reply-hist-track-back                                   1800
writes_reply-hist-track-slice                                    10
writes_reply-hist-track-thresholds                           1,8,64
xdr-delete-shipping-enabled                                    true
xdr-nsup-deletes-enabled                                      false

#5

by ikochergin » Fri Aug 08, 2014 12:49 am

Stats:
   Used RAM in cluster 125 GB
   Used SSD Disk 780 GB
   Master objects 838,622,686
   Replicated objects 838,629,567
   Writes ~ 3000-10000
   Reads  ~ 300-800
   Client connections ~ 100-3000

Monitor> stat
   ====node1:3000====
batch_errors                                                         0
batch_initiate                                              2008820799
batch_queue                                                          0
batch_timeout                                                        0
batch_tree_count                                                     1
client_connections                                                1496
cluster_integrity                                                 true
cluster_key                                           C8924A8AB4F3FE1A
cluster_size                                                         2
data-used-bytes-memory                                               0
err_duplicate_proxy_request                                          0
err_out_of_space                                                     0
err_replica_non_null_node                                            0
err_replica_null_node                                                0
err_rw_cant_put_unique                                               0
err_rw_pending_limit                                                75
err_rw_request_not_found                                          5120
err_storage_queue_full                                               0
err_sync_copy_null_master                                            0
err_sync_copy_null_node                                              0
err_tsvc_requests                                              2774919
err_write_fail_bin_exists                                            0
err_write_fail_generation                                      2226283
err_write_fail_generation_xdr                                        0
err_write_fail_incompatible_type                                     0
err_write_fail_key_exists                                       120442
err_write_fail_key_mismatch                                          0
err_write_fail_not_found                                             0
err_write_fail_noxdr                                                 0
err_write_fail_parameter                                        482105
err_write_fail_prole_delete                                    1156014
err_write_fail_prole_generation                                      0
err_write_fail_prole_unknown                                         0
err_write_fail_unknown                                               0
fabric_msgs_rcvd                                            7571909944
fabric_msgs_sent                                            7571920391
free-pct-disk                                                       55
free-pct-memory                                                     46
heartbeat_received_foreign                                     2969924
heartbeat_received_self                                              0
index-used-bytes-memory                                    53666090240
info_queue                                                           0
migrate_msgs_recv                                           2426664700
migrate_msgs_sent                                           2425190422
migrate_num_incoming_accepted                                     8192
migrate_num_incoming_refused                                   1338085
migrate_progress_recv                                                0
migrate_progress_send                                                0
migrate_rx_objs                                                      0
migrate_tx_objs                                                      0
objects                                                      838532660
ongoing_write_reqs                                                   1
partition_absent                                                     0
partition_actual                                                  2052
partition_desync                                                     0
partition_object_count                                       838532659
partition_ref_count                                               4099
partition_replica                                                 2044
paxos_principal                                        BB970D95F3ACAB8
proxy_action                                                      4705
proxy_initiate                                                    9271
proxy_retry                                                          0
proxy_retry_new_dest                                                 0
proxy_retry_q_full                                                   0
proxy_retry_same_dest                                                0
proxy_unproxy                                                        0
query_abort                                                          2
query_agg                                                            0
query_agg_abort                                                      0
query_agg_avg_rec_count                                              0
query_agg_err                                                        0
query_agg_success                                                    0
query_avg_rec_count                                                  1
query_fail                                                           3
query_long_queue_full                                                0
query_long_running                                                5571
query_lookup_abort                                                   2
query_lookup_avg_rec_count                                           1
query_lookup_err                                                     0
query_lookup_success                                         165766227
query_lookups                                                165766229
query_reqs                                                   165766232
query_short_queue_full                                               0
query_short_running                                          165760658
query_success                                                165766227
query_tracked                                                       20
queue                                                                0
read_dup_prole                                               287419376
reaped_fds                                                        3334
record_locks                                                         2
record_refs                                                  838614445
rw_err_ack_badnode                                                   0
rw_err_ack_internal                                                  0
rw_err_ack_nomatch                                                3802
rw_err_dup_cluster_key                                             751
rw_err_dup_internal                                                  0
rw_err_dup_send                                                      0
rw_err_write_cluster_key                                             0
rw_err_write_internal                                                0
rw_err_write_send                                                    0
sindex-used-bytes-memory                                   13691030399
sindex_gc_locktimedout                                               6
sindex_ucgarbage_found                                            1091
stat_cluster_key_err_ack_dup_trans_reenqueue                         0
stat_cluster_key_err_ack_rw_trans_reenqueue                          0
stat_cluster_key_partition_transaction_queue_count                   0
stat_cluster_key_prole_retry                                         0
stat_cluster_key_regular_processed                                   0
stat_cluster_key_trans_to_proxy_retry                              474
stat_cluster_key_transaction_reenqueue                               0
stat_delete_success                                          457822055
stat_deleted_set_objects                                             0
stat_duplicate_operation                                     122993086
stat_evicted_objects                                                 0
stat_evicted_objects_time                                            0
stat_evicted_set_objects                                             0
stat_expired_objects                                         226214511
stat_nsup_deletes_not_shipped                                446283288
stat_proxy_errs                                                      0
stat_proxy_reqs                                                   9271
stat_proxy_reqs_xdr                                                  0
stat_proxy_success                                                9271
stat_read_errs_notfound                                      180892490
stat_read_errs_other                                                 0
stat_read_reqs                                               619687446
stat_read_reqs_xdr                                                   0
stat_read_success                                            438794505
stat_rw_timeout                                                  19080
stat_slow_trans_queue_batch_pop                                     34
stat_slow_trans_queue_pop                                         1227
stat_slow_trans_queue_push                                        1227
stat_write_errs                                                2774432
stat_write_errs_notfound                                             0
stat_write_errs_other                                          2774432
stat_write_reqs                                             2534815576
stat_write_reqs_xdr                                                  0
stat_write_success                                          2532022515
stat_xdr_pipe_miss                                                   0
stat_xdr_pipe_writes                                                 0
stat_zero_bin_records                                                0
storage_defrag_corrupt_record                                        0
storage_defrag_wait                                                  0
system_free_mem_pct                                                 40
system_sindex_data_memory_used                             13691030399
system_swapping                                                  false
total-bytes-disk                                          947268878336
total-bytes-memory                                        126701535232
transactions                                                5176019957
tree_count                                                           0
tscan_aborted                                                        0
tscan_initiate                                                      12
tscan_pending                                                        0
tscan_succeeded                                                     13
udf_delete_err_others                                                0
udf_delete_reqs                                                      0
udf_delete_success                                                   0
udf_lua_errs                                                         0
udf_query_rec_reqs                                                   0
udf_read_errs_other                                                  0
udf_read_reqs                                                        0
udf_read_success                                                     0
udf_replica_writes                                                   0
udf_scan_rec_reqs                                                    0
udf_write_err_others                                                 0
udf_write_reqs                                                       0
udf_write_success                                                    0
uptime                                                          782763
used-bytes-disk                                           418403531520
used-bytes-memory                                          67357120639
waiting_transactions                                                 0
write_master                                                2640886596
write_prole                                                 4772590044
   ====node2:3000====
batch_errors                                                         0
batch_initiate                                              1415953601
batch_queue                                                          0
batch_timeout                                                        0
batch_tree_count                                                     0
client_connections                                                1052
cluster_integrity                                                 true
cluster_key                                           C8924A8AB4F3FE1A
cluster_size                                                         2
data-used-bytes-memory                                               0
err_duplicate_proxy_request                                          0
err_out_of_space                                                     0
err_replica_non_null_node                                            0
err_replica_null_node                                                0
err_rw_cant_put_unique                                               0
err_rw_pending_limit                                                 0
err_rw_request_not_found                                          4206
err_storage_queue_full                                               0
err_sync_copy_null_master                                            0
err_sync_copy_null_node                                              0
err_tsvc_requests                                              3145827
err_write_fail_bin_exists                                            0
err_write_fail_generation                                      1894073
err_write_fail_generation_xdr                                        0
err_write_fail_incompatible_type                                     0
err_write_fail_key_exists                                       101745
err_write_fail_key_mismatch                                          0
err_write_fail_not_found                                             0
err_write_fail_noxdr                                                 0
err_write_fail_parameter                                       1652858
err_write_fail_prole_delete                                    1840997
err_write_fail_prole_generation                                      0
err_write_fail_prole_unknown                                         0
err_write_fail_unknown                                               0
fabric_msgs_rcvd                                            5233414591
fabric_msgs_sent                                            5233414600
free-pct-disk                                                       55
free-pct-memory                                                     46
heartbeat_received_foreign                                     2296751
heartbeat_received_self                                              0
index-used-bytes-memory                                    53666090240
info_queue                                                           0
migrate_msgs_recv                                           1257214060
migrate_msgs_sent                                           1256497541
migrate_num_incoming_accepted                                     4096
migrate_num_incoming_refused                                         0
migrate_progress_recv                                                0
migrate_progress_send                                                0
migrate_rx_objs                                                      0
migrate_tx_objs                                                      0
objects                                                      838532660
ongoing_write_reqs                                                   2
partition_absent                                                     0
partition_actual                                                  2044
partition_desync                                                     0
partition_object_count                                       838532660
partition_ref_count                                               4098
partition_replica                                                 2052
paxos_principal                                        BB970D95F3ACAB8
proxy_action                                                      9271
proxy_initiate                                                       0
proxy_retry                                                          0
proxy_retry_new_dest                                                 0
proxy_retry_q_full                                                   0
proxy_retry_same_dest                                                0
proxy_unproxy                                                        0
query_abort                                                          2
query_agg                                                            0
query_agg_abort                                                      0
query_agg_avg_rec_count                                              0
query_agg_err                                                        0
query_agg_success                                                    0
query_avg_rec_count                                                  1
query_fail                                                           4
query_long_queue_full                                                0
query_long_running                                                4106
query_lookup_abort                                                   2
query_lookup_avg_rec_count                                           1
query_lookup_err                                                     1
query_lookup_success                                         165741356
query_lookups                                                165741359
query_reqs                                                   165741362
query_short_queue_full                                               0
query_short_running                                          165737253
query_success                                                165741356
query_tracked                                                        5
queue                                                                0
read_dup_prole                                                 9942874
reaped_fds                                                        1201
record_locks                                                         2
record_refs                                                  838577542
rw_err_ack_badnode                                                   0
rw_err_ack_internal                                                  0
rw_err_ack_nomatch                                                 296
rw_err_dup_cluster_key                                               0
rw_err_dup_internal                                                  0
rw_err_dup_send                                                      0
rw_err_write_cluster_key                                             0
rw_err_write_internal                                                0
rw_err_write_send                                                    0
sindex-used-bytes-memory                                   13660413310
sindex_gc_locktimedout                                               7
sindex_ucgarbage_found                                            3828
stat_cluster_key_err_ack_dup_trans_reenqueue                         0
stat_cluster_key_err_ack_rw_trans_reenqueue                          0
stat_cluster_key_partition_transaction_queue_count                   0
stat_cluster_key_prole_retry                                         0
stat_cluster_key_regular_processed                                   0
stat_cluster_key_trans_to_proxy_retry                                0
stat_cluster_key_transaction_reenqueue                               0
stat_delete_success                                          367190681
stat_deleted_set_objects                                             0
stat_duplicate_operation                                     143653055
stat_evicted_objects                                                 0
stat_evicted_objects_time                                            0
stat_evicted_set_objects                                             0
stat_expired_objects                                         176401542
stat_nsup_deletes_not_shipped                                360725544
stat_proxy_errs                                                      0
stat_proxy_reqs                                                      0
stat_proxy_reqs_xdr                                                  0
stat_proxy_success                                                   0
stat_read_errs_notfound                                      148204722
stat_read_errs_other                                                 0
stat_read_reqs                                               500391766
stat_read_reqs_xdr                                                   0
stat_read_success                                            352187040
stat_rw_timeout                                                    302
stat_slow_trans_queue_batch_pop                                      0
stat_slow_trans_queue_pop                                            0
stat_slow_trans_queue_push                                           0
stat_write_errs                                                3145632
stat_write_errs_notfound                                             0
stat_write_errs_other                                          3145632
stat_write_reqs                                             1846347067
stat_write_reqs_xdr                                                  0
stat_write_success                                          1843201142
stat_xdr_pipe_miss                                                   0
stat_xdr_pipe_writes                                                 0
stat_zero_bin_records                                                0
storage_defrag_corrupt_record                                        0
storage_defrag_wait                                                  4
system_free_mem_pct                                                 41
system_sindex_data_memory_used                             13660413310
system_swapping                                                  false
total-bytes-disk                                          947268878336
total-bytes-memory                                        126701535232
transactions                                                3808157259
tree_count                                                           0
tscan_aborted                                                        0
tscan_initiate                                                       7
tscan_pending                                                        0
tscan_succeeded                                                      6
udf_delete_err_others                                                0
udf_delete_reqs                                                      0
udf_delete_success                                                   0
udf_lua_errs                                                         0
udf_query_rec_reqs                                                   0
udf_read_errs_other                                                  0
udf_read_reqs                                                        0
udf_read_success                                                     0
udf_replica_writes                                                   0
udf_scan_rec_reqs                                                    0
udf_write_err_others                                                 0
udf_write_reqs                                                       0
udf_write_success                                                    0
uptime                                                          600854
used-bytes-disk                                           418403486592
used-bytes-memory                                          67326503550
waiting_transactions                                                 0
write_master                                                1966759777
write_prole                                                 3969736736

#6

by ikochergin » Fri Aug 08, 2014 12:50 am

Code:
    Client Java 3.0.26, requests use AsyncClient
    Loot of RW 
    READ: ower  get(Policy policy, RecordArrayListener listener, Key[] keys)
    WRITE: ower put(WritePolicy policy, WriteListener listener, Key key, Bin... bins), or operate(WritePolicy policy, RecordListener listener, Key key, Operation... operations)

Additional description of the code i will write later

Thanks ikochergin

Posts: 12 Joined: Wed Aug 06, 2014 11:39 pm Location: Moscow


#7

by devops01 » Fri Aug 08, 2014 11:23 am

Quick follow up, are you doing batch reads? Data would not be available for a batch read until migrations are completed. Also you seem to have a large Interval of 500 ms configured.

–Lucien


#8

by ikochergin » Tue Aug 12, 2014 7:10 am

Yes our software mainly uses Batch Read. The documentation contains a description of limitation on migration?

We also have delays occur, very similar to the periodic.

Thanks