Network utilisation at migration

Operations
1

2457×1027 1.17 MB

Hi We have some strange situation. After our cluster restarted and migration begin we get full internal (fabric) network utilisation on one node in cluster. When we stop overloaded node, another take ther place. Then another and another one by one.

Tuning migration and client settings did not give any effect. We also noticed that the number of client connections on this node is greater x10 than the other. Only one node in cluster overloaded at some time.

This is normal behavior, or we have a problem with the configuration?

P.S Cluster 10 nodes Intel® Xeon® E5-1650 v3, 256 GB RAM, 4x480 SSD AS v3.9.1.1 Disk utilisation 67% RAM utilisation 46% aerospike-client-3.1.4.jar Ubuntu 14.04 4.2.0-42-generic #49~14.04.1-Ubuntu Network: Internal 1 GBit/s, External 200 Mbit/s

2

Please provide the output of:

asadm -e "show config like migra"
asadm -e "show stat like migra"
3 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Service Configuration~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                    :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-max-num-incoming:   256                           256                            256                           256                           256                           256                           256                           256                           256                           256
migrate-rx-lifetime-ms  :   60000                         60000                          60000                         60000                         60000                         60000                         60000                         60000                         60000                         60000
migrate-threads         :   1                             1                              1                             1                             1                             1                             1                             1                             1                             1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ssp Namespace Configuration~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE         :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-order:   2                             2                              2                             2                             2                             2                             2                             2                             2                             2
migrate-sleep:   1                             1                              1                             1                             1                             1                             1                             1                             1                             1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~enrichment Namespace Configuration~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE         :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-order:   4                             4                              4                             4                             4                             4                             4                             4                             4                             4
migrate-sleep:   1                             1                              1                             1                             1                             1                             1                             1                             1                             1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~dsp Namespace Configuration~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE         :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-order:   1                             1                              1                             1                             1                             1                             1                             1                             1                             1
migrate-sleep:   1                             1                              1                             1                             1                             1                             1                             1                             1                             1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~xdr Namespace Configuration~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE         :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-order:   3                             3                              3                             3                             3                             3                             3                             3                             3                             3
migrate-sleep:   1                             1                              1                             1                             1                             1                             1                             1                             1                             1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Service Statistics~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                        :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate_allowed             :   true                          true                           true                          true                          true                          true                          true                          true                          true                          true
migrate_partitions_remaining:   297                           972                            302                           276                           255                           273                           343                           291                           253                           1686

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~dsp Namespace Statistics~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                           :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-order                  :   1                             1                              1                             1                             1                             1                             1                             1                             1                             1
migrate-sleep                  :   1                             1                              1                             1                             1                             1                             1                             1                             1                             1
migrate_record_receives        :   8314056                       1299341                        5266968                       5264185                       6656615                       7553317                       8337472                       5557640                       29798928                      934256
migrate_record_retransmits     :   7210943                       0                              579                           0                             19462                         403642                        3638844                       0                             687955                        0
migrate_records_skipped        :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_records_transmitted    :   9186235                       76964                          5217004                       4362946                       6110237                       8164597                       9513481                       4262541                       8774354                       1050960
migrate_rx_instances           :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_rx_partitions_active   :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_rx_partitions_initial  :   95                            113                            94                            95                            81                            95                            111                           91                            85                            827
migrate_rx_partitions_remaining:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_instances           :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_active   :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_imbalance:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_initial  :   95                            96                             94                            95                            81                            95                            107                           100                           85                            839
migrate_tx_partitions_remaining:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~enrichment Namespace Statistics~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                           :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-order                  :   4                             4                              4                             4                             4                             4                             4                             4                             4                             4
migrate-sleep                  :   1                             1                              1                             1                             1                             1                             1                             1                             1                             1
migrate_record_receives        :   6163854005                    124639891                      2366146764                    2468812360                    2661885601                    3838430692                    4316964552                    3049274222                    5922336857                    112621927
migrate_record_retransmits     :   122552830                     0                              33934                         32055                         92828                         93729276                      4480169                       64305                         47419170                      0
migrate_records_skipped        :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_records_transmitted    :   4859353207                    0                              3205272405                    1591933878                    3042256311                    4764512122                    4999047209                    1936809212                    6184946260                    25724929
migrate_rx_instances           :   1                             4                              0                             0                             0                             0                             0                             0                             0                             5
migrate_rx_partitions_active   :   1                             3                              0                             0                             0                             0                             0                             0                             0                             5
migrate_rx_partitions_initial  :   109                           944                            103                           112                           93                            101                           119                           100                           92                            827
migrate_rx_partitions_remaining:   105                           880                            103                           101                           93                            101                           119                           100                           92                            780
migrate_tx_instances           :   137                           0                              125                           122                           117                           111                           154                           135                           106                           517
migrate_tx_partitions_active   :   1                             0                              1                             1                             1                             1                             1                             1                             1                             1
migrate_tx_partitions_imbalance:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_initial  :   205                           92                             212                           194                           175                           185                           238                           205                           173                           921
migrate_tx_partitions_remaining:   192                           92                             199                           175                           162                           172                           224                           191                           161                           906

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ssp Namespace Statistics~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                           :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-order                  :   2                             2                              2                             2                             2                             2                             2                             2                             2                             2
migrate-sleep                  :   1                             1                              1                             1                             1                             1                             1                             1                             1                             1
migrate_record_receives        :   671844                        79250                          510110                        407811                        532657                        716911                        780399                        420819                        809823                        112489
migrate_record_retransmits     :   0                             0                              0                             0                             0                             11616                         18863                         0                             22655                         0
migrate_records_skipped        :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_records_transmitted    :   617595                        12959                          465725                        340865                        518088                        769396                        777431                        323183                        848128                        97396
migrate_rx_instances           :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_rx_partitions_active   :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_rx_partitions_initial  :   95                            274                            94                            95                            81                            95                            119                           91                            85                            827
migrate_rx_partitions_remaining:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_instances           :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_active   :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_imbalance:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_initial  :   106                           101                            94                            95                            81                            95                            154                           198                           85                            847
migrate_tx_partitions_remaining:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~xdr Namespace Statistics~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                           :                node1:3000                node10:3000                node2:3000                node3:3000                node4:3000                node5:3000                node6:3000                node7:3000                node8:3000                node9:3000
migrate-order                  :   3                             3                              3                             3                             3                             3                             3                             3                             3                             3
migrate-sleep                  :   1                             1                              1                             1                             1                             1                             1                             1                             1                             1
migrate_record_receives        :   15816220                      2005662                        9470415                       9483809                       10221858                      11676590                      11241409                      10796494                      16984065                      1525595
migrate_record_retransmits     :   525                           0                              67                            0                             0                             0                             238                           0                             37156                         0
migrate_records_skipped        :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_records_transmitted    :   44469025                      113244                         4142554                       3202419                       4997549                       5438918                       6838084                       3468829                       16641937                      2405578
migrate_rx_instances           :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_rx_partitions_active   :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_rx_partitions_initial  :   100                           590                            94                            95                            81                            99                            119                           91                            92                            827
migrate_rx_partitions_remaining:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_instances           :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_active   :   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_imbalance:   0                             0                              0                             0                             0                             0                             0                             0                             0                             0
migrate_tx_partitions_initial  :   204                           103                            144                           95                            81                            122                           224                           196                           156                           863
migrate_tx_partitions_remaining:   0                             0                              0                             0                             0                             0                             0                             0                             0
4

Also While our software is stopped, traffic return to the some level like on other nodes. All overloaded trafic genetrates from 3001 port

fabric {
    address                            192.168.200.20
    port                               3001
}
5

There are two migration throttles, migrate-sleep and migrate-threads. The migrate-sleep is the amount of time (in microseconds) each thread will wait before sending the next record, I would try increasing the sleep to a millisecond per record.

asadm -e "set-config:context=namespace;id=ssp;migrate-sleep=1000"
asadm -e "set-config:context=namespace;id=enrichment;migrate-sleep=1000"
asadm -e "set-config:context=namespace;id=dsp;migrate-sleep=1000"
asadm -e "set-config:context=namespace;id=xdr;migrate-sleep=1000"

Note: this will slow down migrations, if you find that you find you could spend more bandwidth (or less) on migrations I would recommend to continue to tuning this setting for your environment.

6

Neither changing these settings nor disabling migrations at all has no effect.

By “disabling migrations” we mean setting:

migrate-threads                        0
migrate-max-num-incoming               0

and waiting all migraion threads to stop.

We are going to restart all cluster’s nodes and see if that helps.

I will post results laiter.

7

So we did restart all aerospike nodes. We also upgraded them to version 3.10.0.1 along the way, as well as client library (java v3.3.0).

Still no effect. There is always one node that use all available bandwidth and there is always another one doing the same when we stopped the first one.

Can a bounded bandwith to be the cause? It’s 1 gigabit at the moment.

8 
Admin> show config
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                               :   node-1:3000   
advertise-ipv6                     :   false                         
allow-inline-transactions          :   true                          
batch-index-threads                :   4                             
batch-max-buffers-per-queue        :   255                           
batch-max-requests                 :   5000                          
batch-max-unused-buffers           :   256                           
batch-priority                     :   30                            
batch-threads                      :   4                             
clock-skew-max-ms                  :   1000                          
cluster-name                       :   cluster           
enable-benchmarks-fabric           :   false                         
enable-benchmarks-svc              :   false                         
enable-hist-info                   :   false                         
fabric-dump-msgs                   :   false                         
fabric-workers                     :   16                            
hist-track-back                    :   300                           
hist-track-slice                   :   10                            
hist-track-thresholds              :   null                          
info-threads                       :   16                            
ldt-benchmarks                     :   false                         
log-local-time                     :   false                         
max-msgs-per-type                  :   -1                            
memory-accounting                  :   false                         
migrate-max-num-incoming           :   16                            
migrate-rx-lifetime-ms             :   60000                         
migrate-threads                    :   1                             
node-id-interface                  :   null                          
non-master-sets-delete             :   false                         
nsup-delete-sleep                  :   10000                         
nsup-period                        :   600                           
nsup-startup-evict                 :   true                          
paxos-max-cluster-size             :   32                            
paxos-protocol                     :   v3                            
paxos-recovery-policy              :   auto-reset-master             
paxos-retransmit-period            :   5                             
paxos-single-replica-limit         :   1                             
pidfile                            :   /var/run/aerospike/asd.pid    
prole-extra-ttl                    :   0                             
proto-fd-idle-ms                   :   60000                         
proto-fd-max                       :   90000                         
proto-slow-netio-sleep-ms          :   1                             
query-batch-size                   :   100                           
query-buf-size                     :   2097152                       
query-bufpool-size                 :   256                           
query-in-transaction-thread        :   false                         
query-long-q-max-size              :   500                           
query-microbenchmark               :   false                         
query-pre-reserve-partitions       :   false                         
query-priority                     :   30                            
query-priority-sleep-us            :   1                             
query-rec-count-bound              :   18446744073709551615          
query-req-in-query-thread          :   false                         
query-req-max-inflight             :   100                           
query-short-q-max-size             :   500                           
query-threads                      :   6                             
query-threshold                    :   10                            
query-untracked-time-ms            :   1000                          
query-worker-threads               :   15                            
respond-client-on-master-completion:   false                         
run-as-daemon                      :   true                          
scan-max-active                    :   100                           
scan-max-done                      :   100                           
scan-max-udf-transactions          :   32                            
scan-threads                       :   4                             
service-threads                    :   16                            
sindex-builder-threads             :   4                             
sindex-data-max-memory             :   ULONG_MAX                     
sindex-gc-enable-histogram         :   false                         
ticker-interval                    :   10                            
transaction-max-ms                 :   1000                          
transaction-pending-limit          :   128                           
transaction-queues                 :   12                            
transaction-repeatable-read        :   false                         
transaction-retry-ms               :   1000                          
transaction-threads-per-queue      :   6                             
use-queue-per-device               :   false                         
work-directory                     :   /opt/aerospike                
write-duplicate-resolution-disable :   false                         

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                            :   node-1:3000                         
fabric.address                  :   any                                                 
fabric.keepalive-enabled        :   true                                                
fabric.keepalive-intvl          :   1                                                   
fabric.keepalive-probes         :   10                                                  
fabric.keepalive-time           :   1                                                   
fabric.latency-max-ms           :   0                                                   
fabric.port                     :   3001                                                
heartbeat.address               :   int-node-1                          
heartbeat.fabric-grace-factor   :   -1                                                  
heartbeat.interval              :   100                                                 
heartbeat.mode                  :   multicast                                           
heartbeat.mtu                   :   1500                                                
heartbeat.multicast-group       :   239.1.99.2                                          
heartbeat.port                  :   9918                                                
heartbeat.protocol              :   v2                                                  
heartbeat.timeout               :   10                                                  
info.address                    :   any                                                 
info.port                       :   3003                                                
service.access-address          :   int-node-1,node-1   
service.address                 :   any                                                 
service.alternate-access-address:   int-node-1                          
service.alternate-port          :   3000                                                
service.alternate-tls-port      :   0                                                   
service.port                    :   3000                                                
service.tls-name                :   null                                                
service.tls-port                :   0                                                   

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                                    :   node-1:3000               
allow-nonxdr-writes                     :   true                                      
allow-xdr-writes                        :   true                                      
cold-start-evict-ttl                    :   4294967295                                
conflict-resolution-policy              :   generation                                
data-in-index                           :   false                                     
default-ttl                             :   0                                         
disallow-null-setname                   :   false                                     
enable-benchmarks-batch-sub             :   false                                     
enable-benchmarks-read                  :   false                                     
enable-benchmarks-udf                   :   false                                     
enable-benchmarks-udf-sub               :   false                                     
enable-benchmarks-write                 :   false                                     
enable-hist-proxy                       :   false                                     
enable-xdr                              :   false                                     
evict-hist-buckets                      :   10000                                     
evict-tenths-pct                        :   5                                         
geo2dsphere-within.earth-radius-meters  :   6371000                                   
geo2dsphere-within.level-mod            :   1                                         
geo2dsphere-within.max-cells            :   12                                        
geo2dsphere-within.max-level            :   30                                        
geo2dsphere-within.min-level            :   1                                         
geo2dsphere-within.strict               :   true                                      
high-water-disk-pct                     :   99                                        
high-water-memory-pct                   :   99                                        
ldt-enabled                             :   false                                     
ldt-gc-rate                             :   0                                         
ldt-page-size                           :   8192                                      
max-ttl                                 :   315360000                                 
memory-size                             :   30064771072                               
migrate-order                           :   2                                         
migrate-sleep                           :   1                                         
ns-forward-xdr-writes                   :   false                                     
read-consistency-level-override         :   one                                       
repl-factor                             :   2                                         
sets-enable-xdr                         :   true                                      
sindex.data-max-memory                  :   ULONG_MAX                                 
sindex.num-partitions                   :   32                                        
single-bin                              :   false                                     
stop-writes-pct                         :   99                                        
storage-engine                          :   device                                    
storage-engine.cold-start-empty         :   false                                     
storage-engine.data-in-memory           :   true                                      
storage-engine.defrag-lwm-pct           :   50                                        
storage-engine.defrag-queue-min         :   0                                         
storage-engine.defrag-sleep             :   1000                                      
storage-engine.defrag-startup-minimum   :   10                                        
storage-engine.device                   :   /dev/sda7,/dev/sdb7,/dev/sdc7,/dev/sdd7   c7,/dev/sdd7
storage-engine.disable-odirect          :   false                                     
storage-engine.enable-benchmarks-storage:   false                                     
storage-engine.enable-osync             :   false                                     
storage-engine.filesize                 :   17179869184                               
storage-engine.flush-max-ms             :   1000                                      
storage-engine.fsync-max-sec            :   0                                         
storage-engine.max-write-cache          :   134217728                                 
storage-engine.min-avail-pct            :   1                                         
storage-engine.post-write-queue         :   0                                         
storage-engine.scheduler-mode           :   noop                                      
storage-engine.tomb-raider-sleep        :   1000                                      
storage-engine.write-block-size         :   131072                                    
storage-engine.write-threads            :   1                                         
tomb-raider-eligible-age                :   86400                                     
tomb-raider-period                      :   86400                                     
write-commit-level-override             :   master                                    
{ssp}-query-hist-track-back             :   300                                       
{ssp}-query-hist-track-slice            :   10                                        
{ssp}-query-hist-track-thresholds       :   1,8,64                                    
{ssp}-read-hist-track-back              :   300                                       
{ssp}-read-hist-track-slice             :   10                                        
{ssp}-read-hist-track-thresholds        :   1,8,64                                    
{ssp}-udf-hist-track-back               :   300                                       
{ssp}-udf-hist-track-slice              :   10                                        
{ssp}-udf-hist-track-thresholds         :   1,8,64                                    
{ssp}-write-hist-track-back             :   300                                       
{ssp}-write-hist-track-slice            :   10                                        
{ssp}-write-hist-track-thresholds       :   1,8,64                                    

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                                    :   node-1:3000               
allow-nonxdr-writes                     :   true                                      
allow-xdr-writes                        :   true                                      
cold-start-evict-ttl                    :   4294967295                                
conflict-resolution-policy              :   generation                                
data-in-index                           :   false                                     
default-ttl                             :   0                                         
disallow-null-setname                   :   false                                     
enable-benchmarks-batch-sub             :   false                                     
enable-benchmarks-read                  :   false                                     
enable-benchmarks-udf                   :   false                                     
enable-benchmarks-udf-sub               :   false                                     
enable-benchmarks-write                 :   false                                     
enable-hist-proxy                       :   false                                     
enable-xdr                              :   false                                     
evict-hist-buckets                      :   10000                                     
evict-tenths-pct                        :   5                                         
geo2dsphere-within.earth-radius-meters  :   6371000                                   
geo2dsphere-within.level-mod            :   1                                         
geo2dsphere-within.max-cells            :   12                                        
geo2dsphere-within.max-level            :   30                                        
geo2dsphere-within.min-level            :   1                                         
geo2dsphere-within.strict               :   true                                      
high-water-disk-pct                     :   85                                        
high-water-memory-pct                   :   97                                        
ldt-enabled                             :   false                                     
ldt-gc-rate                             :   0                                         
ldt-page-size                           :   8192                                      
max-ttl                                 :   315360000                                 
memory-size                             :   214748364800                              
migrate-order                           :   4                                         
migrate-sleep                           :   10000                                     
ns-forward-xdr-writes                   :   false                                     
read-consistency-level-override         :   one                                       
repl-factor                             :   2                                         
sets-enable-xdr                         :   true                                      
sindex.data-max-memory                  :   ULONG_MAX                                 
sindex.num-partitions                   :   32                                        
single-bin                              :   false                                     
stop-writes-pct                         :   99                                        
storage-engine                          :   device                                    
storage-engine.cold-start-empty         :   false                                     
storage-engine.data-in-memory           :   false                                     
storage-engine.defrag-lwm-pct           :   90                                        
storage-engine.defrag-queue-min         :   100                                       
storage-engine.defrag-sleep             :   10000                                     
storage-engine.defrag-startup-minimum   :   1                                         
storage-engine.device                   :   /dev/sda8,/dev/sdb8,/dev/sdc8,/dev/sdd8   c8,/dev/sdd8
storage-engine.disable-odirect          :   false                                     
storage-engine.enable-benchmarks-storage:   false                                     
storage-engine.enable-osync             :   false                                     
storage-engine.filesize                 :   17179869184                               
storage-engine.flush-max-ms             :   1000                                      
storage-engine.fsync-max-sec            :   0                                         
storage-engine.max-write-cache          :   134217728                                 
storage-engine.min-avail-pct            :   1                                         
storage-engine.post-write-queue         :   2048                                      
storage-engine.scheduler-mode           :   noop                                      
storage-engine.tomb-raider-sleep        :   10000                                     
storage-engine.write-block-size         :   131072                                    
storage-engine.write-threads            :   1                                         
tomb-raider-eligible-age                :   86400                                     
tomb-raider-period                      :   86400                                     
write-commit-level-override             :   master                                    
{enrichment}-query-hist-track-back      :   300                                       
{enrichment}-query-hist-track-slice     :   10                                        
{enrichment}-query-hist-track-thresholds:   1,8,64                                    
{enrichment}-read-hist-track-back       :   300                                       
{enrichment}-read-hist-track-slice      :   10                                        
{enrichment}-read-hist-track-thresholds :   1,8,64                                    
{enrichment}-udf-hist-track-back        :   300                                       
{enrichment}-udf-hist-track-slice       :   10                                        
{enrichment}-udf-hist-track-thresholds  :   1,8,64                                    
{enrichment}-write-hist-track-back      :   300                                       
{enrichment}-write-hist-track-slice     :   10                                        
{enrichment}-write-hist-track-thresholds:   1,8,64                                    

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                                    :   node-1:3000               
allow-nonxdr-writes                     :   true                                      
allow-xdr-writes                        :   true                                      
cold-start-evict-ttl                    :   4294967295                                
conflict-resolution-policy              :   generation                                
data-in-index                           :   false                                     
default-ttl                             :   0                                         
disallow-null-setname                   :   false                                     
enable-benchmarks-batch-sub             :   false                                     
enable-benchmarks-read                  :   false                                     
enable-benchmarks-udf                   :   false                                     
enable-benchmarks-udf-sub               :   false                                     
enable-benchmarks-write                 :   false                                     
enable-hist-proxy                       :   false                                     
enable-xdr                              :   false                                     
evict-hist-buckets                      :   10000                                     
evict-tenths-pct                        :   5                                         
geo2dsphere-within.earth-radius-meters  :   6371000                                   
geo2dsphere-within.level-mod            :   1                                         
geo2dsphere-within.max-cells            :   12                                        
geo2dsphere-within.max-level            :   30                                        
geo2dsphere-within.min-level            :   1                                         
geo2dsphere-within.strict               :   true                                      
high-water-disk-pct                     :   99                                        
high-water-memory-pct                   :   99                                        
ldt-enabled                             :   false                                     
ldt-gc-rate                             :   0                                         
ldt-page-size                           :   8192                                      
max-ttl                                 :   315360000                                 
memory-size                             :   12884901888                               
migrate-order                           :   1                                         
migrate-sleep                           :   1                                         
ns-forward-xdr-writes                   :   false                                     
read-consistency-level-override         :   one                                       
repl-factor                             :   2                                         
sets-enable-xdr                         :   true                                      
sindex.data-max-memory                  :   ULONG_MAX                                 
sindex.num-partitions                   :   32                                        
single-bin                              :   false                                     
stop-writes-pct                         :   99                                        
storage-engine                          :   device                                    
storage-engine.cold-start-empty         :   false                                     
storage-engine.data-in-memory           :   true                                      
storage-engine.defrag-lwm-pct           :   50                                        
storage-engine.defrag-queue-min         :   0                                         
storage-engine.defrag-sleep             :   1000                                      
storage-engine.defrag-startup-minimum   :   10                                        
storage-engine.device                   :   /dev/sda6,/dev/sdb6,/dev/sdc6,/dev/sdd6   c6,/dev/sdd6
storage-engine.disable-odirect          :   false                                     
storage-engine.enable-benchmarks-storage:   false                                     
storage-engine.enable-osync             :   false                                     
storage-engine.filesize                 :   17179869184                               
storage-engine.flush-max-ms             :   1000                                      
storage-engine.fsync-max-sec            :   0                                         
storage-engine.max-write-cache          :   134217728                                 
storage-engine.min-avail-pct            :   1                                         
storage-engine.post-write-queue         :   0                                         
storage-engine.scheduler-mode           :   noop                                      
storage-engine.tomb-raider-sleep        :   1000                                      
storage-engine.write-block-size         :   131072                                    
storage-engine.write-threads            :   1                                         
tomb-raider-eligible-age                :   86400                                     
tomb-raider-period                      :   86400                                     
write-commit-level-override             :   master                                    
{dsp}-query-hist-track-back             :   300                                       
{dsp}-query-hist-track-slice            :   10                                        
{dsp}-query-hist-track-thresholds       :   1,8,64                                    
{dsp}-read-hist-track-back              :   300                                       
{dsp}-read-hist-track-slice             :   10                                        
{dsp}-read-hist-track-thresholds        :   1,8,64                                    
{dsp}-udf-hist-track-back               :   300                                       
{dsp}-udf-hist-track-slice              :   10                                        
{dsp}-udf-hist-track-thresholds         :   1,8,64                                    
{dsp}-write-hist-track-back             :   300                                       
{dsp}-write-hist-track-slice            :   10                                        
{dsp}-write-hist-track-thresholds       :   1,8,64
9 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NODE                                    :   node-1:3000               
allow-nonxdr-writes                     :   true                                      
allow-xdr-writes                        :   true                                      
cold-start-evict-ttl                    :   4294967295                                
conflict-resolution-policy              :   generation                                
data-in-index                           :   false                                     
default-ttl                             :   0                                         
disallow-null-setname                   :   false                                     
enable-benchmarks-batch-sub             :   false                                     
enable-benchmarks-read                  :   false                                     
enable-benchmarks-udf                   :   false                                     
enable-benchmarks-udf-sub               :   false                                     
enable-benchmarks-write                 :   false                                     
enable-hist-proxy                       :   false                                     
enable-xdr                              :   false                                     
evict-hist-buckets                      :   10000                                     
evict-tenths-pct                        :   5                                         
geo2dsphere-within.earth-radius-meters  :   6371000                                   
geo2dsphere-within.level-mod            :   1                                         
geo2dsphere-within.max-cells            :   12                                        
geo2dsphere-within.max-level            :   30                                        
geo2dsphere-within.min-level            :   1                                         
geo2dsphere-within.strict               :   true                                      
high-water-disk-pct                     :   80                                        
high-water-memory-pct                   :   99                                        
ldt-enabled                             :   false                                     
ldt-gc-rate                             :   0                                         
ldt-page-size                           :   8192                                      
max-ttl                                 :   315360000                                 
memory-size                             :   9663676416                                
migrate-order                           :   3                                         
migrate-sleep                           :   1                                         
ns-forward-xdr-writes                   :   false                                     
read-consistency-level-override         :   one                                       
repl-factor                             :   2                                         
sets-enable-xdr                         :   true                                      
sindex.data-max-memory                  :   ULONG_MAX                                 
sindex.num-partitions                   :   32                                        
single-bin                              :   false                                     
stop-writes-pct                         :   80                                        
storage-engine                          :   device                                    
storage-engine.cold-start-empty         :   false                                     
storage-engine.data-in-memory           :   true                                      
storage-engine.defrag-lwm-pct           :   50                                        
storage-engine.defrag-queue-min         :   0                                         
storage-engine.defrag-sleep             :   1000                                      
storage-engine.defrag-startup-minimum   :   10                                        
storage-engine.device                   :   /dev/sda5,/dev/sdb5,/dev/sdc5,/dev/sdd5   c5,/dev/sdd5
storage-engine.disable-odirect          :   false                                     
storage-engine.enable-benchmarks-storage:   false                                     
storage-engine.enable-osync             :   false                                     
storage-engine.filesize                 :   17179869184                               
storage-engine.flush-max-ms             :   1000                                      
storage-engine.fsync-max-sec            :   0                                         
storage-engine.max-write-cache          :   134217728                                 
storage-engine.min-avail-pct            :   1                                         
storage-engine.post-write-queue         :   0                                         
storage-engine.scheduler-mode           :   noop                                      
storage-engine.tomb-raider-sleep        :   1000                                      
storage-engine.write-block-size         :   131072                                    
storage-engine.write-threads            :   1                                         
tomb-raider-eligible-age                :   86400                                     
tomb-raider-period                      :   86400                                     
write-commit-level-override             :   master                                    
{xdr}-query-hist-track-back             :   300                                       
{xdr}-query-hist-track-slice            :   10                                        
{xdr}-query-hist-track-thresholds       :   1,8,64                                    
{xdr}-read-hist-track-back              :   300                                       
{xdr}-read-hist-track-slice             :   10                                        
{xdr}-read-hist-track-thresholds        :   1,8,64                                    
{xdr}-udf-hist-track-back               :   300                                       
{xdr}-udf-hist-track-slice              :   10                                        
{xdr}-udf-hist-track-thresholds         :   1,8,64                                    
{xdr}-write-hist-track-back             :   300                                       
{xdr}-write-hist-track-slice            :   10                                        
{xdr}-write-hist-track-thresholds       :   1,8,64
10 
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:169) NODE-ID bb96f1800f69734 CLUSTER-SIZE 10
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:238)    system-memory: free-kbytes 113194156 free-pct 42
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:252)    in-progress: tsvc-q 186 info-q 0 nsup-delete-q 10890 rw-hash 58 proxy-hash 0 tree-gc-q 0
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:274)    fds: proto (11401,221035311,221023910) heartbeat (0,0,0) fabric (142,142,0)
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:283)    heartbeat-received: self 130796 foreign 1126577
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:337) {xdr} objects: all 15185688 master 7743014 prole 7442674
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:398) {xdr} migrations: complete
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:417) {xdr} memory-usage: total-bytes 1746625299 index-bytes 971884032 sindex-bytes 0 data-bytes 774741267 used-pct 18.07
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:447) {xdr} device-usage: used-bytes 3599555328 avail-pct 44
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:540) {xdr} client: tsvc (0,2) proxy (131,0,0) read (0,0,0,0) write (11690716,26693,113) delete (0,0,0,0) udf (0,0,0) lang (0,0,0,0)
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:596) {xdr} scan: basic (71,49,0) aggr (0,0,0) udf-bg (0,0,0)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:145) histogram dump: {xdr}-write (11717409 total) msec
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (00: 0007515314) (01: 0000708377) (02: 0000936856) (03: 0001143271)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (04: 0000785963) (05: 0000359116) (06: 0000221017) (07: 0000032102)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (08: 0000012171) (09: 0000002459) (10: 0000000758) (11: 0000000005)
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:337) {dsp} objects: all 4262367 master 2258624 prole 2003743
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:398) {dsp} migrations: complete
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:417) {dsp} memory-usage: total-bytes 8277084095 index-bytes 272791488 sindex-bytes 0 data-bytes 8004292607 used-pct 64.24
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:447) {dsp} device-usage: used-bytes 8660156928 avail-pct 81
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:540) {dsp} client: tsvc (0,94) proxy (460,0,9) read (45276,0,2,68999) write (0,0,0) delete (0,0,0,0) udf (148043,0,101) lang (35,148008,0,0)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:145) histogram dump: {dsp}-read (114275 total) msec
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (00: 0000105811) (01: 0000002033) (02: 0000001826) (03: 0000002059)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (04: 0000001299) (05: 0000000704) (06: 0000000284) (07: 0000000230)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:171)  (08: 0000000023) (09: 0000000003) (10: 0000000003)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:145) histogram dump: {dsp}-udf (148043 total) msec
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (00: 0000102263) (01: 0000008168) (02: 0000009606) (03: 0000011652)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (04: 0000008007) (05: 0000005283) (06: 0000002568) (07: 0000000419)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:171)  (08: 0000000067) (09: 0000000007) (10: 0000000003)
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:337) {ssp} objects: all 397301 master 153586 prole 243715
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:398) {ssp} migrations: complete
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:417) {ssp} memory-usage: total-bytes 40525050 index-bytes 25427264 sindex-bytes 0 data-bytes 15097786 used-pct 0.13
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:447) {ssp} device-usage: used-bytes 50854528 avail-pct 99
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:540) {ssp} client: tsvc (0,174540) proxy (1116,0,322) read (7999231,0,152994,22532605) write (404305,0,0) delete (0,0,0,0) udf (0,0,0) lang (0,0,0,0)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:145) histogram dump: {ssp}-read (30531836 total) msec
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (00: 0019256523) (01: 0001462336) (02: 0002246050) (03: 0003174838)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (04: 0002404860) (05: 0001243708) (06: 0000682770) (07: 0000056251)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:171)  (08: 0000003523) (09: 0000000810) (10: 0000000167)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:145) histogram dump: {ssp}-write (404305 total) msec
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (00: 0000182444) (01: 0000033507) (02: 0000047322) (03: 0000058415)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (04: 0000043264) (05: 0000022658) (06: 0000013883) (07: 0000001968)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (08: 0000000676) (09: 0000000136) (10: 0000000031) (11: 0000000001)
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:337) {enrichment} objects: all 2120215778 master 1028219401 prole 0
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:395) {enrichment} migrations: remaining (1188,974) active (1,0) complete-pct 59.28
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:426) {enrichment} memory-usage: total-bytes 135693809792 index-bytes 135693809792 sindex-bytes 0 used-pct 63.19
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:465) {enrichment} device-usage: used-bytes 1242985447552 avail-pct 4 cache-read-pct 51.15
Oct 19 2016 15:49:22 GMT: INFO (info): (ticker.c:540) {enrichment} client: tsvc (0,45897020) proxy (2645,0,425) read (105557816,0,642392,23025181) write (11546007,1832057,4538) delete (0,0,0,0) udf (0,0,0) lang (0,0,0,0)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:145) histogram dump: {enrichment}-read (128582997 total) msec
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (00: 0018687558) (01: 0003504966) (02: 0006088881) (03: 0011932086)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (04: 0019559821) (05: 0044275197) (06: 0022823658) (07: 0001682314)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (08: 0000026054) (09: 0000001826) (10: 0000000602) (11: 0000000029)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:171)  (12: 0000000005)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:145) histogram dump: {enrichment}-write (13378065 total) msec
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (00: 0000527450) (01: 0000485275) (02: 0000855242) (03: 0001597655)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (04: 0002047326) (05: 0002521214) (06: 0003958400) (07: 0000541487)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:162)  (08: 0000283904) (09: 0000289492) (10: 0000260441) (11: 0000008938)
Oct 19 2016 15:49:22 GMT: INFO (info): (hist.c:171)  (12: 0000001188) (13: 0000000053)
11

After some data analytic and tests in our aplication we find that 1/10 of all requests go to one partition. When we stop node partition migrate to another node.

Afrer aplication logic changed problem was solved.

We apologize this is our application problem, not Aerospike.

Thanks