Cluster imbalance (asmonitor shows lots of evicted objects)

aws

#1

Hi,

It looks like one of our cluster nodes has a lot less active master objects than the other ones. I assume that because of that its load is also lower than other nodes. I noticed that asmonitor for it is showing a lot of evicted objects - more than for other nodes - is that unusual? Usually I’ve seen the cluster rebalance itself but this has been like this for at least several days - any ideas why?

Here’s part of my asmonitor output - you can see the node in question here – ip-10-45-54-41

ip/namespace                                Avail       Evicted        Master     Repl     Stop       Used   Used      Used   Used    hwm   hwm
                                              Pct       Objects       Objects   Factor   Writes       Disk   Disk       Mem    Mem   Disk   Mem
                                                .             .             .        .        .          .      %         .      %      .     .
ip-10-16-162-206.ec2.internal/biddingdb        71             0   218,586,680        2    false   275.32 G     19   25.87 G     48     50    60
ip-10-144-96-162.ec2.internal/biddingdb        71             0   201,132,649        2    false   272.33 G     19   25.45 G     51     50    60
ip-10-150-109-85.ec2.internal/biddingdb        72             0   233,792,860        2    false   262.55 G     18   24.79 G     46     50    60
ip-10-45-54-41.ec2.internal/biddingdb          75   197,061,407    94,386,764        2    false   236.60 G     16   17.19 G     58     50    60

Thanks!


#2

Looks like the 10.45.54.41 has less memory configure than the rest.

17.19 / 0.58 = 29.64 GB
25.45 / 0.51 = 49.90 GB

So it is reaching the high-water-memory-pct much sooner than the rest causing it to evict. When evicting only master objects are evicted and the deletes, the corresponding replica objects on other nodes will also be deleted.


#3

Makes sense - thanks!