Asadm monitoring disk memory

garcia · August 6, 2019, 12:25pm

asadm -e info

Seed:        [('127.0.0.1', 3000, None)]
Config_file: /root/.aerospike/astools.conf, /etc/aerospike/astools.conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2019-08-05 15:28:35 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                           Node               Node                   Ip       Build   Cluster   Migrations        Cluster     Cluster         Principal   Client       Uptime
                              .                 Id                    .           .      Size            .            Key   Integrity                 .    Conns            .
aerospike1:3000   *BB9D4512311B36C   1.2.3.4:3000   C-4.3.0.7         2      0.000     930430C64D1D   True        BB9D4512311B36C     2432   3634:13:21
aerospike2:3000   BB9BC512311B36C    5.6.7.8:3000    C-4.3.0.7         2      0.000     930430C64D1D   True        BB9D4512311B36C     2434   2856:13:55
Number of rows: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Usage Information (2019-08-05 15:28:35 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace                              Node      Total     Expirations,Evictions     Stop       Disk    Disk     HWM   Avail%          Mem     Mem    HWM      Stop
        .                                 .    Records                         .   Writes       Used   Used%   Disk%        .         Used   Used%   Mem%   Writes%
puh                         aerospike1:3000    1.197 B   (6.214 B, 205.455 M)      false    2.117 TB   53      80      23       151.774 GB   80      96     98
puh                         aerospike2:3000    1.197 B   (405.285 M, 0.000)        false    2.117 TB   37      80      48       151.104 GB   48      90     90
puh                                         2.393 B   (6.619 B, 205.455 M)               4.234 TB                            302.878 GB
pus                         aerospike1:3000   13.365 K   (154.559 M, 103.768 M)    false         N/E   N/E     50      N/E        2.206 MB   1       90     98
pus                                         13.365 K   (154.559 M, 103.768 M)             0.000 B                               2.206 MB
pus2                        aerospike2:3000    0.000     (0.000,  0.000)           false         N/E   N/E     50      N/E        0.000 B    0       60     90
pus2                                         0.000     (0.000,  0.000)                    0.000 B                               0.000 B
Number of rows: 7

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2019-08-05 15:28:35 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace                              Node      Total     Repl                           Objects                   Tombstones             Pending   Rack
        .                                 .    Records   Factor        (Master,Prole,Non-Replica)   (Master,Prole,Non-Replica)            Migrates     ID
        .                                 .          .        .                                 .                            .             (tx,rx)      .
puh      aerospike1:3000    1.197 B   2        (499.580 M, 697.031 M, 0.000)     (0.000,  0.000,  0.000)      (0.000,  0.000)     0
puh      aerospike2:3000    1.197 B   2        (697.031 M, 499.580 M, 0.000)     (0.000,  0.000,  0.000)      (0.000,  0.000)     0
puh                                         2.393 B            (1.197 B, 1.197 B, 0.000)         (0.000,  0.000,  0.000)      (0.000,  0.000)
pus       aerospike1:3000   13.366 K   1        (13.366 K, 0.000,  0.000)         (0.000,  0.000,  0.000)      (0.000,  0.000)     0
pus                                         13.366 K            (13.366 K, 0.000,  0.000)         (0.000,  0.000,  0.000)      (0.000,  0.000)
pus2      aerospike2:3000    0.000     1        (0.000,  0.000,  0.000)           (0.000,  0.000,  0.000)      (0.000,  0.000)     0
pus2                                         0.000              (0.000,  0.000,  0.000)           (0.000,  0.000,  0.000)      (0.000,  0.000)
Number of rows: 7

How can I monitor Disk Used% and Mem USed%

if Disk Used% is greater than HWM DISK% alarm

if Mem USed% is greater than HWM Mem% alarm

kporter · August 7, 2019, 6:01pm

The asadm tool isn’t intended to be used for alerting, I suggest using a tool intended for alerting:

https://aerospike.com/docs/operations/monitor/index.html

garcia · August 9, 2019, 11:51pm

I have a zabbix monitoring system, I was just thinking about how to get Disk Used% and Mem Used%

Now I set up receiving parameters like this sudo asadm -e info | grep -w "$namespace" | grep $node | awk 'NR == 1 {print $ 12}'

I thought maybe there is a better option

kporter · August 10, 2019, 12:43am

These metrics are straight from the statistics command - the zabbix plugin can query them. See https://www.aerospike.com/docs/reference/metrics/#device_free_pct.

garcia · August 13, 2019, 7:53am

something the data somehow diverges

asadm -e info

puh      aerospike2.:3000    1.547 B   (1.198 B, 0.000)        false    2.760 TB   48      80      33       190.429 GB   60      90     9
pus      aerospike2.:3000   36.093 M   (0.000,  548.121 K)     false         N/E   N/E     50      N/E       19.205 GB   65      96     98

./aerospike_discovery.py -h aerospike2 -n puh | grep -A 1 -E "(device_free_pct|memory_free_pct)"

    "{#METRICNAME}":"memory_free_pct",
    "{#METRICVALUE}":"40"
    --
    "{#METRICNAME}":"device_free_pct",
    "{#METRICVALUE}":"52"

./aerospike_discovery.py -h aerospike2 -n pus | grep -A 1 -E "(device_free_pct|memory_free_pct)"

    "{#METRICNAME}":"memory_free_pct",
    "{#METRICVALUE}":"36"

through asadm - e info I see for puh for Disk Used% - 29, HWM Disk% - 90, i.e. free should be 90 - 29 = 61, and the aerospike_discovery.py script shows device_free_pct - 52, also with memory, for pus Mem Used% - 65, HWM Used% - 96, 96 - 65 = 26, and aerospike_discovery.py memory_free_pct - 36

Am I misunderstanding something?

kporter · August 13, 2019, 11:04pm

No, we wouldn’t compute as HWM% - USED%, it would be 100% - USED%.

These metrics aren’t computed on the client, they come from the server, for example, here is how device_free_pct is computed: aerospike-server/thr_info.c at master · aerospike/aerospike-server · GitHub.

That said, could you provide the full output from asadm - the snippet you provided is difficult to read without the header.

garcia · August 15, 2019, 2:56pm

here is the complete output of the command

asadm -e info
Seed:        [('127.0.0.1', 3000, None)]
Config_file: /root/.aerospike/astools.conf, /etc/aerospike/astools.conf
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Network Information (2019-08-15 14:52:12 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                           Node               Node                   Ip       Build   Cluster   Migrations        Cluster     Cluster         Principal   Client       Uptime
                              .                 Id                    .           .      Size            .            Key   Integrity                 .    Conns            .
aerospike1:3000   *BB9D4512311B36C   1.2.3.4:3000   C-4.5.3.5         2    725.000     1BA1A383CBD7   True        BB9D4512311B36C     2202   103:16:14
aerospike2:3000   BB9BC512311B36C    5.6.7.8:3000    C-4.3.0.7         2    725.000     1BA1A383CBD7   True        BB9D4512311B36C     2550   3095:37:33
Number of rows: 2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Usage Information (2019-08-15 14:52:12 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace                              Node      Total   Expirations,Evictions     Stop       Disk    Disk     HWM   Avail%          Mem     Mem    HWM      Stop
        .                                 .    Records                       .   Writes       Used   Used%   Disk%        .         Used   Used%   Mem%   Writes%
puh      aerospike1:3000    1.772 B   (24.336 M, 0.000)       false    3.193 TB   49      90      31       218.897 GB   69      96     90
puh      aerospike2:3000    1.891 B   (1.322 B, 0.000)        false    3.391 TB   58      80      17       236.536 GB   74      90     98
puh                                         3.663 B   (1.346 B, 0.000)                 6.584 TB                            455.432 GB
pus       aerospike1:3000    0.000     (0.000,  0.000)         false         N/E   N/E     50      N/E       36.000 KB   1       60     90
pus                                          0.000     (0.000,  0.000)                  0.000 B                              36.000 KB
pus2      aerospike2:3000   10.972 M   (0.000,  548.121 K)     false         N/E   N/E     50      N/E        7.449 GB   25      96     98
pus2                                        10.972 M   (0.000,  548.121 K)              0.000 B                               7.449 GB
Number of rows: 7

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Namespace Object Information (2019-08-15 14:52:12 UTC)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Namespace                              Node      Total     Repl                         Objects                   Tombstones                 Pending   Rack
        .                                 .    Records   Factor      (Master,Prole,Non-Replica)   (Master,Prole,Non-Replica)                Migrates     ID
        .                                 .          .        .                               .                            .                 (tx,rx)      .
puh      aerospike1:3000    1.772 B   2        (767.886 M, 1.004 B, 0.000)     (0.000,  0.000,  0.000)      (0.000,  725.000)       0
puh      aerospike2:3000    1.891 B   2        (1.116 B, 774.592 M, 0.000)     (0.000,  0.000,  0.000)      (725.000,  0.000)       0
puh                                         3.663 B            (1.884 B, 1.779 B, 0.000)       (0.000,  0.000,  0.000)      (725.000,  725.000)
pus       aerospike1:3000    0.000     1        (0.000,  0.000,  0.000)         (0.000,  0.000,  0.000)      (0.000,  0.000)         0
pus                                          0.000              (0.000,  0.000,  0.000)         (0.000,  0.000,  0.000)      (0.000,  0.000)
pus2      aerospike2:3000   10.972 M   1        (10.972 M, 0.000,  0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)         0
pus2                                        10.972 M            (10.972 M, 0.000,  0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)
Number of rows: 7

garcia · September 2, 2019, 8:19am

thanks for the help, made such checks

./aerospike_discovery.py -h $1 -n $2 | grep -A 1 "device_free_pct" | grep -v "#METRICNAME" | cut -d: -f2 | tr -d "\" "
./aerospike_discovery.py -h $1 -n $2 | grep -A 1 "memory_free_pct" | grep -v "#METRICNAME" | cut -d: -f2 | tr -d "\" "

just need to substitute your variables $1 - node, $2 - namespace

Topic		Replies	Views
Aerospike memory overflow	3	883	May 8, 2018
Communication problem bewteen nodes Operations setup	2	37	July 9, 2024
Cluster Visibility error asadmin	6	3194	May 2, 2018
Aerospike data expire	9	4941	September 8, 2021
Trying to create a test namespace : Errors are not verbose Configuration	5	341	April 21, 2023

Asadm monitoring disk memory

Related topics