Aerospike using double memory

Rod_McCutcheon · April 1, 2020, 6:05am

I’ve inherited a project that uses Aerospike as a timeseries datastore. I’m a bit unsure about the configuration. The box is sitting at around 90% ram usage (of 128GB) even though the aerospike allocation is only about 56GB. That doesn’t seem right. Is it because replication-factor is set to 2, even though there is only one node in the cluster?

Initially Aerospike was running on a linux box with 64gb of ram and 64gb of disk space, but it stopped writing to one namespace (FIVE_MINUTES). So we doubled the ram and disk space to 128gb.

Full configuration:

$ cat /etc/aerospike/aerospike.conf
# Aerospike database configuration file for deployments using mesh heartbeats.
service {
    user root
    group root
    paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
    pidfile /var/run/aerospike/asd.pid
    service-threads 4
    transaction-queues 4
    transaction-threads-per-queue 4
    proto-fd-max 15000
}
logging {
    # Log file must be an absolute path.
    file /var/log/aerospike/aerospike.log {
	    context any warning
	    }
}
network {
        service {
                address any
                port 3000
        }
    heartbeat {
                mode mesh
                port 3002 # Heartbeat port for this node.
                # List one or more other nodes, one ip-address & port per line:
                mesh-seed-address-port << >>
#               mesh-seed-address-port << >>
#               mesh-seed-address-port << >>
#               mesh-seed-address-port << >>
#               mesh-seed-address-port << >>
                interval 250
                timeout 10
        }
        fabric {
                port 3001
        }
        info {
                port 3003
        }
}
namespace LIVE {
    replication-factor 2
    memory-size 4G
    default-ttl 2d
    # storage-engine memory
    storage-engine device {
    file /opt/aerospike/data/LIVE.dat
    filesize 4G
        data-in-memory true
    }
}
namespace FIVE_MINUTES {
    replication-factor 2
    memory-size 40G
    default-ttl 0
    #storage-engine memory
    storage-engine device {
        file /opt/aerospike/data/FIVE_MINUTES.dat
        filesize 40G
        data-in-memory true
    }
}
namespace HOURLY {
    replication-factor 2
    memory-size 8G
    default-ttl 0
    #storage-engine memory
    storage-engine device {
        file /opt/aerospike/data/HOURLY.dat
        filesize 8G
        data-in-memory true
    }
}
namespace DAILY {
    replication-factor 2
    memory-size 2G
    default-ttl 0
    # storage-engine memory
    storage-engine device {
    file /opt/aerospike/data/DAILY.dat
    filesize 2G
    data-in-memory true
    }
}
namespace WEEKLY {
    replication-factor 2
    memory-size 1G
    default-ttl 0
    # storage-engine memory
    storage-engine device {
        file /opt/aerospike/data/WEEKLY.dat
        filesize 1G
        data-in-memory true
    }
}
namespace MONTHLY {
    replication-factor 2
    memory-size 1G
    default-ttl 0
    # storage-engine memory
    storage-engine device {
        file /opt/aerospike/data/MONTHLY.dat
        filesize 1G
        data-in-memory true
    }
}

pgupta · April 1, 2020, 6:22am

1 - One box => replication factor will be 1 even if you specify 2.

2 - All your configs, are keeping data in file and copy in memory.

3 - It may have hit “stop-writes” on FIVE_MINUTES upon reaching 90% of 40G usage. Adding more RAM to the box won’t change that condition. You have to bump up the RAM allocated to the namespace.

4 - Do you have secondary indexes on these namespaces? Quite likely they are consuming the additional RAM on the box.

$ aql
aql> show indexes

Rod_McCutcheon · April 1, 2020, 6:25am

Ok thanks!

$ aql
Aerospike Query Client
Version 3.13.0.1
C Client Version 4.1.6
Copyright 2012-2016 Aerospike. All rights reserved.
aql> show indexes
+----------------+---------+-----------+----------+-------+----------------------+---------+-----------+
| ns             | bin     | indextype | set      | state | indexname            | path    | type      |
+----------------+---------+-----------+----------+-------+----------------------+---------+-----------+
| "LIVE"         | "start" | "NONE"    | 14050761 | "RW"  | "idx_14050761_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 23016569 | "RW"  | "idx_23016569_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1044247  | "RW"  | "idx_1044247_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 15462877 | "RW"  | "idx_15462877_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1272402  | "RW"  | "idx_1272402_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 7830561  | "RW"  | "idx_7830561_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1911     | "RW"  | "idx_1911_start"     | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 3240     | "RW"  | "idx_3240_start"     | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 10904797 | "RW"  | "idx_10904797_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 7830523  | "RW"  | "idx_7830523_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1320344  | "RW"  | "idx_1320344_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 8603747  | "RW"  | "idx_8603747_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14996355 | "RW"  | "idx_14996355_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539311 | "RW"  | "idx_14539311_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14019288 | "RW"  | "idx_14019288_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 25094800 | "RW"  | "idx_25094800_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539519 | "RW"  | "idx_14539519_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539574 | "RW"  | "idx_14539574_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539396 | "RW"  | "idx_14539396_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 10754445 | "RW"  | "idx_10754445_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14890570 | "RW"  | "idx_14890570_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14692458 | "RW"  | "idx_14692458_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 16059181 | "RW"  | "idx_16059181_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539258 | "RW"  | "idx_14539258_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539473 | "RW"  | "idx_14539473_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 26844073 | "RW"  | "idx_26844073_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 9447726  | "RW"  | "idx_9447726_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1182974  | "RW"  | "idx_1182974_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 16059870 | "RW"  | "idx_16059870_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 10501664 | "RW"  | "idx_10501664_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 27410111 | "RW"  | "idx_27410111_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539267 | "RW"  | "idx_14539267_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 20006042 | "RW"  | "idx_20006042_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 13371274 | "RW"  | "idx_13371274_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 13436238 | "RW"  | "idx_13436238_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1182925  | "RW"  | "idx_1182925_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 13645005 | "RW"  | "idx_13645005_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 13930635 | "RW"  | "idx_13930635_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1166423  | "RW"  | "idx_1166423_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 22392551 | "RW"  | "idx_22392551_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 20621007 | "RW"  | "idx_20621007_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14653646 | "RW"  | "idx_14653646_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539220 | "RW"  | "idx_14539220_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 8145847  | "RW"  | "idx_8145847_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 2        | "RW"  | "idx_2_start"        | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 24268199 | "RW"  | "idx_24268199_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1883     | "RW"  | "idx_1883_start"     | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 13175091 | "RW"  | "idx_13175091_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 10572080 | "RW"  | "idx_10572080_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 6842580  | "RW"  | "idx_6842580_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 19775925 | "RW"  | "idx_19775925_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1044196  | "RW"  | "idx_1044196_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 12791078 | "RW"  | "idx_12791078_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1984     | "RW"  | "idx_1984_start"     | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 18793767 | "RW"  | "idx_18793767_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 13503572 | "RW"  | "idx_13503572_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 42       | "RW"  | "idx_42_start"       | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 12681975 | "RW"  | "idx_12681975_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539557 | "RW"  | "idx_14539557_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 18584943 | "RW"  | "idx_18584943_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 1271922  | "RW"  | "idx_1271922_start"  | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14539323 | "RW"  | "idx_14539323_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14819159 | "RW"  | "idx_14819159_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 14179572 | "RW"  | "idx_14179572_start" | "start" | "NUMERIC" |
| "LIVE"         | "start" | "NONE"    | 10443697 | "RW"  | "idx_10443697_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 16059181 | "RW"  | "idx_16059181_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 13645005 | "RW"  | "idx_13645005_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 13175091 | "RW"  | "idx_13175091_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14653646 | "RW"  | "idx_14653646_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14050761 | "RW"  | "idx_14050761_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 16059870 | "RW"  | "idx_16059870_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 10754445 | "RW"  | "idx_10754445_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 13436238 | "RW"  | "idx_13436238_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539258 | "RW"  | "idx_14539258_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1911     | "RW"  | "idx_1911_start"     | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 26844073 | "RW"  | "idx_26844073_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 7830523  | "RW"  | "idx_7830523_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 10572080 | "RW"  | "idx_10572080_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 23016569 | "RW"  | "idx_23016569_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1984     | "RW"  | "idx_1984_start"     | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 8145847  | "RW"  | "idx_8145847_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 18793767 | "RW"  | "idx_18793767_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1182974  | "RW"  | "idx_1182974_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539557 | "RW"  | "idx_14539557_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1320344  | "RW"  | "idx_1320344_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539574 | "RW"  | "idx_14539574_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539267 | "RW"  | "idx_14539267_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 27410111 | "RW"  | "idx_27410111_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 42       | "RW"  | "idx_42_start"       | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 15462877 | "RW"  | "idx_15462877_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 3240     | "RW"  | "idx_3240_start"     | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539519 | "RW"  | "idx_14539519_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 6842580  | "RW"  | "idx_6842580_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 10443697 | "RW"  | "idx_10443697_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14819159 | "RW"  | "idx_14819159_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 8603747  | "RW"  | "idx_8603747_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 10904797 | "RW"  | "idx_10904797_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539396 | "RW"  | "idx_14539396_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539311 | "RW"  | "idx_14539311_start" | "start" | "NUMERIC" |
+----------------+---------+-----------+----------+-------+----------------------+---------+-----------+
+----------------+---------+-----------+----------+-------+----------------------+---------+-----------+
| ns             | bin     | indextype | set      | state | indexname            | path    | type      |
+----------------+---------+-----------+----------+-------+----------------------+---------+-----------+
| "FIVE_MINUTES" | "start" | "NONE"    | 9447726  | "RW"  | "idx_9447726_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1883     | "RW"  | "idx_1883_start"     | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 18584943 | "RW"  | "idx_18584943_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 20006042 | "RW"  | "idx_20006042_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14890570 | "RW"  | "idx_14890570_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539323 | "RW"  | "idx_14539323_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 22392551 | "RW"  | "idx_22392551_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 13930635 | "RW"  | "idx_13930635_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 2        | "RW"  | "idx_2_start"        | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 19775925 | "RW"  | "idx_19775925_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1044247  | "RW"  | "idx_1044247_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1271922  | "RW"  | "idx_1271922_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1182925  | "RW"  | "idx_1182925_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 25094800 | "RW"  | "idx_25094800_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 10501664 | "RW"  | "idx_10501664_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1166423  | "RW"  | "idx_1166423_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14019288 | "RW"  | "idx_14019288_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 24268199 | "RW"  | "idx_24268199_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14996355 | "RW"  | "idx_14996355_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1044196  | "RW"  | "idx_1044196_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 13371274 | "RW"  | "idx_13371274_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 13503572 | "RW"  | "idx_13503572_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539473 | "RW"  | "idx_14539473_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14539220 | "RW"  | "idx_14539220_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 12791078 | "RW"  | "idx_12791078_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 12681975 | "RW"  | "idx_12681975_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 1272402  | "RW"  | "idx_1272402_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 7830561  | "RW"  | "idx_7830561_start"  | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14692458 | "RW"  | "idx_14692458_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 14179572 | "RW"  | "idx_14179572_start" | "start" | "NUMERIC" |
| "FIVE_MINUTES" | "start" | "NONE"    | 20621007 | "RW"  | "idx_20621007_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539323 | "RW"  | "idx_14539323_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14819159 | "RW"  | "idx_14819159_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1911     | "RW"  | "idx_1911_start"     | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1044196  | "RW"  | "idx_1044196_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 16059181 | "RW"  | "idx_16059181_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 20621007 | "RW"  | "idx_20621007_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 10572080 | "RW"  | "idx_10572080_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 13645005 | "RW"  | "idx_13645005_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 25094800 | "RW"  | "idx_25094800_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 26844073 | "RW"  | "idx_26844073_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 2        | "RW"  | "idx_2_start"        | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1182974  | "RW"  | "idx_1182974_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539574 | "RW"  | "idx_14539574_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 20006042 | "RW"  | "idx_20006042_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 15462877 | "RW"  | "idx_15462877_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539557 | "RW"  | "idx_14539557_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 6842580  | "RW"  | "idx_6842580_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539267 | "RW"  | "idx_14539267_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14653646 | "RW"  | "idx_14653646_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 8603747  | "RW"  | "idx_8603747_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 24268199 | "RW"  | "idx_24268199_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 13175091 | "RW"  | "idx_13175091_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14019288 | "RW"  | "idx_14019288_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539519 | "RW"  | "idx_14539519_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 13371274 | "RW"  | "idx_13371274_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 10904797 | "RW"  | "idx_10904797_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14890570 | "RW"  | "idx_14890570_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 23016569 | "RW"  | "idx_23016569_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 16059870 | "RW"  | "idx_16059870_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 13930635 | "RW"  | "idx_13930635_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539311 | "RW"  | "idx_14539311_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 3240     | "RW"  | "idx_3240_start"     | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 10443697 | "RW"  | "idx_10443697_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1044247  | "RW"  | "idx_1044247_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 18793767 | "RW"  | "idx_18793767_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14996355 | "RW"  | "idx_14996355_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539473 | "RW"  | "idx_14539473_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 7830561  | "RW"  | "idx_7830561_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 18584943 | "RW"  | "idx_18584943_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1182925  | "RW"  | "idx_1182925_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 8145847  | "RW"  | "idx_8145847_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1883     | "RW"  | "idx_1883_start"     | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 12681975 | "RW"  | "idx_12681975_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1272402  | "RW"  | "idx_1272402_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 27410111 | "RW"  | "idx_27410111_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 9447726  | "RW"  | "idx_9447726_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1166423  | "RW"  | "idx_1166423_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539220 | "RW"  | "idx_14539220_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 10754445 | "RW"  | "idx_10754445_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1984     | "RW"  | "idx_1984_start"     | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 22392551 | "RW"  | "idx_22392551_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539258 | "RW"  | "idx_14539258_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 7830523  | "RW"  | "idx_7830523_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14539396 | "RW"  | "idx_14539396_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 12791078 | "RW"  | "idx_12791078_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14050761 | "RW"  | "idx_14050761_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14692458 | "RW"  | "idx_14692458_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 42       | "RW"  | "idx_42_start"       | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 13436238 | "RW"  | "idx_13436238_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 19775925 | "RW"  | "idx_19775925_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1320344  | "RW"  | "idx_1320344_start"  | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 10501664 | "RW"  | "idx_10501664_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 13503572 | "RW"  | "idx_13503572_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 14179572 | "RW"  | "idx_14179572_start" | "start" | "NUMERIC" |
| "HOURLY"       | "start" | "NONE"    | 1271922  | "RW"  | "idx_1271922_start"  | "start" | "NUMERIC" |
| "DAILY"        | "start" | "NONE"    | 1984     | "RW"  | "idx_1984_start"     | "start" | "NUMERIC" |
| "DAILY"        | "start" | "NONE"    | 14050761 | "RW"  | "idx_14050761_start" | "start" | "NUMERIC" |
| "DAILY"        | "start" | "NONE"    | 14539323 | "RW"  | "idx_14539323_start" | "start" | "NUMERIC" |
+----------------+---------+-----------+----------+-------+----------------------+---------+-----------+
+----------+---------+-----------+----------+-------+----------------------+---------+-----------+
| ns       | bin     | indextype | set      | state | indexname            | path    | type      |
+----------+---------+-----------+----------+-------+----------------------+---------+-----------+
| "DAILY"  | "start" | "NONE"    | 1166423  | "RW"  | "idx_1166423_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 23016569 | "RW"  | "idx_23016569_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 10443697 | "RW"  | "idx_10443697_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 27410111 | "RW"  | "idx_27410111_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 10754445 | "RW"  | "idx_10754445_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539519 | "RW"  | "idx_14539519_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 26844073 | "RW"  | "idx_26844073_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 6842580  | "RW"  | "idx_6842580_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 18584943 | "RW"  | "idx_18584943_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 8603747  | "RW"  | "idx_8603747_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 9447726  | "RW"  | "idx_9447726_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 25094800 | "RW"  | "idx_25094800_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14019288 | "RW"  | "idx_14019288_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 13175091 | "RW"  | "idx_13175091_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 10501664 | "RW"  | "idx_10501664_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539574 | "RW"  | "idx_14539574_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1911     | "RW"  | "idx_1911_start"     | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 7830561  | "RW"  | "idx_7830561_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1320344  | "RW"  | "idx_1320344_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539473 | "RW"  | "idx_14539473_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 10572080 | "RW"  | "idx_10572080_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 10904797 | "RW"  | "idx_10904797_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 42       | "RW"  | "idx_42_start"       | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539258 | "RW"  | "idx_14539258_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 2        | "RW"  | "idx_2_start"        | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 16059870 | "RW"  | "idx_16059870_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 19775925 | "RW"  | "idx_19775925_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14179572 | "RW"  | "idx_14179572_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14819159 | "RW"  | "idx_14819159_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14692458 | "RW"  | "idx_14692458_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 16059181 | "RW"  | "idx_16059181_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 13436238 | "RW"  | "idx_13436238_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 22392551 | "RW"  | "idx_22392551_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1182925  | "RW"  | "idx_1182925_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 24268199 | "RW"  | "idx_24268199_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1182974  | "RW"  | "idx_1182974_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 20006042 | "RW"  | "idx_20006042_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14996355 | "RW"  | "idx_14996355_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 13371274 | "RW"  | "idx_13371274_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 7830523  | "RW"  | "idx_7830523_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539267 | "RW"  | "idx_14539267_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14890570 | "RW"  | "idx_14890570_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539557 | "RW"  | "idx_14539557_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539311 | "RW"  | "idx_14539311_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1883     | "RW"  | "idx_1883_start"     | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 12681975 | "RW"  | "idx_12681975_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 8145847  | "RW"  | "idx_8145847_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 20621007 | "RW"  | "idx_20621007_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 15462877 | "RW"  | "idx_15462877_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 13930635 | "RW"  | "idx_13930635_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14653646 | "RW"  | "idx_14653646_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1271922  | "RW"  | "idx_1271922_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1044196  | "RW"  | "idx_1044196_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539220 | "RW"  | "idx_14539220_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 3240     | "RW"  | "idx_3240_start"     | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 12791078 | "RW"  | "idx_12791078_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 18793767 | "RW"  | "idx_18793767_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 14539396 | "RW"  | "idx_14539396_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1044247  | "RW"  | "idx_1044247_start"  | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 13503572 | "RW"  | "idx_13503572_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 13645005 | "RW"  | "idx_13645005_start" | "start" | "NUMERIC" |
| "DAILY"  | "start" | "NONE"    | 1272402  | "RW"  | "idx_1272402_start"  | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 13175091 | "RW"  | "idx_13175091_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 1182925  | "RW"  | "idx_1182925_start"  | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 14539311 | "RW"  | "idx_14539311_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 1182974  | "RW"  | "idx_1182974_start"  | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 13371274 | "RW"  | "idx_13371274_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 22392551 | "RW"  | "idx_22392551_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 25094800 | "RW"  | "idx_25094800_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 14179572 | "RW"  | "idx_14179572_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 14819159 | "RW"  | "idx_14819159_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 13436238 | "RW"  | "idx_13436238_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 15462877 | "RW"  | "idx_15462877_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 1044247  | "RW"  | "idx_1044247_start"  | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 20006042 | "RW"  | "idx_20006042_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 20621007 | "RW"  | "idx_20621007_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 19775925 | "RW"  | "idx_19775925_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 10501664 | "RW"  | "idx_10501664_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 3240     | "RW"  | "idx_3240_start"     | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 14019288 | "RW"  | "idx_14019288_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 18584943 | "RW"  | "idx_18584943_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 27410111 | "RW"  | "idx_27410111_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 8603747  | "RW"  | "idx_8603747_start"  | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 10904797 | "RW"  | "idx_10904797_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 16059870 | "RW"  | "idx_16059870_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 18793767 | "RW"  | "idx_18793767_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 14539267 | "RW"  | "idx_14539267_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 1044196  | "RW"  | "idx_1044196_start"  | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 26844073 | "RW"  | "idx_26844073_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 13645005 | "RW"  | "idx_13645005_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 1911     | "RW"  | "idx_1911_start"     | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 1883     | "RW"  | "idx_1883_start"     | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 42       | "RW"  | "idx_42_start"       | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 14890570 | "RW"  | "idx_14890570_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 13503572 | "RW"  | "idx_13503572_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 16059181 | "RW"  | "idx_16059181_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 2        | "RW"  | "idx_2_start"        | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 10443697 | "RW"  | "idx_10443697_start" | "start" | "NUMERIC" |
| "WEEKLY" | "start" | "NONE"    | 13930635 | "RW"  | "idx_13930635_start" | "start" | "NUMERIC" |
+----------+---------+-----------+----------+-------+----------------------+---------+-----------+
+-----------+---------+-----------+----------+-------+----------------------+---------+-----------+
| ns        | bin     | indextype | set      | state | indexname            | path    | type      |
+-----------+---------+-----------+----------+-------+----------------------+---------+-----------+
| "WEEKLY"  | "start" | "NONE"    | 1166423  | "RW"  | "idx_1166423_start"  | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 14050761 | "RW"  | "idx_14050761_start" | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 12791078 | "RW"  | "idx_12791078_start" | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 1984     | "RW"  | "idx_1984_start"     | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 12681975 | "RW"  | "idx_12681975_start" | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 10572080 | "RW"  | "idx_10572080_start" | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 14539323 | "RW"  | "idx_14539323_start" | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 1272402  | "RW"  | "idx_1272402_start"  | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 14692458 | "RW"  | "idx_14692458_start" | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 14539258 | "RW"  | "idx_14539258_start" | "start" | "NUMERIC" |
| "WEEKLY"  | "start" | "NONE"    | 6842580  | "RW"  | "idx_6842580_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 12791078 | "RW"  | "idx_12791078_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 8603747  | "RW"  | "idx_8603747_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 13645005 | "RW"  | "idx_13645005_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14539258 | "RW"  | "idx_14539258_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 3240     | "RW"  | "idx_3240_start"     | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 25094800 | "RW"  | "idx_25094800_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1272402  | "RW"  | "idx_1272402_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1166423  | "RW"  | "idx_1166423_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 13436238 | "RW"  | "idx_13436238_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14890570 | "RW"  | "idx_14890570_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1182925  | "RW"  | "idx_1182925_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 16059181 | "RW"  | "idx_16059181_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 13371274 | "RW"  | "idx_13371274_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14179572 | "RW"  | "idx_14179572_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 13930635 | "RW"  | "idx_13930635_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 10443697 | "RW"  | "idx_10443697_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1044247  | "RW"  | "idx_1044247_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 2        | "RW"  | "idx_2_start"        | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 26844073 | "RW"  | "idx_26844073_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14819159 | "RW"  | "idx_14819159_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 16059870 | "RW"  | "idx_16059870_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 24268199 | "RW"  | "idx_24268199_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 20006042 | "RW"  | "idx_20006042_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 27410111 | "RW"  | "idx_27410111_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 22392551 | "RW"  | "idx_22392551_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 20621007 | "RW"  | "idx_20621007_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 42       | "RW"  | "idx_42_start"       | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1984     | "RW"  | "idx_1984_start"     | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 15462877 | "RW"  | "idx_15462877_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 19775925 | "RW"  | "idx_19775925_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1911     | "RW"  | "idx_1911_start"     | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14539323 | "RW"  | "idx_14539323_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 10501664 | "RW"  | "idx_10501664_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14019288 | "RW"  | "idx_14019288_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 18584943 | "RW"  | "idx_18584943_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1044196  | "RW"  | "idx_1044196_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1182974  | "RW"  | "idx_1182974_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14996355 | "RW"  | "idx_14996355_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1271922  | "RW"  | "idx_1271922_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1883     | "RW"  | "idx_1883_start"     | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 23016569 | "RW"  | "idx_23016569_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 10904797 | "RW"  | "idx_10904797_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 10572080 | "RW"  | "idx_10572080_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 18793767 | "RW"  | "idx_18793767_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14050761 | "RW"  | "idx_14050761_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 6842580  | "RW"  | "idx_6842580_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 1320344  | "RW"  | "idx_1320344_start"  | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 13175091 | "RW"  | "idx_13175091_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 13503572 | "RW"  | "idx_13503572_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14539267 | "RW"  | "idx_14539267_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 12681975 | "RW"  | "idx_12681975_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 10754445 | "RW"  | "idx_10754445_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14539311 | "RW"  | "idx_14539311_start" | "start" | "NUMERIC" |
| "MONTHLY" | "start" | "NONE"    | 14692458 | "RW"  | "idx_14692458_start" | "start" | "NUMERIC" |
+-----------+---------+-----------+----------+-------+----------------------+---------+-----------+
362 rows in set (0.003 secs)
OK

pgupta · April 1, 2020, 6:33am

Wow … never seen anyone build that many secondary indexes. Thats what is hogging the RAM. You need to take a look at your data model. Why do you need all these SIs. Although you can have max 256 SIs per namespace but in your application you can only invoke or use one at a time. See https://www.aerospike.com/docs/operations/plan/capacity#capacity-planning-for-secondary-indexes - that should explain all the memory consumption. But you really need to get into the data model of your application. Also, you are limited to 1023 sets per namespace - and once you use a set name, it uses that quota forever. That is also not a typical use of a set. I have few short (~25 min) videos on youtube - which may be helpful to get started. 01, 02, 03 … here are links to 01 and 02. 02 - Intro To Aerospike - YouTube is intro, and 01 - Relational To NoSQL Modeling - YouTube thats 01 - relational to nosql, and 03: 03 - Hands On With Aerospike - YouTube handson

pgupta · April 1, 2020, 6:40am

BTW, you can drop any SIs (using aql) that you are not using which should free up the RAM.

Rod_McCutcheon · April 6, 2020, 4:22am

Thanks for the info/links!

Our use case is an IoT solution for buildings capturing sensor data from a large number of sensors per building. The incoming data is read from MQTT and stored in Aerospike and is periodically aggregated by a batch process. We provide a web portal providing analytics/insights into the sensor data at different resolutions (live, five minutes, hourly, daily, weekly, monthly), which uses a custom query engine which was designed to be fast, flexible and scalable. The system has been in development since around 2015.

Namespaces

Our namespaces are live, five minutes, hourly, daily, weekly, and monthly.

Sets

We then use sets as logical groupings for buildings. You stated the limit is 1023 for sets per namespace, so theoretically we can only store data for 1023 buildings. This isn’t an issue yet, but could be down the track. Can you give an example of a typical use case for sets?

Bins

start : Timestamp in epoch. This is indexed and hence queries can only run against this bin.
end : Timestamp in epoch.
zone : Timezone.
type : The type of sensor data we are storing
data : Binary data which contains the sensor data as written by an ingestion service.

Indexes

There is one index used, idx_<building_id>_start - bin (start) on every set (building) for each namespace (data type); querying can only be done by start time. We dynamically add secondary indexes when we put data into the Aerospike store using the Java client via the ingestion service.

User Defined Queries (UDF)

The querying happens on the primary index, start but the other parameters are still used to filter data returned by the queries.

Have we modelled our data correctly? What would the performance impact be if we removed the secondary indexes?

Rod_McCutcheon · May 15, 2020, 3:57am

I’ve set up some metrics with Prometheus/Grafana, and it looks like we only use ~3GB for indexes (also confirmed using asadm -e “info”. I’ve also changed our config to be replication-factor 1 (just in case) and data-in-memory false for all namespaces. What could be using all the memory/ram?

Topic		Replies	Views
Hybrid configurations for aerospike? setup	2	1913	December 11, 2015
Development Environment with Small Ram	2	1549	August 31, 2015
Aerospike in-memory DB uses much memory than expected	23	7279	July 11, 2022
Aerospike slow performance write/batch-read	3	3549	October 16, 2017
Setting Aerospike in memory configuration Configuration	6	4345	July 5, 2020