Configure to send instantaneous values for the metrics defined as cumulative

Operations Monitoring
1

For example, the metric, client_read_success, is being sent in cumulative value as defined by aerospike. Can we configure it to send instantaneous value? Actually I’m calculating the delta between the two values on the receiver side and this seems not natural as I have to aggregate two consecutive event streams.

2

Nope, cannot configure how metric is calculated.

Are you using graphite or something else? These tools normally provide a ability to translate these into a derivative.

Our collectd plugin already takes care of this btw:

3

Yes we’re testing collectd plugin for now, and then graphite too. Can you kindly point to the feature addressing my question? I couldn’t find it in the git hub. Thanks!

4

The metric you mentioned, client_read_success, is defined as an “operations” type here.

Collectd defines the “operations” type here. Notice it is a “DERIVE” type. You may also find the documentation of collectd types useful.

5

Thanks and Interesting to know the definitions of data source. But still I cannot be convinced that the client_read_success indicates the rate. Because my benchmark run (using the aeroapike provided java-client) shows its value increases always with quite amount of values. If it’s rate, it should decrease sometimes especially as the memory usage goes up. For our case we’re setting the collectd interval to 1 sec.

Wondering it’s also related to carbon plugin configuration in collectd.conf (for the license issue, we don’t use collectd network or graphite plugin)

<Plugin "python">
    # carbon_writer.py is at path /opt/collectd-plugins/carbon_writer.py
    ModulePath "/opt/collectd-plugins/"
    Import "carbon_writer"
    <Module "carbon_writer">
        LineReceiverHost "...."
        LineReceiverPort 2003
        LineReceiverProtocol "udp"
        DifferentiateCountersOverTime true
        LowercaseMetricNames true
        TypesDB "/usr/share/collectd/types.db"
        TypesDB "/opt/collectd-plugins/aerospike_types.db"
    </Module>
</Plugin>
6

  1. What is the exact command used to initiate the benchmark?

  2. Is it possible that read_success is increasing because an increasing number of records have been written and can now be found?

  3. Are you also monitoring client_read_error, client_read_timeout, and client_read_not_found which are all defined as type “operations”?

7

For example, you could try this command:

./run_benchmarks -h 127.0.0.1 -p 3000 -n test -k 100000000 -b 1 -o B:1400 -w RU,50 -g 1000 -T 100000 -z 100 -maxRetries 0

I could see all client_write_* metrics also ever increasing as the benchmark writes.

When I don’t run benchmark, all the values are staying constant, not falling to zero (if they are rate).

Hope this helps.

8

If you are using the collectd plugin, you should be seeing “operations-client_write_*” metrics arriving, in which case these should be rates. I’ve double checked our clusters to verify collectd is reporting rates for these metrics.

9

But the doc says they are “cumulative” numbers. Metrics Reference | Aerospike Documentation Could you point out what I’m misunderstanding?

e.g.,

client_read_success [cumulative] Location: Namespace Introduced: 3.9 Number of successful client read transactions.

10

They are emitted from the server as “cumulative” stats. The collectd plugin defines them to be “operations” type stats for which collectd will compute the derivative.