AMC Enterprise 3.6.0 missing write latency

Tools AMC
1

Hi,

We got trouble with missing latency graph for writes as you can see here:

The dashboard shows the writes. Any information about this?

We are running AMC 3.6.0 and Aero 3.5.12 Both Enterprise

2

Could you let us know which browser you are using?

3

Firefox/Chrome both latest

4

Are they successful writes ? Only successful writes are reflected here. Please cross check with the latency histogram in logs or stat_write_success counter from the stats and see if its changing.

5

Yes there are about 600 successfull writes / second.

Im currently unsure how to cross check, i guess you mean somehow with asinfo?

Coud you please post the full command so i can cross check?

6

To get the write and read stats from command line you can run the following on a node:

asinfo -v stats -l|grep stat

or cluster wide stats

 asmonitor -e 'asinfo -v statistics'|tr ";" "\n"|egrep ':3000|stat_'
1 Like
7

Thanks lucien.

The issue seems to be that our client write policy commit level is set on COMMIT_MASTER. When changed to COMMIT_ALL, the write latency graph works. Is it expected behavior that there is no writes_master data available when using commit_master?

8

Thanks for the additional info. This is not expected behavior. Could you confirm the aerospike client (Java, Python, C?) and version you are using.

Both writes and reads successfully shows in in AMC 3.6.0 for the latest python client using from the examples script:

 #  config = { 'hosts': [ (options.host, options.port) ] }
    config = {
            'hosts':    [ (options.host, options.port) ],
             'policies': {'timeout': 1000, 'commit_level':aerospike.POLICY_COMMIT_LEVEL_MASTER}}
#            'policies': {'timeout': 1000, 'commit_level':aerospike.POLICY_COMMIT_LEVEL_ALL}}
    client = aerospike.client(config).connect()
9

We are using the Java Client version 3.1.2.

wPolicy = new WritePolicy();
wPolicy.expiration = 0;
wPolicy.timeout = 5000;
wPolicy.recordExistsAction = RecordExistsAction.REPLACE;
wPolicy.commitLevel = CommitLevel.COMMIT_MASTER;
10

Thanks for the info. We will try to replicate with version java client 3.1.2 and latest Chrome browser.

One more thing. How much RAM to you have on the client computer running the browser?

I see from your original post that AMC was set to 30 minutes. This would require more the 4GB of free memory on client running the browser.

Here is the info on browser and memory specs:

  • Mozilla Firefox (preferred) or Google Chrome
  • Depending on how you set your throughput monitoring window you will also need:
  • 2 GB free RAM if throughput view is up to 10 minutes
  • 4 GB free RAM if throughput view is greater than 10 minutes
11

The data is missing even when set to 1 minute. The browser client also has more than enough free memory (> 16 GB).

12

We were able to replicate the behavior you experienced with AMC and commit level COMMIT_MASTER with the same Java client using the java benchmark tool. A bug (AER-3816) has been opened internally to track this issue. Thanks for reporting this issue.

2 Likes