ArrayIndexOutOfBoundsException in client.put

Client Libraries Java Client
1

Hi,

I am getting this error while inserting bins :

java.lang.ArrayIndexOutOfBoundsException: 70331 at com.aerospike.client.command.Buffer.stringToUtf8(Buffer.java:186) ~[aerospike-client-3.0.31.jar:na] at com.aerospike.client.Value$StringValue.write(Value.java:352) ~[aerospike-client-3.0.31.jar:na] at com.aerospike.client.command.Command.writeOperation(Command.java:524) ~[aerospike-client-3.0.31.jar:na] at com.aerospike.client.command.Command.setWrite(Command.java:83) ~[aerospike-client-3.0.31.jar:na] at com.aerospike.client.command.WriteCommand.writeBuffer(WriteCommand.java:49) ~[aerospike-client-3.0.31.jar:na] at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:47) ~[aerospike-client-3.0.31.jar:na] at com.aerospike.client.AerospikeClient.put(AerospikeClient.java:295) ~[aerospike-client-3.0.31.jar:na]

Am I reaching any limit here ?

Thanks

2

Hi Holmes,

Can you post the exact code you are using in the put(), and how many Bins you are writing?

There is a limit of 32K unique Bin names per namespace. How many have you written already?

Peter

3

Hi,

We have not reached that limit for sure. Its not related to unique binNames as we handled that case separately. Also I have tested that limit and the error message is not this.

This may have to to do with the amount of data being pushed in bins. I am inserting about 20 unique bins (all less than 13 characters) Here is the code snippet :

final Bin[] bins = new Bin[row.size()];//row is --> Map<String, Object> row
int i = 0;
for (Map.Entry<String, Object> entry : row.entrySet()) {
            String column = entry.getKey();
            bins[i++] = new Bin(column, (String) entry.getValue());
}
client.put(writePolicy, k, bins); // WritePolicy writePolicy = new WritePolicy();

While debugging I found that in this loop in Command class Line:83 (aerospike-client-3.0.31)

    for (Bin bin : bins) {
        writeOperation(bin, operation);
    }

its able to process few bins and then AIOBE is thrown.

Thanks.

4

More update :

There are a total of 29 bins here. In the same Command.setWrite method , there is first a estimation of buffer size. Here its estimate is 70331.

But on WriteCommand.writeOperation(Bin bin, Operation.Type operation) method , when it is updating the last bin (#29 in the above loop)

int valueLength = bin.value.write(dataBuffer, dataOffset + OPERATION_HEADER_SIZE + nameLength);

the value of dataOffset is 70320 and total value of dataOffset + OPERATION_HEADER_SIZE + nameLength = 70331.

So, next in Buffer.stringToUtf8(String s, byte[] buf, int offset) method, it tries to increment the buffer index to 70331+1 and fails.

for (int i = 0; i < length; i++) {
            int c = s.charAt(i);
            if (c < 0x80) {
                buf[offset++] = (byte) c; // <--- Error here
            }
            else if (c < 0x...

Thanks.

5

Hi Holmes,

In your code, where is i declared and initialized?

Peter

6

Hi @helipilot50,

Updated the code.
I saw the default size of buffer is 8192 defined in ThreadLocalData

7

Hi Holmes,

Your analysis is correct. The write process is:

  1. Calculate buffer size from command arguments.

  2. Find or allocate a suitable buffer.

  3. Write command to buffer.

If the buffer size is not calculated correctly and the size is greater than the default buffer size (8K), then you will get a buffer overflow. I suspect “Buffer.estimateSizeUtf8()” may be calculating a different size than is actually written, but it could be other estimations as well.

Is it possible to provide a small sample program that reproduces the problem?

Most strings will be calculated correctly, but I’m not sure about some multi-byte UTF8 characters.

8

We’re also getting the same error with client version 3.0.33.

client.put(asWritePolicy, new Key("main", "test1", keyValue), new Bin("data", data), new Bin("schema", "1"))

where keyValue is a 36 length string and data is a byte array of length 8208. Write policy has sendkey = true as well.

Update: After a little more testing. It seems that the issue doesn’t occur if we disable sendkey. For now, I’m adding the key to a bin, but this still seems like an issue that should get resolved.

9

Hi Ayush,

Thanks for the sample code. The “sendkey=true” was the crucial piece as that code had a bug in it. The fix will be in the next java client release.

-Brian

10

@ayush , thanks !
I too found out about sendkey feature causing this issue and have disabled it for now. Also, @helipilot50 suggested to use extra bin to store user key data in this post .

@Brian, thanks for looking into it. Is Java Client Library Release Note | Download | Aerospike the release as I can see a fix for user key send buffer byte size calculation.

11

Yes, 3.0.34 contains the sendKey fix. I recommend upgrading to the latest client, 3.0.35 to get other fixes.

1 Like