[Resolved] Cause of AerospikeError: -2 & message "invalid size for buffer:"? [Bug]

Client Libraries Go Client
,
1

Occasionally I’ll see this error in our production logs, then with a message like

Invalid size for buffer: 11008865

It seems to happen when I call

 client.GetLargeList(defaultWritePolicy, key(ID), listName, "")

Any idea what causes this or what I can do about it? As it is I end up in a panic which percolates up to the user.

2

This looks like a subtle bug in the connection pooling mechanism.

I’ve been trying to reproduce this with little luck. Do you think you can provide a snippet of code that reproduces the issue?

3

There is not much more to it, it goes like this:

var entries []interface{}
list := client.GetLargeList(defaultWritePolicy, key(ID), listName, "")
entries, err = list.Scan()

Where that is the error returned by Scan.

There are no other scans in our code, nothing that needs to be closed that I’m aware of. We have one client for the whole lifetime of the application.

We’ve had an issue with some corrupted records, but this error has been happening long before we noticed those.

4

whoa, corrupted records? Do you think the client has had something to do with it?

5

Your colleagues didn’t seem to think so. Those started after a version upgrade migration, some bug they said. This error has been happening for a lot longer. I wish I could help more, anything I can do?

6

No worries. Will let you know as soon as I have a question or a new release.

7

I still cannot reproduce this issue. Here’s the code I’m trying:

package main

import (
    "flag"
    "fmt"
    "log"
    "math/rand"
    "runtime"
    "sync"

    as "github.com/aerospike/aerospike-client-go"
    asl "github.com/aerospike/aerospike-client-go/logger"
)

var host = flag.String("h", "127.0.0.1", "host")
var port = flag.Int("p", 3000, "port")
var user = flag.String("U", "", "User.")
var password = flag.String("P", "", "Password.")
var clientPolicy *as.ClientPolicy

func main() {
    runtime.GOMAXPROCS(runtime.NumCPU())

    asl.Logger.SetLevel(asl.DEBUG)

    flag.Parse()
    clientPolicy = as.NewClientPolicy()
    if *user != "" {
        clientPolicy.User = *user
        clientPolicy.Password = *password
    }

    client, err := as.NewClientWithPolicyAndHost(clientPolicy, as.NewHost(*host, *port))
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    var wg sync.WaitGroup

    wg.Add(1)

    fmt.Println("Populating the LLists...")
    l := make([]interface{}, 1000)
    for j := 0; j < 1000; j++ {
        l[j] = j
    }
    for i := 0; i < 1000; i++ {
        fmt.Println(i)
        key, _ := as.NewKey("test", "test", i)
        listPut := client.GetLargeList(nil, key, "listName", "")
        err = listPut.Destroy()
        if err != nil {
            panic(err)
        }

        err = listPut.Add(l...)
        if err != nil {
            panic(err)
        }
    }

    fmt.Println("Scanning the LLists...")
    for i := 0; i < 32; i++ {
        go func() {
            defer wg.Done()

            for {
                key, _ := as.NewKey("test", "test", rand.Intn(1000))
                list := client.GetLargeList(nil, key, "listName", "")
                _, err := list.Scan()

                // fmt.Println(e)

                if err != nil {
                    fmt.Println(err)
                }
            }
        }()
    }

    fmt.Println("Scanning the Records...")
    for i := 0; i < 32; i++ {
        go func() {
            defer wg.Done()

            for {
                rs, err := client.ScanAll(nil, "test", "test")

                if err != nil {
                    fmt.Println(err)
                }
                for res := range rs.Results() {
                    if res.Err != nil {
                        fmt.Println(err)
                    }
                    // fmt.Println(res.Record)
                }

            }
        }()
    }

    fmt.Println("PUT/GET the Records...")
    for i := 0; i < 32; i++ {
        go func() {
            defer wg.Done()

            for {
                key, _ := as.NewKey("test", "test", rand.Intn(1000))
                err := client.Put(nil, key, as.BinMap{"a": 1, "b": 2})
                if err != nil {
                    fmt.Println(err)
                }

                _, err = client.Get(nil, key)
                if err != nil {
                    fmt.Println(err)
                }

            }
        }()
    }

    wg.Wait()
    fmt.Println("Success...")
}

It does LList Scan, Normal Scan on Set, Get and Put on parallel. Could you run it on your end to see if it prints out an error somewhere?

It would be nice to check if you are using the latest Go client as well ($ go get -u …)

8

It appears that this is not a bug, but a feature :slight_smile:

You may have way more than a few items in the list while the buffer allocator is trying to limit the size of buffer to protect the memory.

I’ll release a new client version today that will let you tweak this number to suit your use case.

9

Ah, that might make sense. So this happens when the Scan returns more records than you can fit in the buffer?

We’ve actually been trying to avoid just that, but hit another snag here: https://discuss.aerospike.com/t/bug-actual-limits-of-a-llist-aer-4300/1702/3

If you can confirm I’ll just wait for that fix. Adjusting the buffer might help, but in the end I should limit how much data want to get back.

10

The new client was released on Friday. You can now tweak the MaxBufferSize.

Please let me know in case you come by any other issue.

1 Like