Need help for "Skip the unpacker error" in Aerospike Go Client


#1

Hi Team.

I have been evaluating Aerospike for one of our data store needs. Before moving to enterprise edition, I wanted to see if this can really handle our usecases quickly. For this I have generated tons of data and checked for read and write throughput/latencies with the available class of hardware. Before describing the issue that we are hitting, I have done lot of googling around it and surprised that nobody had see this error before. Lets go for more details now,

  1. OS version:
    uname -a Linux temp-1042071 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u2~bpo8+1 (2017-01-05) x86_64 GNU/Linux

  2. Aerospike server version: aerospike-server-community-4.0.0.4-debian8.tgz

  3. Aerospike Go client version: ## November 29 2017: v1.31.0

  4. Disc Hardware type: SSD

  5. Version of Go: go version go1.10 linux/amd64

The row under contention: I have inserted a row with primary key: P2898 in two different sets, signal1 & signal2 with AQL I can access and see the contents for this key in both sets, even with digest of the row, I can access it,

**aql> select * from signals.signal1 where PK = 'P2898'**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**| value                                                                                                                                 |**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**| LIST('[2898.2899, 2898.789234565288, 2898.789234565288, 2898.789234565288, 2898.789234565232, 2898.789234565232, 2898.789234565232]') |**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**1 row in set (0.001 secs)**

**OK**

**aql> select * from signals.signal2 where PK = 'P2898'**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**| value                                                                                                                                 |**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**| LIST('[2898.2899, 2898.723238923457, 2898.723238923457, 2898.723238923457, 2898.723238923457, 2898.723238923456, 2898.723238923456]') |**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**1 row in set (0.001 secs)**

**OK**

**aql> select * from signals.signal2 where DIGEST='c59a06ff228fce104be523cdbec987ad557fcfc0'**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**| value                                                                                                                                 |**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**| LIST('[2898.2899, 2898.723238923457, 2898.723238923457, 2898.723238923457, 2898.723238923457, 2898.723238923456, 2898.723238923456]') |**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**1 row in set (0.001 secs)**

**OK**

**aql> select * from signals.signal2 where DIGEST='05a4f03e6e8ea5ec9a17df6ccd0c92fd39b122fc'**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**| value                                                                                                                                 |**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**| LIST('[2898.2899, 2898.789234565288, 2898.789234565288, 2898.789234565288, 2898.789234565232, 2898.789234565232, 2898.789234565232]') |**
**+---------------------------------------------------------------------------------------------------------------------------------------+**
**1 row in set (0.001 secs)**

Now the problem is that, (consistently reproducible) I am not able to get data using the Go client .Get() or BatchGet() method. I see the following error when I invoke the get command.

**Skip the unpacker error**

**/usr/local/go/src/runtime/panic.go:505 (0x42b1c8)**
**  gopanic: reflectcall(nil, unsafe.Pointer(d.fn), deferArgs(d), uint32(d.siz), uint32(d.siz))**

Just to ensure that this is indeed a client-side issue, I have written one get UDF which just returns the value for this record, and there I am able to see the data flowing in,

The UDF is simple get as follows: function getRecord(rec, binName, pk) if aerospike:exists(rec) then return rec[“value”] end return nil end

If I look closely on what data is transferred between systems, I see that go client is getting following bytes for this,

The problem is only for the 15th pair(index 14 in buffer array) which I am receiving in problematic record, as, 199 instead of 203 which is getting in for the good records. The switch-case code in client which gets executed because of this is,

for 199 . this returns non-nil error -

in unpacker.go:
case 0xc7: // Skip over type extension with 8 bit header and bytes
 		count := int(upckr.buffer[upckr.offset] & 0xff)
		upckr.offset += count + 1 + 1
		return nil, skipHeaderErr

for 203,

in unpacker.go:
    	case 0xcb:
    		val := Buffer.BytesToFloat64(upckr.buffer, upckr.offset)
    		upckr.offset += 8
    		return val, nil

I want to use the batch calls (because of inbuilt workgroup code and concurrency instead of building my own around the same). Any suggestions on why this must be happening and why I am getting results of 199 for type for this particular record+set combination.

One option in parallel I would try out is upgrading the go client version which is released in March. Your assistance is highly appreciated for this.


#2

Just looking at your AQL output - read with PK is from signal1 and signal2, different sets. But, with digests, you are showing both from signal2 set. I would think one would be from signal1 set. Can you clarify or can you post the AQL output for: (Though when reading with digest, you don’t have to give the set name - its ignored.)

aql> set output json
aql> select * from signals.signal1 where PK = 'P2898'     
aql> explain select * from signals.signal1 where PK = 'P2898'
aql> select * from signals.signal2 where PK = 'P2898'
aql> explain select * from signals.signal2 where PK = 'P2898'

#3

Thank you for very quick revert Mr Gupta. Here are required details -

First one: aql> select * from signals.signal1 where PK = ‘P2898’

    [
        [
            {
              "value": [
                2898.2899000000002,
                2898.7892345652881,
                2898.7892345652881,
                2898.7892345652881,
                2898.7892345652322,
                2898.7892345652322,
                2898.7892345652322
              ]
            }
        ],
        [
            {
              "Status": 0
            }
        ]
    ]

2nd one:

aql> explain select * from signals.signal1 where PK = 'P2898'

[
    [
        {
          "SET": "signal1",
          "DIGEST": "05A4F03E6E8EA5EC9A17DF6CCD0C92FD39B122FC",
          "NAMESPACE": "signals",
          "PARTITION": 1029,
          "STATUS": "",
          "UDF": "FALSE",
          "KEY_TYPE": "STRING",
          "POLICY_REPLICA": "AS_POLICY_REPLICA_MASTER",
          "NODE": "BB94555200A0102",
          "POLICY_KEY": "AS_POLICY_KEY_DEFAULT",
          "TIMEOUT": 1000
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

3rd one:

aql> select * from signals.signal2 where PK = 'P2898'

[
    [
        {
          "value": [
            2898.2899000000002,
            2898.7232389234568,
            2898.7232389234568,
            2898.7232389234568,
            2898.7232389234568,
            2898.7232389234564,
            2898.7232389234564
          ]
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

4th one:

aql> explain select * from signals.signal2 where PK = 'P2898'

[
    [
        {
          "SET": "signal2",
          "DIGEST": "C59A06FF228FCE104BE523CDBEC987AD557FCFC0",
          "NAMESPACE": "signals",
          "PARTITION": 2757,
          "STATUS": "",
          "UDF": "FALSE",
          "KEY_TYPE": "STRING",
          "POLICY_REPLICA": "AS_POLICY_REPLICA_MASTER",
          "NODE": "BB99072210A0102",
          "POLICY_KEY": "AS_POLICY_KEY_DEFAULT",
          "TIMEOUT": 1000
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

And for set thing, it doesn’t matter with the digest so didn’t change, so thought of intended behavior. Still just in case, pasting with the correct query:

aql> select * from signals.signal1 where DIGEST='05a4f03e6e8ea5ec9a17df6ccd0c92fd39b122fc'

[
    [
        {
          "value": [
            2898.2899000000002,
            2898.7892345652881,
            2898.7892345652881,
            2898.7892345652881,
            2898.7892345652322,
            2898.7892345652322,
            2898.7892345652322
          ]
        }
    ],
    [
        {
          "Status": 0
        }
    ]
]

#4

Suggest you change the topic title to: “Curious case … error in Go Client” - so it gets the right focus from the community.


#5

Title corrected, Mr Gupta.

Tried with latest unreleased go-client from trunk today. still seeing the same issue with the keys. Also this issue is seen with roughly 10% of the keys in 40Million test records that I generated for testing.


#6

The Go client dev here. I just noticed this thread. Thanks for the time you put in to help us find the issue.

I just tried to reproduce your issue and failed. Here is what I did:

  1. Inserted the record via AQL: INSERT INTO test.signal1 (PK, value) VALUES ('P2898', LIST('[2898.2899, 2898.789234565288, 2898.789234565288, 2898.789234565288, 2898.789234565232, 2898.789234565232, 2898.789234565232]'))
  2. Used the following code to retrieve it:
package main

import (
	"log"
	"math"
	"time"

	as "github.com/aerospike/aerospike-client-go"
)

const (
	namespace = "test"
	set       = "signal1"
	key       = "P2898"
)

func main() {

	policy := as.NewClientPolicy()
	policy.Timeout = 5 * time.Second
	client, err := as.NewClientWithPolicyAndHost(policy, as.NewHost("vmu1604", 3000))
	if err != nil {
		log.Panicf("failed to connect (%s)", err)
	}

	client.DefaultPolicy.MaxRetries = 10
	client.DefaultWritePolicy = as.NewWritePolicy(0, math.MaxUint32)

	aeroKey, err := as.NewKey(namespace, set, key)
	if err != nil {
		log.Panic(err)
	}

	record, err := client.Get(nil, aeroKey)
	if err != nil {
		log.Panicf("Get failed %s: %s", aeroKey, err)
	}

	if record == nil || record.Bins == nil {
		log.Printf("Get found nothing")
		return
	}

	log.Printf("Get got %d bins with gen %d (%s)", len(record.Bins), record.Generation, aeroKey)
}

Could you provide an AQL insert command with data that could reproduce the issue via the above code snippet?

Thanks, K.


#7

Thank you K for reply and code suggestions. much appreciated. In this particular case, I don’t have a direct reproducible case, but here is some more information around writing records is this, I have a UDF with following code,

function pushSignal(rec, value)
  local binName = "value"
  local l = rec[binName]
  if (l == nil) then
   l = list()
 end


  if #l >= 7 then
    -- lets make room for incoming guy by dropping the last element.
     list.trim(l, 7)
  end
  -- prepend the new incomer to the first position and store the list for get optimization.
  list.prepend(l, value)

  -- store back the list in bin.
  rec[binName] = l
  if aerospike:exists(rec) then
    aerospike:update(rec)
  else
    aerospike:create(rec)
  end

  -- Lets return success so that it can be returned to api caller if the writing was indeed successful.
  return l
end

At the write side, I have the following code segment,

var asKey *as.Key
		asKey, _ = as.NewKey(namespace, signal.Name, signals.idvalue)
		_, err = client.Execute(shared.WritePolicy, asKey, "hello", "pushSignal", as.FloatValue(signal.Value))

I have generated all the payloads to this code and sending it via jmeter calls. When I see the asadm output with info, I see all the keys are getting written as well, as shown below,

Namespace                                             Node       Total     Repl                           Objects                   Tombstones             Pending   Rack
        .                                                .     Records   Factor        (Master,Prole,Non-Replica)   (Master,Prole,Non-Replica)            Migrates     ID
        .                                                .           .        .                                 .                            .             (tx,rx)      .
signals    ip1:3000                                  50.294 M   2        (26.027 M, 24.267 M, 0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)     0
signals     ip2:3000                                51.845 M   2        (26.511 M, 25.335 M, 0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)     0
signals     ip3:3000                                  50.730 M   2        (25.484 M, 25.245 M, 0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)     0
signals     ip4:3000                                 48.914 M   2        (24.748 M, 24.166 M, 0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)     0
signals     ip5:3000                                  47.171 M   2        (23.292 M, 23.879 M, 0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)     0
signals     ip6:3000                                51.233 M   2        (24.475 M, 26.757 M, 0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)     0
signals     ip7:3000                                  49.461 M   2        (24.406 M, 25.055 M, 0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)     0
signals     ip8:3000    50.352 M   2        (25.056 M, 25.296 M, 0.000)       (0.000,  0.000,  0.000)      (0.000,  0.000)     0
signals                                                      400.000 M            (200.000 M, 200.000 M, 0.000)     (0.000,  0.000,  0.000)      (0.000,  0.000)

Now after this when I make get calls, randomly some of these keys(10% signals) return empty lists (please note that it is not <nil> record that I am getting, I am getting empty list)

The Get is always reproducible. If I do the write call again and check in AQL it shows updates reaching to server, but next GET will still be empty. Can you please tell me in what circumstances, the server returns 0xcb and 0xc7 as 15th byte in the result buffer, which is particularly different in my case(see the code lines in my first post here).

Apologies for big response but wanted to provide as much relevant information as possible.


#8

Strange, I still can’t reproduce your issue.

Those numbers are headers for msgpack protocol.

One signifies a float64, and the other signifies a map type.

You can read the spec here:

Is there a chance you could share backup of your test db with us, and the keys that do not work in Go, but with aql?

Thanks, K.


#9

A few more questions:

  1. Have you been experimenting with ordered lists?
  2. Have you previously or do you send any other data value other than float64 numbers to the UDF?

Thanks,


#10

Answers to both of the questions,

  1. No, for me these are simple lua lists. there is no natural order maintained for float64 numbers. I just wanted to keep track of number of insertions that have happened on these lists. hence from my lua code you will see, I always keep a maximum of 7 entries in list against “value” bin.

  2. That is not possible, because Go would have given me exception right there (I use as.Floatvalue for wrapping it). also the post is generated using the following go code, please generate few million ids if testing is required.

code:

    package main

    import (
    "fmt"
    "encoding/json"
    "bytes"
     "strconv"
     "strings"
    "sort"
    ) type signal struct {
    Name          string  `json:"name"`
    Value         float64 `json:"value"`
    Fact1  bool    `json:"fact1"`
    Fact2 bool    `json:"fact2"` }  type signalPacket struct {
        Id                   string   `json:"id"`
        Subgroupid                string   `json:"subgroupid"`
        Groupid              string   `json:"groupid"`
        Label                 string   `json:"label"`
        LabelPath             string   `json:"labelPath"`
        Groupquantile    float64  `json:"groupquantile"`
        LabelQuantile float64  `json:"labelQuantile"`
        Date                  string   `json:"date"`
        Signals               []signal `json:"signals"` }


    func main() {
      //bulkGetGenerator(80, 1000);
      //bulkGetAssertGenerator(80, 1000);

      bulkSetGenerator(10,10);
    }

    func bulkSetGenerator(lines int, packetSize int) {
      for i := 0 ; i < lines ; i++ {
        signalsList := []signalPacket{}
         for j := 0 ; j < packetSize ; j++ {

           counter := i * packetSize + j
           floatCounter := float64(counter)

           var buffer bytes.Buffer
           buffer.WriteString("P")
           buffer.WriteString(strconv.Itoa(counter))

           signals:= []signal{{Name: "signal1", Value: 0.7892345652323 + floatCounter, Fact1: false, Fact2: false}, {Name: "signal2", Value: floatCounter + 0.7232389234565, Fact1: false, Fact2: false}, {Name: "signal3", Value: floatCounter+0.78922343234565, Fact1: false, Fact2: false}, {Name: "signal4", Value: floatCounter + 0.78234329234565, Fact1: false, Fact2: false}}
           signalsList = append(signalsList, signalPacket{Id:buffer.String(), Subgroupid: "i"+strconv.Itoa(counter), Groupid:"v"+strconv.Itoa(counter), Label:"b"+strconv.Itoa(counter), LabelPath:"l1/l2/l"+strconv.Itoa(counter), LabelQuantile:50.23423, Groupquantile: 53.23423432, Date: "21-03-2018", Signals:signals});
         }
           bytes,_ := json.Marshal(&signalsList)
           fmt.Println(string(bytes))
      }
    }

In my go code, I just do a double iteration (first on json list of packets and second on signals - kind of hacky yeah, for the moment) and execute the udf that I sent in the previous post, because there is no direct Batch Set available similar to BatchGet. so my writes will be generic and type safe(same float64 applicable to all writes).

For your previous question, please suggest steps and method to create and share the DB from Aerospike, I can do that as well.

regards. Arun.


#11

I think we have a fix now.

Can you try the latest code in github/master to see if the issue is resolved?


#12

Thank you Mr Khosrow, I will give it a try with next opportunity and let you know the results.


#13

So I tried out the snapshot from head and the errors have now reduced to .10 % as opposed to 15% before. some of the keys where these errors were seen have also fixed. for .1% of errors, I still see the skip the unpacker error. I will do more deep study around that and come back on the thread.

Until then I request you not to close this thread as fixed.


#14

The best way you can help us fix the issue is to post the byte array received from the server (the same way you posted above in a screenshot).

Using that we can get a better understanding of what’s happening under the hood.

Thanks,


#15

Done Mr Khosrow. I will do that. Thank you.


#16

This is not as easily reproducible as we did the last time. I would have to deploy the debug code in our production environment instead of generating the debug in intellij. I will send updates on reproduction soon. Thank you for your patience and support so far.


#17

No problem at all. We’re here to help. Let us know when you have some results to share


#18

Hey Mr Khosrow,

I think I have some handle on this problem now, at least some more understanding. Here are the observations listed,

  1. This doesn’t happen with .Get() calls.
  2. It only happens with .BatchGet() calls.

I still see 0xc7 where skip the unpacker error comes, but it happily moves in most of the cases, the cases where it fails has a peculiar symptom (thats the only correlation I found between failed and not failed cases),

The byte sequence where it used to fail but started passing(please pad with all zeros for the remaining buffer bytes in the examples below):

0xc7 Buffer is:%!(EXTRA []uint8=[152 199 0 1 203 64 166 164 148 109 197 214 57 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 225 203 64 166 165 114 76 95 70 225 1 0 1 0 0 0 8 0 115 105 103 110 97 108 115 0 0 0 9 1 0 0 5 118 97 108 117 101 0 ........]) 

0xc7 Buffer is:%!(EXTRA []uint8=[152 199 0 1 203 64 166 164 148 109 197 214 57 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 225 203 64 166 165 114 76 95 70 225 1 0 1 0 0 0 8 0 115 105 103 110 97 108 115 0 0 0 9 1 0 0 5 118 97 108 117 101 0 0........])

0xc7 Buffer is:%!(EXTRA []uint8=[152 199 0 1 203 64 166 164 148 109 197 214 57 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 225 203 64 166 165 114 76 95 70 225 1 0 1 0 0 0 8 0 115 105 103 110 97 108 115 0 0 0 9 1 0 0 5 118 97 108 117 101 0 0 ....])

The place where it fails has many many more bytes now, here are few examples,

0xc7 Buffer is:%!(EXTRA []uint8=[152 199 0 1 203 65 34 137 187 114 76 95 71 203 65 34 137 187 114 76 95 71 203 65 34 137 187 114 76 95 71 203 65 34 137 187 114 76 95 71 203 65 34 137 187 114 76 95 71 203 65 34 137 187 114 76 95 71 203 65 34 137 187 114 76 95 71 1 0 1 0 0 0 8 0 115 105 103 110 97 108 115 0 0 0 9 1 0 0 5 118 97 108 117 101 0 0 0 5 26 23 121 53 75 232 248 43 32 85 63 51 117 191 195 16 202 62 28 56 1 0 0 0 13 204 140 189 26 34 229 202 46 8 234 118 106 36 96 231 233 81 113 75 126 1 0 0 0 15 60 71 101 14 51 126 140 76 44 10 121 175 247 238 31 87 44 78 98 183 1 0 0 0 16 3 193 67 97 112 172 57 28 148 224 98 41 44 91 137 213 171 101 123 185 1 0 0 0 26 117 94 231 241 20 173 38 107 213 103 249 199 8 37 255 116 186 25 10 33 1 0 0 0 27 56 247 5 176 190 249 14 134 94 85 254 200 190 122 99 179 53 148 222 33 1 0 0 0 28 85 133 213 241 183 107 106 251 244 65 147 145 185 55 73 12 78 243 135 82 1 0 0 0 31 217 8 46 75 198 199 245 142 187 22 106 133 29 80 15 201 128 40 188 236 1 0 0 0 32 6 222 207 37 178 245 208 102 203 154 140 114 134 232 186 26 36 218 175 176 1 0 0 0 37 28 37 20 83 195 192 69 139 144 61 175 98 166 36 146 25 70 103 252 91 1 0 0 0 55 181 110 33 55 9 159 217 232 16 25 84 84 216 82 250 94 183 193 220 31 1 0 0 0 74 19 191 230 232 246 122 73 74 165 83 253 112 72 157 47 65 197 155 242 86 1 0 0 0 79 127 44 61 132 8 226 29 111 146 93 70 154 236 41 223 153 210 75 199 43 1 0 0 0 94 81 92 142 214 190 38 48 168 247 44 27 191 24 68 2 224 39 205 65 231 1 0 0 0 104 243 68 61 37 246 14 26 34 208 241 71 30 204 197 23 94 182 50 132 217 1 0 0 0 111 7 250 237 122 208 207 48 129 169 172 243 65 162 49 206 16 49 199 6 231 1 0 0 0 122 252 246 137 101 65 218 141 231 248 156 247 208 238 204 71 54 176 129 55 180 1 0 0 0 143 95 196 204 192 95 189 146 38 230 151 101 248 239 177 187 136 52 151 98 54 1 0 0 0 145 73 159 118 206 31 26 191 164 33 242 108 23 196 210 204 247 29 140 219 31 1 0 0 0 146 86 35 172 218 11 20 197 215 133 109 244 233 255 48 45 86 147 240 137 38 1 0 0 0 148 213 245 31 231 130 184 145 49 88 246 189 218 191 171 159 76 223 26 248 24 1 0 0 0 149 91 16 204 198 61 163 193 217 171 10 72 251 138 58 111 248 56 169 177 165 1 0 0 0 153 113 81 189 217 138 186 96 141 183 172 175 135 225 194 104 172 237 214 29 27 1 0 0 0 157 73 209 112 234 81 136 24 112 163 134 12 153 236 122 212 96 36 72 84 175 1 0 0 0 159 15 3 252 218 185 15 147 8 237 16 255 105 98 202 206 144 18 108 152 159 1 0 0 0 166 225 157 68 187 140 157 20 201 165 9 136 190 142 208 88 85 176 19 234 47 1 0 0 0 178 194 182 80 205 88 148 36 168 246 17 253 169 124 98 116 248 33 130 76 56 1 0 0 0 183 200 138 66 80 26 185 153 187 102 247 50 205 147 89 161 2 205 22 8 105 1 0 0 0 185 251 87 106 195 9 171 216 175 32 217 212 236 204 80 217 57 167 185 209 123 1 0 0 0 192 116 193 249 127 130 214 63 54 142 245 44 240 31 167 208 37 63 172 225 238 1 0 0 0 196 204 204 192 128 13 222 75 214 24 68 168 26 27 62 252 80 76 36 3 201 1 0 0 0 200 60 123 249 144 98 231 231 146 151 221 39 206 64 104 220 246 8 47 234 245 1 0 0 0 213 111 181 6 118 251 214 246 158 20 9 85 104 94 252 39 90 43 37 22 103 1 0 0 0 234 198 1 128 51 193 232 252 238 174 221 165 148 187 44 172 135 232 20 1 80 1 0 0 0 241 71 31 151 197 55 170 13 158 181 152 234 107 145 188 191 11 62 141 156 132 1 0 0 1 2 18 87 213 254 35 43 247 28 12 130 91 248 226 46 81 220 84 204 30 159 1 0 0 1 15 176 255 146 217 82 94 39 228 165 39 53 116 119 255 247 254 66 208 5 11 1 0 0 1 38 182 235 111 165 75 27 158 219 85 1 20 152 183 159 188 209 238 194 227 103 1 0 0 1 41 240 14 173 183 50 131 142 200 251 61 89 207 199 178 21 118 72 76 206 191 1 0 0 1 48 196 37 86 175 122 95 64 130 102 250 200 35 0 37 226 92 153 57 240 79 1 0 0 1 50 11 84 223 34 194 254 67 158 239 138 110 152 55 157 213 157 88 138 203 20 1 0 0 1 51 57 83 119 19 19 121 151 239 239 14 231 76 79 195 250 232 245 39 177 192 1 0 0 1 52 143 252 180 60 74 189 18 226 16 8 169 142 4 129 83 158 129 170 238 68 1 0 0 1 58 129 81 102 251 32 248 3 51 254 194 212 30 130 164 154 138 197 40 168 23 1 0 0 1 69 48 26 183 77 137 230 152 131 36 234 81 69 227 22 121 101 48 9 94 4 1 0 0 1 71 30 181 186 251 234 91 133 74 20 63 173 120 117 113 221 0 58 113 62 152 1 0 0 1 81 157 167 66 220 7 109 79 223 78 172 100 132 86 75 170 69 85 131 162 18 1 0 0 1 90 13 138 7 75 51 186 25 17 196 180 224 135 229 12 235 170 121 181 2 117 1 0 0 1 91 229 172 227 107 24 214 84 222 201 152 188 171 211 64 176 78 51 84 100 66 1 0 0 1 104 230 62 0 132 148 52 99 181 48 206 212 212 242 205 154 67 92 249 79 202 1 0 0 1 109 31 55 91 140 157 150 180 6 18 155 203 90 162 136 131 23 234 111 65 217 1 0 0 1 120 194 98 217 249 191 71 82 143 37 4 49 98 129 172 252 109 167 169 147 63 1 0 0 1 124 30 175 141 148 29 255 92 57 171 174 220 142 70 26 55 197 228 151 46 215 1 0 0 1 140 235 187 67 233 249 9 59 72 20 80 91 60 206 100 74 239 54 100 170 226 1 0 0 ...])

0xc7 Buffer is:%!(EXTRA []uint8=[152 199 0 1 203 64 144 211 33 30 153 155 122 203 64 144 211 33 30 153 155 122 203 64 144 211 33 30 153 155 122 203 64 144 211 33 30 153 155 122 203 64 144 211 33 30 153 155 122 203 64 144 211 33 30 153 155 122 203 64 144 211 33 30 153 155 122 1 0 1 0 0 0 8 0 115 105 103 110 97 108 115 0 0 0 9 1 0 0 5 118 97 108 117 101 0 0 0 16 33 37 96 65 26 109 182 159 71 64 220 190 147 141 120 69 48 228 201 255 1 0 0 0 17 250 102 33 200 253 16 79 111 64 6 8 125 153 225 27 21 13 199 110 42 1 0 0 0 18 121 221 232 82 171 162 84 183 28 43 158 226 22 248 90 35 65 123 103 68 1 0 0 0 39 23 27 25 61 144 249 43 78 176 195 137 131 31 123 38 125 204 209 137 142 1 0 0 0 43 193 62 119 29 171 157 125 15 46 99 207 66 232 131 205 233 206 230 204 200 1 0 0 0 45 248 130 14 180 107 184 135 240 47 38 218 27 83 26 131 146 78 67 180 196 1 0 0 0 50 253 2 133 184 227 53 8 118 100 191 157 183 66 155 111 50 170 204 27 200 1 0 0 0 55 145 160 202 158 126 216 144 218 71 116 131 10 182 123 233 97 235 53 66 105 1 0 0 0 58 194 176 111 187 90 254 220 5 49 44 228 50 78 179 99 85 165 190 211 163 1 0 0 0 67 137 93 164 242 174 102 140 51 23 207 199 189 140 185 76 162 59 23 118 169 1 0 0 0 72 154 85 158 207 24 111 221 63 74 60 152 184 119 33 143 210 41 134 166 183 1 0 0 0 90 232 190 110 244 83 250 62 89 122 178 242 209 172 160 67 45 212 24 218 53 1 0 0 0 113 3 58 15 34 192 220 36 180 125 22 9 252 50 47 164 98 35 34 78 102 1 0 0 0 125 91 163 18 215 82 65 173 98 21 102 73 213 218 150 235 162 182 192 195 219 1 0 0 0 145 28 17 92 119 238 127 27 66 100 248 251 8 110 66 44 209 225 114 181 127 1 0 0 0 148 101 187 89 163 50 136 49 118 57 70 178 218 58 79 100 80 174 228 68 16 1 0 0 0 149 198 27 198 30 250 219 179 213 1 187 132 187 118 241 249 55 148 13 9 74 1 0 0 0 150 150 225 44 253 166 109 130 53 65 2 72 213 164 145 167 102 2 57 160 168 1 0 0 0 157 197 176 170 132 56 146 202 185 177 149 246 40 15 48 88 102 187 192 8 171 1 0 0 0 160 87 90 40 18 103 143 173 208 108 145 243 114 231 3 25 45 54 174 82 43 1 0 0 0 170 40 59 181 209 75 193 225 75 3 220 218 166 103 130 245 65 64 9 102 105 1 0 0 0 183 180 142 35 94 178 235 231 78 84 232 191 71 162 151 71 245 243 120 40 135 1 0 0 0 186 193 254 235 214 79 69 189 181 14 93 237 185 76 204 239 48 247 126 165 108 1 0 0 0 188 192 11 131 149 186 37 152 30 15 182 215 213 111 50 204 171 55 113 159 171 1 0 0 0 190 253 18 240 152 250 161 144 5 61 8 149 182 93 123 56 241 197 111 157 224 1 0 0 0 194 16 140 96 207 21 21 75 196 8 96 169 41 42 58 52 179 69 146 169 116 1 0 0 0 202 95 134 178 81 23 23 73 177 125 139 92 135 223 176 99 209 53 176 36 2 1 0 0 0 206 139 83 112 24 134 145 180 114 192 236 218 94 61 92 7 135 200 98 154 121 1 0 0 0 226 69 196 247 37 65 180 233 246 88 207 6 231 216 129 56 84 157 37 136 77 1 0 0 0 229 27 35 159 136 156 19 180 223 64 22 203 149 108 159 77 240 10 196 71 47 1 0 0 0 234 107 125 101 63 251 115 59 166 141 28 233 13 12 12 99 8 139 8 124 149 1 0 0 0 238 169 77 32 79 140 92 52 62 217 163 96 41 27 243 55 228 0 139 228 3 1 0 0 0 244 30 153 249 151 68 135 138 113 225 244 128 149 121 113 153 115 100 11 90 251 1 0 0 1 10 126 203 238 24 4 17 3 48 122 215 162 168 138 57 169 192 146 181 118 140 1 0 0 1 14 14 62 133 25 103 6 2 216 9 145 200 244 45 206 54 146 224 252 197 4 1 0 0 1 16 10 202 137 26 216 29 74 93 38 11 3 97 59 194 77 18 195 96 35 247 1 0 0 1 21 173 29 121 49 170 90 131 47 98 13 60 92 120 69 21 60 89 49 62 243 1 0 0 1 35 16 160 61 156 102 45 135 146 91 147 206 250 77 40 150 117 247 168 96 96 1 0 0 1 43 107 141 70 192 180 145 5 218 38 51 33 42 27 179 44 234 235 20 237 223 1 0 0 1 47 63 235 240 34 247 140 195 186 78 185 33 47 67 113 156 224 162 73 18 69 1 0 0 1 51 234 194 66 150 85 165 220 71 191 50 214 190 147 95 228 200 46 42 149 119 1 0 0 1 52 7 5 193 90 126 2 213 63 140 123 209 74 60 251 148 109 111 219 205 246 1 0 0 1 58 37 50 160 95 172 41 174 186 19 154 27 110 106 154 169 248 115 72 200 173 1 0 0 1 61 30 121 57 231 241 35 223 26 113 244 142 56 187 75 1 226 73 63 29 234 1 0 0 1 80 13 6 77 130 131 179 4 214 239 40 160 72 35 161 96 28 34 199 156 120 1 0 0 1 99 253 230 188 63 119 52 78 213 180 78 159 183 64 73 167 201 232 148 79 248 1 0 0 1 105 47 153 26 215 30 233 159 25 137 4 146 141 157 51 85 90 185 98 12 123 1 0 0 1 110 163 214 88 64 31 107 177 38 135 140 37 239 30 192 153 54 215 8 229 205 1 0 0 1 116 123 160 94 195 90 6 51 90 238 78 212 187 119 92 189 138 25 253 62 20 1 0 0 1 134 73 12 116 216 110 68 154 250 181 21 205 231 184 81 143 8 217 250 216 225 1 0 0 0 0....])

0xc7 Buffer is:%!(EXTRA []uint8=[152 199 0 1 203 64 166 164 148 109 197 214 57 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 226 203 64 166 165 114 76 95 70 225 203 64 166 165 114 76 95 70 225 1 0 1 0 0 0 8 0 115 105 103 110 97 108 115 0 0 0 9 1 0 0 5 118 97 108 117 101 0 0 0 23 154 41 85 46 91 166 175 79 42 250 103 51 46 66 99 206 40 191 208 231 1 0 0 0 50 60 82 90 116 173 126 109 92 33 147 19 233 166 109 98 157 7 128 217 64 1 0 0 0 68 63 83 195 39 117 180 42 73 25 207 101 7 17 199 61 174 40 164 144 77 1 0 0 0 69 40 27 209 128 73 131 77 116 150 9 120 11 8 26 227 74 231 254 118 220 1 0 0 0 80 213 191 25 203 208 96 198 177 38 98 18 77 225 130 111 42 204 99 93 20 1 0 0 0 83 202 27 252 0 128 196 92 17 73 115 250 57 92 41 225 243 74 148 132 242 1 0 0 0 95 222 34 49 113 212 27 147 144 51 82 251 189 72 119 69 144 120 151 160 248 1 0 0 0 99 72 22 49 106 21 34 132 225 40 79 247 158 245 3 9 58 207 133 206 250 1 0 0 0 100 36 239 57 39 43 108 130 160 127 110 238 181 148 60 135 74 76 165 109 250 1 0 0 0 117 3 238 114 131 99 24 68 216 127 193 121 250 30 249 8 55 96 119 34 99 1 0 0 0 140 171 204 182 8 86 46 172 183 159 5 108 67 94 237 124 199 11 232 56 248 1 0 0 0 141 126 254 254 123 57 208 171 236 73 69 14 156 55 86 197 191 154 255 176 36 1 0 0 0 145 38 126 8 160 132 54 150 220 72 196 125 125 99 169 20 104 116 255 243 188 1 0 0 0 149 207 230 55 67 101 211 36 49 50 155 63 193 150 35 149 58 143 252 248 10 1 0 0 0 161 65 108 53 203 251 171 15 50 172 137 74 110 10 42 193 103 145 98 167 100 1 0 0 0 190 227 16 114 123 210 169 91 32 31 63 137 89 237 18 61 126 161 44 188 148 1 0 0 0 198 133 242 122 130 242 246 65 177 136 90 97 141 117 17 225 179 192 114 138 153 1 0 0 0 199 88 123 127 160 188 230 152 130 74 254 156 197 171 201 129 51 135 100 141 46 1 0 0 0 203 62 2 229 122 65 7 2 165 197 113 204 229 169 136 188 83 198 230 145 17 1 0 0 0 226 134 59 223 182 36 201 213 120 172 205 169 173 104 211 227 5 45 194 8 88 1 0 0 0 229 241 125 176 25 249 51 133 41 16 222 65 161 104 189 101 12 8 33 233 108 1 0 0 0 237 78 108 60 75 199 146 100 147 0 216 147 29 106 25 211 44 152 132 1 246 1 0 0 0 244 251 57 4 151 141 143 138 32 7 179 117 38 68 210 79 182 138 61 255 2 1 0 0 0 249 39 191 5 81 168 31 180 173 139 246 99 243 62 65 116 208 224 12 68 92 1 0 0 0 251 124 163 92 237 146 19 56 118 126 158 165 23 150 50 189 255 46 21 177 5 1 0 0 0 255 142 233 1 71 229 21 222 217 29 176 60 10 200 178 243 97 166 24 82 46 1 0 0 1 4 18 234 105 32 142 226 194 239 229 146 203 235 135 214 80 77 32 58 59 191 1 0 0 1 13 135 154 222 34 101 38 49 176 111 235 253 108 96 190 3 233 31 211 49 149 1 0 0 1 16 164 199 225 137 83 76 125 123 133 200 117 158 238 22 244 96 132 255 54 205 1 0 0 1 25 26 43 13 196 45 160 117 101 69 33 139 203 128 110 54 3 124 57 104 47 1 0 0 1 30 12 226 198 104 205 240 9 83 42 77 194 236 68 197 128 224 49 14 4 77 1 0 0 1 37 165 200 211 224 59 231 213 215 137 1 108 187 66 224 57 238 187 11 63 205 1 0 0 1 41 243 211 165 122 217 188 177 191 12 115 235 82 198 23 169 191 166 10 31 227 1 0 0 1 68 50 202 137 144 232 57 252 101 111 89 50 228 29 35 52 3 16 15 12 171 1 0 0 1 80 3 222 246 36 81 33 211 118 155 10 49 97 50 187 41 10 118 120 0 233 1 0 0 1 82 104 126 209 152 8 157 181 1 96 210 148 19 166 86 141 226 29 218 81 113 1 0 0 1 91 180 62 25 106 83 178 80 55 245 203 56 210 60 229 121 54 25 223 33 93 1 0 0 1 101 106 154 30 46 78 3 216 118 244 38 217 153 8 163 89 222 243 56 88 144 1 0 0 1 105 139 35 164 137 111 136 79 142 117 255 39 194 48 51 72 130 186 78 217 94 1 0 0 1 114 252 211 174 164 192 240 97 242 174 81 250 130 162 182 77 208 52 83 127 22 1 0 0 1 124 10 233 42 160 178 165 22 228 93 17 147 119 158 65 252 165 115 158 235 63 1 0 0 1 127 65 92 41 205 10 11 11 112 100 78 254 220 146 55 18 218 221 68 157 199 1 0 0 1 137 197 154 6 255 34 143 206 16 75 229 35 205 190 201 135 173 85 127 207 192 1 0 0...])

It could possibly be that your are reading those bins recursively/iteratively and consuming bytes/chunks of bytes as per your message protocol between client and server. but this is the only peculiar distinction (more bytes in buffer for failed cases) I found out. please let me know if this is sufficient buffer information to assist with debugging or you want more information from my side on these lines.


#19

Hey Mr Khosrow, did you get a chance to revisit on this? the Gets/UDFs(single) are working fine just that BatchGet has some issues. So alternative exists as of now but the performance is very poor compare to batches.

I have looked into client code briefly and have one request - currently UDF operate on single record so the executes are very slow. if we can provide a BatchExecute method with similar concurrent WaitGroup mechanism(which exists in BatchGet today), it would solve not only the batchget problem but also would provide additional power to UDF as a feature.


#20

Hey Team, Any luck with this. we are going to see a 10x performance boost with Aerospike if this is fixed in our setup. we would need your little help before our setup is dismantled :frowning: