Iterating through rec.Bins["record"]

no1youknowz · July 19, 2015, 12:41am

So in python, I am saving to aerospike with:

client.put(_key, { 'record': value })

In Golang, I’m retrieving the records:

func Query(bin string, table string, _key string) interface{} {
	client, _ := Connect()

	key, err := as.NewKey(bin, table, _key)
	panicOnError(err)

	rec, err := client.Get(nil, key)
	panicOnError(err)

	record := rec.Bins["record"]

	return record
}

This returns:

[{"block_id": 0, "keyword_token": null, "url": "http://www2.www-bar.com", "cpc": 0, "campaign_id": 3, "sub_id_separator": "_", "domain_id": 1, "traffic_id": 0}]

I try:

for _, value := range query {
	fmt.Println(value)
}

I get

cannot range over query (type interface {})

So my question. How do I return something that I can iterate over?

Thanks

no1youknowz · July 19, 2015, 4:13pm

I have tried doing

record := rec.Bins["record"].(string)

but when I try to iterate.

for key, value := range record {
	fmt.Println("Key:", key, "Value:", value)
}

I get:

Key: 0 Value: 91
Key: 1 Value: 123
Key: 2 Value: 34
Key: 3 Value: 98
Key: 4 Value: 108
Key: 5 Value: 111
Key: 6 Value: 99
Key: 7 Value: 107
Key: 8 Value: 95
Key: 9 Value: 105
Key: 10 Value: 100
Key: 11 Value: 34
Key: 12 Value: 58
Key: 13 Value: 32
Key: 14 Value: 48
Key: 15 Value: 44
Key: 16 Value: 32
Key: 17 Value: 34
Key: 18 Value: 107
Key: 19 Value: 101
Key: 20 Value: 121
Key: 21 Value: 119
Key: 22 Value: 111
Key: 23 Value: 114
Key: 24 Value: 100
Key: 25 Value: 95
Key: 26 Value: 116
Key: 27 Value: 111
Key: 28 Value: 107
Key: 29 Value: 101
Key: 30 Value: 110
Key: 31 Value: 34
Key: 32 Value: 58
Key: 33 Value: 32
Key: 34 Value: 110
Key: 35 Value: 117
Key: 36 Value: 108
Key: 37 Value: 108
Key: 38 Value: 44
Key: 39 Value: 32
Key: 40 Value: 34
Key: 41 Value: 117
Key: 42 Value: 114
Key: 43 Value: 108
Key: 44 Value: 34
Key: 45 Value: 58
Key: 46 Value: 32
Key: 47 Value: 34
Key: 48 Value: 104
Key: 49 Value: 116
Key: 50 Value: 116
Key: 51 Value: 112
Key: 52 Value: 58
Key: 53 Value: 47
Key: 54 Value: 47
Key: 55 Value: 119
Key: 56 Value: 119
Key: 57 Value: 119
Key: 58 Value: 50
Key: 59 Value: 46
Key: 60 Value: 119
Key: 61 Value: 119
Key: 62 Value: 119
Key: 63 Value: 45
Key: 64 Value: 99
Key: 65 Value: 111
Key: 66 Value: 110
Key: 67 Value: 110
Key: 68 Value: 101
Key: 69 Value: 99
Key: 70 Value: 116
Key: 71 Value: 105
Key: 72 Value: 110
Key: 73 Value: 103
Key: 74 Value: 46
Key: 75 Value: 99
Key: 76 Value: 111
Key: 77 Value: 109
Key: 78 Value: 34
Key: 79 Value: 44
Key: 80 Value: 32
Key: 81 Value: 34
Key: 82 Value: 99
Key: 83 Value: 112
Key: 84 Value: 99
Key: 85 Value: 34
Key: 86 Value: 58
Key: 87 Value: 32
Key: 88 Value: 48
Key: 89 Value: 44
Key: 90 Value: 32
Key: 91 Value: 34
Key: 92 Value: 99
Key: 93 Value: 97
Key: 94 Value: 109
Key: 95 Value: 112
Key: 96 Value: 97
Key: 97 Value: 105
Key: 98 Value: 103
Key: 99 Value: 110
Key: 100 Value: 95
Key: 101 Value: 105
Key: 102 Value: 100
Key: 103 Value: 34
Key: 104 Value: 58
Key: 105 Value: 32
Key: 106 Value: 51
Key: 107 Value: 44
Key: 108 Value: 32
Key: 109 Value: 34
Key: 110 Value: 115
Key: 111 Value: 117
Key: 112 Value: 98
Key: 113 Value: 95
Key: 114 Value: 105
Key: 115 Value: 100
Key: 116 Value: 95
Key: 117 Value: 115
Key: 118 Value: 101
Key: 119 Value: 112
Key: 120 Value: 97
Key: 121 Value: 114
Key: 122 Value: 97
Key: 123 Value: 116
Key: 124 Value: 111
Key: 125 Value: 114
Key: 126 Value: 34
Key: 127 Value: 58
Key: 128 Value: 32
Key: 129 Value: 34
Key: 130 Value: 95
Key: 131 Value: 34
Key: 132 Value: 44
Key: 133 Value: 32
Key: 134 Value: 34
Key: 135 Value: 100
Key: 136 Value: 111
Key: 137 Value: 109
Key: 138 Value: 97
Key: 139 Value: 105
Key: 140 Value: 110
Key: 141 Value: 95
Key: 142 Value: 105
Key: 143 Value: 100
Key: 144 Value: 34
Key: 145 Value: 58
Key: 146 Value: 32
Key: 147 Value: 49
Key: 148 Value: 44
Key: 149 Value: 32
Key: 150 Value: 34
Key: 151 Value: 116
Key: 152 Value: 114
Key: 153 Value: 97
Key: 154 Value: 102
Key: 155 Value: 102
Key: 156 Value: 105
Key: 157 Value: 99
Key: 158 Value: 95
Key: 159 Value: 105
Key: 160 Value: 100
Key: 161 Value: 34
Key: 162 Value: 58
Key: 163 Value: 32
Key: 164 Value: 48
Key: 165 Value: 125
Key: 166 Value: 93

no1youknowz · July 19, 2015, 4:57pm

Got some help in the #go-nuts channel on irc.

record := []byte(rec.Bins["record"].(string))

	type Data struct {
		SubIDSeparator string  `json:"sub_id_separator"`
		DomainID       float64 `json:"domain_id"`
		TrafficID      float64 `json:"traffic_id"`
		BlockID        float64 `json:"block_id"`
		KeywordToken   *string `json:"keyword_token"`
		URL            string
		CPC            float64
		CampaingID     float64 `json:"campaign_id"`
	}

	var rows []Data
	json.Unmarshal(record, &rows)

	fmt.Println(rows)

Gives me:

[{_ 1 0 0 <nil> http://www2.www-connecting.com 0 3}]

So I give up.

Would be nice to have some better examples in the documentation? For those who don’t know Golang?

Khosrow_Afroozeh · July 19, 2015, 5:35pm

Hi there, thanks for giving our golang driver a try.

To be fair, teaching Go is out of the scope of a database driver documentation. That being said, you have already got very good advice on the #IRC.

Don’t give up. You are almost there:


// rec.Bins is of sub-type map[string]interface{}
record := []byte(rec.Bins["record"].(string))

type Data struct {
    SubIDSeparator string  `json:"sub_id_separator"`
    DomainID       float64 `json:"domain_id"`
    TrafficID      float64 `json:"traffic_id"`
    BlockID        float64 `json:"block_id"`
    KeywordToken   *string `json:"keyword_token"`
    URL            string
    CPC            float64
    CampaingID     float64 `json:"campaign_id"`
}

var rows []Data

// since you are saving your whole record as a json string,
// you will have to unmarshall it back into a struct. You can also pass a map[string]interface{}
// to unmarshaller 
err := json.Unmarshal(record, &rows)
// always handle errors to make sure everything went well
if err != nil {
   log.Fatal(err)
}

// rows is an array of records, of type 'Data'
// You can iterate over this array, and access each attribute of the object
for _, rec := range rows {
    fmt.Println(rec.SubIDSeparator)
    fmt.Println(rec.DomainID)
    fmt.Println(rec.TrafficID)
   // ...
}

Putting all your data as a json string in one bin (of name “record”) in python is not a good idea. You will lose the ability to index those attributes (e.g. DomainID), and will lose performance since not only you can’t just retrieve a single bin, but each put/set will require a de/serialization which is slow in all languages.

I encourage you to discuss your use case in our Python forum to get advice. If you are trying to rewrite your app in Go, let me know so that I can give you some pointers to get started.

Cheers, K.

no1youknowz · July 19, 2015, 5:48pm

Thanks for your answer.

I actually just managed to do this:

record := rec.Bins["record"]

	type Data struct {
		SubIDSeparator string  `json:"sub_id_separator"`
		DomainID       float64 `json:"domain_id"`
		TrafficID      float64 `json:"traffic_id"`
		BlockID        float64 `json:"block_id"`
		KeywordToken   *string `json:"keyword_token"`
		URL            string
		CPC            float64
		CampaingID     float64 `json:"campaign_id"`
	}

	var rows []Data
	json.Unmarshal([]byte(record.(string)), &rows)

	for _, row := range rows {
		fmt.Printf(string(row.URL))
	}

	fmt.Println("")

	return record

But your answer is better. So I will look at this more in detail.

I’m actually rewriting a PHP app in Golang and I have python as a backend process marshaling the data between them.

Thanks for the post, I appreciate it.

no1youknowz · July 19, 2015, 6:05pm

Ok, so it’s better to explain what I’m doing.

I’m getting a JSON payload from php. It’s a backend process so that the user doesn’t have to wait.

Python gets a hold of it and does:

def process_payload(payload):
        bin = str(payload['bin'])
        table = str(payload['table'])
        key = str(payload['key'])
        value = payload['value']

        # Records are addressable via a tuple of (namespace, set, key)
        _key = (bin, table, key)

        print "updating key - " + key

        client.put(_key, { 'record': value })

Which gets put into aerospike.

What should I be doing instead? There will also be times when I will have nested sets for multiple records.

I think the root cause of all this.

Is that I’m getting a [ { “foo” : “bar” } ] json and if I don’t have record, then I get an error from aerospike saying there should be a key/pair.

I think if I can get python to receive { “foo” : “bar” }, then that will get inserted fine into aerospike and I won’t have these issues.

Another problem is that I’m retrieving the data from a database from another python process which makes it .

I think I need to simplify my data before anything else.

Khosrow_Afroozeh · July 19, 2015, 6:17pm

Deserialize the value into a map (assuming it’s a string) and pass that map into Aerospike.

If value is a map, just pass it directly to put:

client.put(_key, value)

from the Go side, you can read such a record this way:

	rec, err := client.Get(nil, key)
	panicOnError(err)
        
        fmt.Println(rec.Bins["URL"].(string));

OR even better:

    rec := &Data{}
    err := client.GetObj(nil, key, rec)
    panicOnError(err)

    fmt.Println(rec.URL);

Nested objects are retrieved as map[interface{}]interface{}; nested arrays are retrieved as []interface{}

The second method is better since there is less boiler plate, and it is type safe.

Just make sure you handle errors instead of panic in production.

no1youknowz · July 19, 2015, 6:33pm

Awesome, I’ll give this a go tomorrow.

Thanks!

no1youknowz · July 20, 2015, 2:44pm

I resolved the data coming into aerospike, so now I’m trying to fix the issue overall.

This is my complete code for the package:

package core

import (
	"fmt"
	as "github.com/aerospike/aerospike-client-go"
)

func Connect() (*as.Client, error) {
	client, err := as.NewClient("172.16.153.175", 3000)
	panicOnError(err)

	return client, err
}

func Query(bin string, table string, _key string) {
	client, _ := Connect()

	type Data struct {
		SubIDSeparator string  `json:"sub_id_separator"`
		DomainID       float64 `json:"domain_id"`
		TrafficID      float64 `json:"traffic_id"`
		BlockID        float64 `json:"block_id"`
		KeywordToken   *string `json:"keyword_token"`
		URL            string
		CPC            float64
		CampaingID     float64 `json:"campaign_id"`
	}

	key, _ := as.NewKey(bin, table, _key)

	rec := &Data{}
	err := client.GetObj(nil, key, rec)

	fmt.Println(rec.URL)
}

func panicOnError(err error) {
	if err != nil {
		panic(err)
	}
}

I’m getting

 core/base_model.go:32: client.GetObj undefined (type *aerospike.Client has no field or method GetObj)

Khosrow_Afroozeh · July 20, 2015, 3:41pm

Duh, The API is Client.GetObject()

You can consult GoDoc for the API documentation.

no1youknowz · July 20, 2015, 3:55pm

Thanks for the reply. Appreciate it.

I’m kinda doing this at 20% of my time. In between when I get stuck on something else.

I even looked at the client.go and I didn’t even twig.

func (clnt *Client) GetObject(policy *BasePolicy, key *Key, obj interface{}) error {

I completely missed it lol!

no1youknowz · July 20, 2015, 4:49pm

Sorry I need so much help, but I don’t understand this completely yet.

key, _ := as.NewKey(bin, table, _key)

rec, err := client.Get(nil, key)
panicOnError(err)

fmt.Println(rec.Bins)

Works:

 map[subid_sep:_ traffic_id:0 cpc:0 domain_id:1 campaign_id:4]

.

type Data struct {
		SubIDSeparator string `json:"subid_sep"`
		DomainID       int64  `json:"domain_id"`
		TrafficID      int64  `json:"traffic_id"`
		//BlockID        float64 `json:"block_id"`
		//KeywordToken   *string `json:"keyword_token"`
		//URL            string
		CPC        float64 `json:"cpc"`
		CampaingID int64   `json:"campaign_id"`
	}

	key, _ := as.NewKey(bin, table, _key)

	rec := &Data{}
	err2 := client.GetObject(nil, key, rec)

	fmt.Println(err2)

	if err2 != nil {
		fmt.Println(rec)
	}

Does not.

I get

 <nil>

Khosrow_Afroozeh · July 20, 2015, 5:04pm

Comments inline:

// This is what prints the nil
fmt.Println(err2)

// since err2 == nil, you won't get into this block, and it is not printed
if err2 != nil {
    fmt.Println(rec)
}

no1youknowz · July 20, 2015, 5:17pm

I know that. I should have given you my updated code:

type Data struct {
		SubIDSeparator string `json:"subid_sep"`
		DomainID       int64  `json:"domain_id"`
		TrafficID      int64  `json:"traffic_id"`
		//BlockID        float64 `json:"block_id"`
		//KeywordToken   *string `json:"keyword_token"`
		//URL            string
		CPC        float64 `json:"cpc"`
		CampaingID int64   `json:"campaign_id"`
	}

	key, _ := as.NewKey(bin, table, _key)

	rec := &Data{}
	client.GetObject(nil, key, rec)

	fmt.Println(rec)

Which prints:

&{ 0 0 0 0}

no1youknowz · July 20, 2015, 5:28pm

To be clear:

The output of

 client.GetObject(nil, key, rec)

is

 <nil>

Khosrow_Afroozeh · July 20, 2015, 6:04pm

That is because you’re not telling the client what the name of the fields are. You should tag your struct fields:

type Data struct {
		SubIDSeparator string `json:"subid_sep" as:"subid_sep"`
		DomainID       int64  `json:"domain_id" as:"domain_id"`
		TrafficID      int64  `json:"traffic_id" as:"traffic_id"`
		//BlockID        float64 `json:"block_id" as:"block_id"`
		//KeywordToken   *string `json:"keyword_token" as:"keyword_token"`
		//URL            string
		CPC        float64 `json:"cpc" as:"cpc"`
		CampaingID int64   `json:"campaign_id" as:"campaign_id"`
	}

no1youknowz · July 20, 2015, 7:40pm

Thanks for your help. I got what I needed to work!

Cheers

Topic		Replies	Views
Golang API Executing Query Go Client	2	2323	November 11, 2015
Is there a way to get only a particular bin of a record in Python client? query , python	3	1271	May 19, 2022
Iterating via Scan Seems to be Quite Slow Go Client query , secondary , scan , primary , index	5	3868	June 23, 2015
The record passed to the query callback function does not have the key populated correctly Query & Indexing	2	1078	December 14, 2021
Select set of bins Java Client	1	1449	August 16, 2014

Iterating through rec.Bins["record"]

Related Topics