Rate limit in golang

Client Libraries Go Client
1

Hi,

I want to use Aerospike to cache ratelimit counters for my golang server app:

  • request comes in
  • increment counter, and only set expiration when the counter is created (not when it is updated)
  • check result and abort when counter exceeds limit
  • use different TTL’s for different keys, so the default namespace TTL is not used

I am able to do this in golang, but only with 2 roundtrips to the server (that pose an issue). Is there another way to do this with one roundtrip?

This is my working example code

func rateLimit(gin *Gin.Context, client *Aerospike.Client, limit uint32, seconds uint32) (bool, error) {

	const setName = "limit"

	const binName = "count"

	// policies

	policyDoNotTouchExpiration := Aerospike.NewWritePolicy(0, math.MaxUint32-1)

	policyUpdateExpiration := Aerospike.NewWritePolicy(0, seconds)

	// create key

	keyname := fmt.Sprint(gin.ClientIP(), ":", gin.Request.Method, ":", gin.Request.URL.Path)

	key, err := Aerospike.NewKey("default", setName, keyname)

	if err != nil {

		return false, err
	}

	record, err := client.Operate(policyDoNotTouchExpiration, key,

		Aerospike.AddOp(Aerospike.NewBin(binName, 1)),

		Aerospike.GetOp(),
	)

	if err != nil {

		return false, err
	}

	count := uint32(record.Bins[binName].(int))

	// only if record is created, adjust expiration
	// TODO can this be done in one roundtrip?

	if count == 1 {

		if err := client.Touch(policyUpdateExpiration, key); err != nil {

			return false, err
		}
	}

	return count <= limit, nil
}

The problem with this approach is that when the Touch fails, the rate limiter would not work.

It would be great if the WritePolicy would contain some kind of ExpirationPolicy where I can set behaviour to set expiration only on record creation (and possibly some other edge cases).

Or, there is a much simpler solution for my problem that I am not aware of …

Thanks, Cheers,

Gerd

2

Ver 3.10.1 introduced expiration value = -2, update the record without changing its current TTL.

3

I am running : Aerospike Community Edition build 3.11.0.2

But when I use … policy.Expiration = math.MaxUint32 - 2 … I get an error: “Parameter error”

Is the golang client not yet updated?

4

I don’t know Go - so don’t know the syntax. Just wondering how MaxUnit32 which should be 2^32, minus 2 is equal to -2? Please check that. There is max limit on TTL value now of 10 years. So you may be seeing that. Based on what I am seeing with AQL testing, if I create a record with expiration ttl = -2, on create it takes the default TTL of the namespace, on update it does not modify the TTL. Is that not what you are looking for? Try that. BTW, always good practice to set TTL that you need in namespace and not really modify through your application if you don’t have to.

5

Sorry, but I don’t see how this would solve my problem. I want to increment a counter, set its expiration on record create, and do not touch expiration when record already existed. All in 1 roundtrip.

How would… expiration value = -2 … set an expiration value to start with?

6

What is your namespace configuration for this record’s namespace? Don’t you have a default TTL there?

See my AQL testing. I start with ttl - 2 policy in application (AQL). when i create, takes default ttl of namespace, on each update, does not increment or refresh ttl, it keeps decreasing per original ttl:

aql> set record_ttl -2 aql> get record_ttl RECORD_TTL = -2

aql> INSERT INTO test.demo (PK, foo, bar) VALUES (‘key2’, 123, ‘abc’) OK, 1 record affected.

aql> select * from test.demo where pk = ‘key2’ [ { “digest”: “AAAAAAAAAAAAAAAAAAAAAAAAAAA=”, “ttl”: 2591995, “gen”: 1, “bins”: { “foo”: 123, “bar”: “abc” } } ]

aql> INSERT INTO test.demo (PK, foo, bar) VALUES (‘key2’, 123, ‘ab’) OK, 1 record affected.

aql> get record_ttl RECORD_TTL = -2

aql> select * from test.demo where pk = ‘key2’ [ { “digest”: “AAAAAAAAAAAAAAAAAAAAAAAAAAA=”, “ttl”: 2591965, “gen”: 2, “bins”: { “foo”: 123, “bar”: “ab” } } ]

aql> INSERT INTO test.demo (PK, foo, bar) VALUES (‘key2’, 123, ‘a’) OK, 1 record affected.

aql> select * from test.demo where pk = ‘key2’ [ { “digest”: “AAAAAAAAAAAAAAAAAAAAAAAAAAA=”, “ttl”: 2591455, “gen”: 3, “bins”: { “foo”: 123, “bar”: “a” } } ]

My namespace config is:

namespace test {
	replication-factor 2
	memory-size 4G
	default-ttl 30d # 30 days, use 0 to never expire/evict. (2,592,000 sec)

	storage-engine memory
}
7

in GO, the expiration value is an UINT32, I have checked the api:

I guess by expiration value = -2, in GO I should use TTLDontUpdate (MaxUint32 - 1). And by examining my example code, you can see that I use that approach.

A default TTL is of no use to me, because I want to set different expiration values for different keys. So I MUST set an expiration value on record create, and I MUST NOT set/update an expiration value on record update/increment. And that in 1 roundtrip. The workaround that I use (my example code) works, but rate-limiting fails when the second roundtrip for some reason would fail.

Does that make sense?

From Aerospike GO API:

// Expiration determines record expiration in seconds. Also known as TTL (Time-To-Live).
    // Seconds record will live before being removed by the server.
    // Expiration values:
    // TTLServerDefault (0): Default to namespace configuration variable "default-ttl" on the server.
    // TTLDontExpire (MaxUint32): Never expire for Aerospike 2 server versions >= 2.7.2 and Aerospike 3 server
    // TTLDontUpdate (MaxUint32 - 1): Do not change ttl when record is written. Supported by Aerospike server versions >= 3.10.1
    // > 0: Actual expiration in seconds.
    Expiration uint32
8

I am thinking that if you want to move operational logic to the Aerospike server and do in one roundtrip, you may have to use a record udf.

9

I am fairly new to Aerospike, so I have to look into that.

An elegant option would be to expand:

// TTLServerDefault (0): Default to namespace configuration variable "default-ttl" on the server.
// TTLDontExpire (MaxUint32): Never expire for Aerospike 2 server versions >= 2.7.2 and Aerospike 3 server
// TTLDontUpdate (MaxUint32 - 1): Do not change ttl when record is written. Supported by Aerospike server versions >= 3.10.1

Add new values

// TTLDontUpdateOnNew (MaxUint32 - 2): Do not change ttl when record is created
// TTLDontUpdateOnExisting (MaxUint32 - 3): Do not change ttl when record is update

that would be great…

Do you know how I can make suggestions to the development team?

10

When you say dont change ttl when record is created, what ttl should the record have? Because you don’t seem to want the default ttl of the namespace? how does that solve your problem? you want to take different actions based on whether the record initially exists or not. writing a record udf is pretty straightforward and will solve your problem.

11

Thanks for your help

12

I was referring to this proposal from you.

Post here if you run into issues with UDF, will try to help the best I can.

13

I added that for completeness, not for my use-case. But giving it a second thought, you are right that is indeed of no use…

14

I managed to get this working with UDF’s

function rateLimit(rec, requests, seconds)

	if aerospike:exists(rec) then

		local count = rec['count'] + 1

		rec['count'] = count

		record.set_ttl(rec, -2)

		aerospike:update(rec)

		return count > requests

	end

	rec['count'] = 1

	record.set_ttl(rec, seconds)

	aerospike:create(rec)

    return false

end

Thanks!

1 Like