Stream UDF gets stuck when used with Gunicorn

Client Libraries Python Client
1

Hi,

we ran wsgi compliant web server which calls Stream UDF lua function. Everything was working well with gevent.wsgi, but recently we are moving to Gunicorn Wsgi. Unfornutelly with this setup Stream UDF call get stuck. Function doesn’t return any value and blockes forever.

query = self.client.query(self.COUNTERS_NAMESPACE, self.COUNTERS_SET)

query.where(p.equals(self.UID_BIN, uid))

query.apply(self.FREQUENCY_MODULE, self.FREQUENCY_REACHED_FUNCTION, [ts_now])

records = query.results()

Stucks on the last line.

Any thoughts on this?

Thanks

Zdenek T.

2

Hi, This is something we will look into. I have a couple of additional questions:

  1. Do all stream UDFs get stuck, or is it just one in particular?
  2. Does a query without a stream UDF get stuck? See the following code for an example query with a no-op callback
query = self.client.query(self.COUNTERS_NAMESPACE, self.COUNTERS_SET)
query.where(p.equals(self.UID_BIN, uid))
def noop_callback(*args):
    return True
query.foreach(noop_callback)
3

I forgot to ask: Which version of Python are you running, and which version of the Aerospike Python Client?

4

I think the issue is due to the Python client multiple threads; when Gunicorn, or anything else , forks the Python interpreter it copies memory into a new interpreter process, but does not copy running threads.

So when query.results() gets called: the new interpreter processor waits to acquire a lock so it can do work safely; but this will never happen because the thread which is holding that lock is not running in the new interpeter’s process. So the lock remains locked forever, and the interpreter hangs.

I think you can get around this if you make sure to create and connect the Aerospike Client after the fork. So in Gunicorn, I imagine this would be creating the client in the post_fork hook: Settings — Gunicorn 20.1.0 documentation

5

Hi,

we are running python 3.6 and the latest aerospike client 3.4.2. We dont have any other Stream UDF so far, but classic key-based query (client.get(key)) works well. Anyway, it seems that you are right with the locking problem. We postpone connect method and UDF STream function doesn’t block anymore.

Thanks a lot for great product and support.

Zdenek