Exception in aerospike_batch_get C client lib 4.6.17

Client Libraries C Client Library
1

as_main_single_async_get: …/nptl/pthread_mutex_lock.c:115: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0’ failed.

Thread 13 “as_main_single_” received signal SIGFPE, Arithmetic exception.

[Switching to Thread 0x7fffc9ffe700 (LWP 9890)] 0x0000555555580be3 in as_queue_get (queue=0x555555cdd3b8, queue=0x555555cdd3b8, queue=0x555555cdd3b8, index=) at modules/common/target/Linux-x86_64/include/aerospike/as_queue.h:158 158 modules/common/target/Linux-x86_64/include/aerospike/as_queue.h: No such file or directory. (gdb) bt

#0 0x0000555555580be3 in as_queue_get (queue=0x555555cdd3b8, queue=0x555555cdd3b8, queue=0x555555cdd3b8, index=) at modules/common/target/Linux-x86_64/include/aerospike/as_queue.h:158

#1 as_queue_pop (ptr=0x7fffc9ffcab0, queue=0x555555cdd3b8) at modules/common/target/Linux-x86_64/include/aerospike/as_queue.h:171

#2 as_conn_pool_pop_head (sock=0x7fffc9ffcab0, pool=0x555555cdd390) at src/include/aerospike/as_conn_pool.h:76

#3 as_node_get_connection (err=err@entry=0x7fffc9ffda70, node=node@entry=0x555555ccfbd0, socket_timeout=330, deadline_ms=1713350659, sock=sock@entry=0x7fffc9ffcb80) at src/main/aerospike/as_node.c:519

#4 0x000055555556fffe in as_command_execute (cmd=cmd@entry=0x7fffc9ffcd30, err=err@entry=0x7fffc9ffda70) at src/main/aerospike/as_command.c:527

#5 0x00005555555663d2 in as_batch_execute_keys (btk=btk@entry=0x7fffc9ffd510, err=err@entry=0x7fffc9ffda70, parent=parent@entry=0x0) at src/main/aerospike/aerospike_batch.c:740

#6 0x0000555555569866 in as_batch_keys_execute (as=, err=0x7fffc9ffda70, policy=, batch=, read_attr=read_attr@entry=3, bins=bins@entry=0x0, n_bins=, n_bins@entry=0, callback=, callback_xdr=, callback_xdr@entry=0x0, udata=) at src/main/aerospike/aerospike_batch.c:1004

#7 0x000055555556ac7e in aerospike_batch_get (as=, err=, policy=, batch=, callback=, udata=) at src/main/aerospike/aerospike_batch.c:1940 #8 0x000055555556160a in play_batch_once_sync () at as_main_single_async_get.c:315 #9 0x000055555556299a in run_all_waves (arg=0x5) at as_main_single_async_get.c:579 #10 0x00007ffff7bbd6db in start_thread (arg=0x7fffc9ffe700) at pthread_create.c:463 #11 0x00007ffff6beca3f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb)

Before I added these lines, the code was good

    g_as_config.conn_pools_per_node = 400;
    g_as_config.min_conns_per_node  = 400;
    g_as_config.max_conns_per_node  = 400;
2

You have configured 400 connection pools for every node. This means each connection pool manages a single connection (which does not make sense). conn_pools_per_node should usually be left at the default (1) and rarely ever set to more than 5.

3

Thanks Brian for the inputs.

I am also trying async calls.

I have 400 application threads

From each thread I am firing aerospike_key_get_async for 4 keys

And then doing a common wait

The aerospike cluster server is a 3 node cluster, with 2 way replication.

Could you please give me a good formula to calculate

uint32_t as_config::async_max_conns_per_node

uint32_t as_config::async_min_conns_per_node

uint32_t as_config::conn_pools_per_node

event_loops required for as_create_event_loops()

The documentation is good, but this combination will need some more advice.

thanks

-Vikrant

4

event_loops required for as_create_event_loops()

Create one event loop for every cpu core.

uint32_t as_config::conn_pools_per_node

Leave at default (1). Async commands do not use conn_pools_per_node.

uint32_t as_config::async_max_conns_per_node

uint32_t as_config::async_min_conns_per_node

1600 (400 * 4) concurrent commands requires 1600 concurrent connections. This is way too many concurrent commands. You will need to set as_policy_event.max_commands_in_process to something reasonable like 40 per event loop. Then set async_max_conns_per_node to (40 * {number of event loops}).