Increase Maximum number of openfiles


#1

Summary

By default aerospike init script will set the maximum number of open files to 100000. (ulimit -n 100000) Its possible that this system setting change may fail due to permission issues.

Verification of current values

The current max open files setting can be verified on a running Aerospike server by reading the following file /proc/XXX/limits (where XXX is the pid of the running aerospike process)

root@u15:~# cat /proc/`pgrep asd`/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             256380               256380               processes 
Max open files            100000               100000               files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       256380               256380               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

if for some reason the Soft and hard limit are smaller than expected, you may need to manually set this value by modifying /etc/security/limits.conf

Setting Open Files limit for all users

Edit the following file: /etc/security/limits.conf

In the following example we are setting the wild card * for any user to 100000 files.

*         hard    nofile      100000
*         soft    nofile      100000

Note: You may need to logout and login again after having set this setting in the file for it to take affect.

Setting Open Files limit for asd process

Non-Systemd environments

Edit the following file /etc/init.d/aerospike

Modify the following line and add the maximum number file descriptors for asd process.

		ulimit -n 100000

Systemd environments

Create an override.conf file

Create a file named override.conf under /etc/systemd/system/aerospike.service.d/

With the following lines:

[Service]
LimitNOFILE=
LimitNOFILE=<MAX NUMBER OF FILE DESCRIPTORS> 

example file:

root@myserver:~# cat /etc/systemd/system/aerospike.service.d/override.conf 
[Service]
LimitNOFILE=
LimitNOFILE=200000

Reload systemd daemon

systemctl daemon-reload

Restart the aerospike service

systemctl restart aerospike.service

Error message in aerospike logs

May 20 2015 03:41:56 GMT: INFO (demarshal): (thr_demarshal.c::297) warning: hit OS file descript limit (EMFILE on accept), consider raising limit
May 20 2015 03:41:56 GMT: WARNING (cf:socket): (socket.c::274) socket: Too many open files
May 20 2015 03:41:56 GMT: WARNING (cf:socket): (socket.c::274) socket: Too many open files
May 20 2015 03:41:56 GMT: WARNING (cf:socket): (socket.c::274) socket: Too many open files

The above INFO message was changed to WARNING since 3.7.5:

Nov 14 2017 00:54:57 GMT: WARNING (demarshal): (thr_demarshal.c:454) Hit OS file descriptor limit (EMFILE on accept). Consider raising limit for uid 0

Note: always ensure that proto-fd-max configuration setting has a value that is less than the system/process limit. Depending on the size of cluster and configuration, it should be hundreds to thousands smaller. Please use this command to verify the open files for asd:

       lsof -p `pgrep asd`

For the limits set at the process:

       cat /proc/`pgrep asd`/limits

If there are not enough file descriptors, asd will crash:

Nov 14 2017 00:55:36 GMT: WARNING (socket): (socket.c:1616) Error while creating netlink socket: 24 (Too many open files)
Nov 14 2017 00:55:36 GMT: FAILED ASSERTION (socket): (socket.c:2050) Error while enumerating network links
Nov 14 2017 00:55:36 GMT: WARNING (as): (signal.c:210) SIGUSR1 received, aborting Aerospike Enterprise Edition build 3.15.0.2 os el6

Async client ArrayIndexOutOfBoundsException
#2

when i fired ulimit -n from one box its giving 1024. but max open files is 100000 as shown above . i am using async client and max connections per node i am seeing only 100 how can i increase them ??

i given asyncMaxCommands = 1000;

asyncSelectorThreads =10

please update how above two parameters will effect the speed