By default aerospike init script will set the maximum number of open files to 100000. (ulimit -n 100000) Its possible that this system setting change may fail due to permission issues.
Verification of current values
The current max open files setting can be verified on a running Aerospike server by reading the following file /proc/XXX/limits (where XXX is the pid of the running aerospike process)
root@u15:~# cat /proc/`pgrep asd`/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 256380 256380 processes Max open files 100000 100000 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 256380 256380 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
if for some reason the Soft and hard limit are smaller than expected, you may need to manually set this value by modifying /etc/security/limits.conf
Setting Open Files limit for all users
Edit the following file: /etc/security/limits.conf
In the following example we are setting the wild card * for any user to 100000 files.
* hard nofile 100000 * soft nofile 100000
Note: You may need to logout and login again after having set this setting in the file for it to take affect.
Setting Open Files limit for asd process
Edit the following file /etc/init.d/aerospike
Modify the following line and add the maximum number file descriptors for asd process.
ulimit -n 100000
Create an override.conf file
Create a file named override.conf under /etc/systemd/system/aerospike.service.d/
With the following lines:
[Service] LimitNOFILE= LimitNOFILE=<MAX NUMBER OF FILE DESCRIPTORS>
root@myserver:~# cat /etc/systemd/system/aerospike.service.d/override.conf [Service] LimitNOFILE= LimitNOFILE=200000
Reload systemd daemon
Restart the aerospike service
systemctl restart aerospike.service
Error message in aerospike logs
May 20 2015 03:41:56 GMT: INFO (demarshal): (thr_demarshal.c::297) warning: hit OS file descript limit (EMFILE on accept), consider raising limit May 20 2015 03:41:56 GMT: WARNING (cf:socket): (socket.c::274) socket: Too many open files May 20 2015 03:41:56 GMT: WARNING (cf:socket): (socket.c::274) socket: Too many open files May 20 2015 03:41:56 GMT: WARNING (cf:socket): (socket.c::274) socket: Too many open files
The above INFO message was changed to WARNING since 3.7.5:
Nov 14 2017 00:54:57 GMT: WARNING (demarshal): (thr_demarshal.c:454) Hit OS file descriptor limit (EMFILE on accept). Consider raising limit for uid 0
Note: always ensure that proto-fd-max configuration setting has a value that is less than the system/process limit. Depending on the size of cluster and configuration, it should be hundreds to thousands smaller. Please use this command to verify the open files for asd:
lsof -p `pgrep asd`
For the limits set at the process:
cat /proc/`pgrep asd`/limits
If there are not enough file descriptors, asd will crash:
Nov 14 2017 00:55:36 GMT: WARNING (socket): (socket.c:1616) Error while creating netlink socket: 24 (Too many open files) Nov 14 2017 00:55:36 GMT: FAILED ASSERTION (socket): (socket.c:2050) Error while enumerating network links Nov 14 2017 00:55:36 GMT: WARNING (as): (signal.c:210) SIGUSR1 received, aborting Aerospike Enterprise Edition build 188.8.131.52 os el6