What is relationship between NIC queues and CPU cores in auto-pin?

FAQ - What is the relationship between NIC queues and CPU cores when configuring auto-pin

Detail

Why did the Aerospike server fail to start with the following message:

WARNING (hardware): (hardware.c:1605) eth0 has very few NIC queues; only 8 out of 32 CPUs handle(s) NIC interrupts

Answer

Depending on the hardware and its configuration, the Network Interface Card (NIC) can have multiple queues which can be verified with the ethtool -l command.

The optimal configuration is that the number of CPU handling interrupts should be the same as the number of NIC queues as mentioned in the Achieving Data Locality with auto-pin article.

The main doc reference auto-pin has the following under the additional information section:

“It is also necessary for the ratio of NIC queues to CPU cores be greater than 1/4. The following message would otherwise be logged on the console and the server would not start…”

To confirm the auto-pin settings, check the output of the following system files before and after auto-pin is enabled:

  • CPU affinity interrupts are configured under:
    • /proc/irq/<irq-number>/smp_affinity
  • RPS is configured under:
    • /sys/class/net/<if-name>/queues/rx-<queue-index>/rps_cpus
  • RFS is configured under:
    • /proc/sys/net/core/rps_sock_flow_entries
    • /sys/class/net/<if-name>/queues/rx-<queue-index>/rps_flow_cnt
  • XPS is configured under:
    • /sys/class/net/<if-name>/queues/tx-<queue-index>/xps_cpus

Notes

  • It may be better to add more NICs if there are more CPU cores than network queues.
  • It is recommended to always test and benchmark the results before and after auto-pin is enabled.
  • More details on irqbalance_ban.sh script can be found on the following FAQ.

Keywords

AUTO-PIN NIC QUEUES CPU CORES NUMA

Timestamp

June 17 2019