Auto-pin cpu and multiple NICs

Operations Configuration
1

I’ve read the topic above and tried to add more NICs as advised.

I’m planning to use a r5d.24xlarge ec2 instance and every NIC has 8 queues.

$ ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 8This text will be hidden
TX: 8
Other: 0
Combined: 0
Current hardware settings:
RX: 8
TX: 8
Other: 0
Combined: 0

There are 96 CPUs and I attached 12 NICs. But I’m not sure how to configure them.

service {
        paxos-single-replica-limit 1 # Number of nodes where the replica count is automatically reduced to 1.
        proto-fd-max 15000
        auto-pin cpu
}
network {
        service {
                address eth0
                address eth1
                address eth2
                address eth3
                address eth4
                address eth5
                address eth6
                address eth7
                address eth8
                address eth9
                address eth10
                address eth11
                port 3000
        }
}

But I still got WARNING messages like this.

Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth0 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth1 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth2 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth3 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth4 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth5 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth6 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth7 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth8 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth9 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth10 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: WARNING (hardware): (hardware.c:1832) eth11 has very few NIC queues; only 8 out of 96 CPUs handle(s) NIC interrupts
Apr 19 2020 11:42:10 GMT: INFO (as): (as.c:316) <><><><><><><><><><>  Aerospike Community Edition build 4.7.0.12  <><><><><><><><><><>

How should I configure auto-pin cpu with multiple NICs?

2

Your NIC queues to CPU ratio needs to be at least 1 to 4. So you would need at least 3 NICs with 8 queues each to support 96 cores.

You may be able to configure Receive-Side Scaling (RSS) or similar on your NIC for more queues, see: 9.3. Configuration Tools Red Hat Enterprise Linux 7 | Red Hat Customer Portal.

Be sure to benchmark your hardware with these configurations.

3

Thank you for the reply.

Yes, you can see that I attached 12 NICs with 8 queues.

But I still got messages like this. The aerospike started but I’m not sure if it is in the proper status. I want to know what is the proper configuration for combination of auto-pin cpu and multiple NICs.

4

You need at least 24 queues, the RedHat page I linked to suggests 1 queue per core. You will need to benchmark your machine to see what works best.

5

Thank you for your reply but I’m a little bit confused. I did attach 12 NICs with 8 queues each, which means there are totally 96 queues and 96 cpu cores.

I supposed that the WARNING messages mean Aerospike doesn’t work with all of the NICs concurrently and properly. I need help to configure Aerospike to use all of the NICs concurrently with auto-pin cpu.