Kubernetes - Too many addresses for interface kube-ipvs0

Kubernetes: Too many addresses for interface kube-ipvs0 (versions prior to 4.9)

Problem Description

When a customer is running Aerospike version 4.8 or earlier in Kubernetes, the Aerospike server crashes with the following error Too many addresses for interface kube-ipvs0. The errors in the log will also show the following stack trace.

Mar 10 2020 17:02:09 GMT: FAILED ASSERTION (socket): (socket.c:2247) Too many addresses for interface kube-ipvs0
Mar 10 2020 17:02:09 GMT: WARNING (as): (signal.c:213) SIGUSR1 received, aborting Aerospike Enterprise Edition build 4.8.0.5 os debian9
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1178) stacktrace: registers: rax 0000000000000000 rbx 000000000000000a rcx ffffffffffffffff rdx 0000000000000000 rsi 00007ffe9a489640 rdi 0000000000000002 rbp 00007ffe9a489c70 rsp 00007ffe9a4896b8 r8 0000000000000000 r9 00007ffe9a489640 r10 0000000000000008 r11 0000000000000246 r12 0000000000000006 r13 0000000000000000 r14 0000000000000070 r15 000055a75b42d1c0 rip 00007f5f46fabfbf
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1192) stacktrace: found 17 frames: 0x3391bb 0x10fd8a 0x7f5f46fac0e0 0x7f5f46fabfbf 0x3371f7 0x34c04e 0x34a1d3 0x34b337 0x34ffa5 0x35043b 0x18e2cc 0x18e4a4 0x18f601 0x13cf50 0x81a3e 0x7f5f45d592e1 0x823da offset 0x55a7546d3000
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 0: asd(cf_fault_print_signal_context+0x10e) [0x55a754a0c1bb]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 1: asd(as_sig_handle_usr1+0x5c) [0x55a7547e2d8a]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 2: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7f5f46fac0e0]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 3: /lib/x86_64-linux-gnu/libpthread.so.0(raise+0xcf) [0x7f5f46fabfbf]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 4: asd(cf_fault_event+0x231) [0x55a754a0a1f7]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 5: asd(+0x34c04e) [0x55a754a1f04e]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 6: asd(+0x34a1d3) [0x55a754a1d1d3]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 7: asd(+0x34b337) [0x55a754a1e337]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 8: asd(cf_inter_is_inter_name+0x2a) [0x55a754a22fa5]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 9: asd(cf_ip_addr_is_dns_name+0x9) [0x55a754a2343b]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 10: asd(+0x18e2cc) [0x55a7548612cc]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 11: asd(+0x18e4a4) [0x55a7548614a4]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 12: asd(as_service_list_init+0x9b) [0x55a754862601]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 13: asd(as_info_init+0x21c9) [0x55a75480ff50]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 14: asd(main+0x38e) [0x55a754754a3e]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 15: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f5f45d592e1]
Mar 10 2020 17:02:09 GMT: WARNING (as): (fault.c:1200) stacktrace: frame 16: asd(_start+0x2a) [0x55a7547553da]
Mar 10 2020 17:02:09 GMT: FAILED ASSERTION (socket): (socket.c:2247) Too many addresses for interface kube-ipvs0

Explanation

This error will result when using IPVS as the kube-proxy mode and hostNetwork. When a ClusterIP type Service is created the IPVS proxier does the following 3 things.

  • Makes sure a dummy interface exists in the node, this defaults to kube-ipvs0.
  • Binds Service IP addresses to the dummy interface.
  • Creates IPVS virtual servers for each Service IP address respectively.

Aerospike will iterate through every interface that is visible to it. As kube-ipvs0 is the default, many other services will be using it and so many different addresses will be bound to it. This can exceed the address limit that Aerospike has per interface and generate the error above. In Aerospike server versions 4.8 and earlier, his limit is set to 20.

Solution

Fundamentally the issue lies with the usage of hostNetwork. Avoid the use of hostNetwork to prevent this issue. In version 4.9, the Aerospike server will ignore additional addresses for an interface once the limit (20) is reached. The following WARNING messages would be logged:

Mar 26 2020 19:48:27 GMT: WARNING (socket): (socket.c:2219) too many addresses for interface enp0s8 - truncating 1.2.1.21
Mar 26 2020 19:48:27 GMT: WARNING (socket): (socket.c:2219) too many addresses for interface enp0s8 - truncating 1.2.1.22
Mar 26 2020 19:48:27 GMT: WARNING (socket): (socket.c:2219) too many addresses for interface enp0s8 - truncating 1.2.1.21

Keywords

KUBERNETES HOSTNETWORK ADDRESSES IPVC

Timestamp

March 2020

© 2015 Copyright Aerospike, Inc. | All rights reserved. Creators of the Aerospike Database.