Container networking broken after upgrade to 2.6.18-308.8.2.el5.028stab101.1, stange kernel errors
Hi there, not sure if this should be a bug report, or if there is a strange configuraiton issue on my server. Running a CentOS 5.9 x86_64 server, recelntly rebooted into the 2.6.18-308.8.2.el5.028stab101.1 kernel last weekend. Upon booting into the new kernel I saw many kernel CPT ERR's while starting the containers: " Jan 20:46:08 vz02 last message repeated 8 times Jan 26 20:46:12 vz02 kernel: CT: 104: started Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :open_listening_socket: sock_create_kern: -97 Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :rst_sockets: open_listening_socket: -97 Jan 26 20:46:14 vz02 kernel: CPT ERR: ffff81031e46a000,104 :rst_sockets: -97 Jan 26 20:46:14 vz02 kernel: CT: 104: stopped Jan 26 20:46:15 vz02 kernel: CT: 104: started Jan 26 20:46:17 vz02 kernel: CT: 105: started Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :NLMERR: -22 Jan 26 20:46:19 vz02 last message repeated 8 times Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :open_listening_socket: sock_create_kern: -97 Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :rst_sockets: open_listening_socket: -97 Jan 26 20:46:19 vz02 kernel: CPT ERR: ffff81032156a000,105 :rst_sockets: -97 Jan 26 20:46:19 vz02 kernel: CT: 105: stopped Jan 26 20:46:21 vz02 kernel: CT: 105: started Jan 26 20:46:24 vz02 gconfd (root-8825): GConf server is not in use, shutting down. Jan 26 20:46:24 vz02 gconfd (root-8825): Exiting Jan 26 20:46:24 vz02 kernel: CT: 106: started Jan 26 20:46:26 vz02 kernel: CPT ERR: ffff8103185f3000,106 :NLMERR: -22 Jan 26 20:46:26 vz02 last message repeated 8 times Jan 26 20:46:26 vz02 kernel: CPT ERR: ffff8103185f3000,106 :open_listening_socket: sock_create_kern: -97 Jan 26 20:46:26 vz02 kernel: CPT ERR: ffff8103185f3000,106 :rst_sockets: open_listening_socket: -97 Jan 26 20:46:26 vz02 kernel: CPT ERR: ffff8103185f3000,106 :rst_sockets: -97 Jan 26 20:46:27 vz02 kernel: CT: 106: stopped Jan 26 20:46:28 vz02 kernel: CT: 106: started Jan 26 20:46:31 vz02 kernel: CT: 107: started Jan 26 20:46:35 vz02 kernel: CPT ERR: ffff810310d5b000,107 :open_listening_socket: sock_create_kern: -97 Jan 26 20:46:35 vz02 kernel: CPT ERR: ffff810310d5b000,107 :rst_sockets: open_listening_socket: -97 Jan 26 20:46:35 vz02 kernel: CPT ERR: ffff810310d5b000,107 :rst_sockets: -97 Jan 26 20:46:35 vz02 kernel: CT: 107: stopped Jan 26 20:46:36 vz02 kernel: CT: 107: started Jan 26 20:46:40 vz02 kernel: Can't load ipv6 module due to service vz is started Jan 26 20:46:40 vz02 modprobe: FATAL: Error inserting ipv6 (/lib/modules/2.6.18-308.8.2.el5.028stab101.1/kernel /net/ipv6/ipv6.ko): Device or resource busy Jan 26 20:46:42 vz02 kernel: CT: 108: started Jan 26 20:46:44 vz02 kernel: CPT ERR: ffff81030be0f000,108 :add ifaddr err -96 for 5 eth0 Jan 26 20:46:44 vz02 kernel: CPT ERR: ffff81030be0f000,108 :rst_restore_net: -96 Jan 26 20:46:46 vz02 kernel: CT: 108: stopped Jan 26 20:46:47 vz02 kernel: CT: 108: started Jan 26 20:46:51 vz02 kernel: CT: 111: started Jan 26 20:46:54 vz02 kernel: CPT ERR: ffff8103108ba000,111 :unknown interface 2 as0t0 Jan 26 20:46:54 vz02 kernel: CPT ERR: ffff8103108ba000,111 :unknown ifaddr 2 for 25 Jan 26 20:46:54 vz02 kernel: CPT ERR: ffff8103108ba000,111 :rst_restore_net: -22 Jan 26 20:46:54 vz02 kernel: CT: 111: stopped Jan 26 20:46:55 vz02 kernel: CT: 111: started Jan 26 20:46:57 vz02 kernel: CT: 201: started Jan 26 20:47:08 vz02 kernel: CPT ERR: ffff8103108ba000,201 :open_listening_socket: sock_create_kern: -97 Jan 26 20:47:08 vz02 kernel: CPT ERR: ffff8103108ba000,201 :rst_sockets: open_listening_socket: -97 Jan 26 20:47:08 vz02 kernel: CPT ERR: ffff8103108ba000,201 :rst_sockets: -97 Jan 26 20:47:08 vz02 kernel: CT: 201: stopped Jan 26 20:47:10 vz02 kernel: CT: 201: started Jan 26 20:47:21 vz02 kernel: CT: 203: started Jan 26 20:47:26 vz02 kernel: CT: 204: started Jan 26 20:47:28 vz02 kernel: CPT ERR: ffff8102fd325000,204 :open_listening_socket: sock_create_kern: -97 Jan 26 20:47:28 vz02 kernel: CPT ERR: ffff8102fd325000,204 :rst_sockets: open_listening_socket: -97 Jan 26 20:47:28 vz02 kernel: CPT ERR: ffff8102fd325000,204 :rst_sockets: -97 Jan 26 20:47:28 vz02 kernel: CT: 204: stopped Jan 26 20:47:30 vz02 kernel: CT: 204: started Jan 26 20:47:35 vz02 kernel: CT: 205: started Jan 26 20:47:53 vz02 kernel: CT: 206: started Jan 26 20:47:59 vz02 kernel: CPT ERR: ffff8102fd216000,206 :open_listening_socket: sock_create_kern: -97 Jan 26 20:47:59 vz02 kernel: CPT ERR: ffff8102fd216000,206 :rst_sockets: open_listening_socket: -97 Jan 26 20:47:59 vz02 kernel: CPT ERR: ffff8102fd216000,206 :rst_sockets: -97 Jan 26 20:47:59 vz02 kernel: CT: 206: stopped Jan 26 20:48:00 vz02 kernel: CT: 206: started " The containers started, but could not access the internet and I could not access them. After manually doing a vzctl restart on each container, networking started working again. Very strange! Saw different errors after vzctl restart: " Jan 26 21:06:33 vz02 kernel: proc: unrecognized mount option "relatime" or missing value Jan 26 21:06:36 vz02 kernel: CT: 103: stopped Jan 26 21:06:37 vz02 kernel: CT: 103: started Jan 26 21:06:51 vz02 kernel: proc: unrecognized mount option "relatime" or missing value Jan 26 21:06:53 vz02 kernel: CT: 104: stopped Jan 26 21:06:54 vz02 kernel: CT: 104: started Jan 26 21:07:16 vz02 kernel: proc: unrecognized mount option "relatime" or missing value Jan 26 21:07:18 vz02 kernel: CT: 107: stopped Jan 26 21:07:19 vz02 kernel: CT: 107: started Jan 26 21:07:59 vz02 kernel: proc: unrecognized mount option "relatime" or missing value Jan 26 21:08:01 vz02 kernel: CT: 111: stopped Jan 26 21:08:02 vz02 kernel: CT: 111: started " But networking started working... I've spent so many hours trying to figure this out, I have no idea where to look at this point. Running: vzctl-core-4.1.2-1 vzctl-4.1.2-1 Example ctid.conf, the x.x.x.x's are of course not in there: " ONBOOT="yes" NUMPROC="5102:5102" AVNUMPROC="2551:2551" NUMTCPSOCK="5102:5102" NUMOTHERSOCK="5102:5102" VMGUARPAGES="262144:9223372036854775807" # Secondary parameters KMEMSIZE="209012940:229914234" TCPSNDBUF="48773188:69670980" TCPRCVBUF="48773188:69670980" OTHERSOCKBUF="24386594:45284386" DGRAMRCVBUF="24386594:24386594" OOMGUARPAGES="151485:9223372036854775807" PRIVVMPAGES="256000:262140" # Auxiliary parameters LOCKEDPAGES="10205:10205" SHMPAGES="90891:90891" PHYSPAGES="0:9223372036854775807" NUMFILE="81632:81632" NUMFLOCK="1000:1100" NUMPTY="510:510" NUMSIGINFO="1024:1024" DCACHESIZE="45650516:47020032" NUMIPTENT="3072:3072" # Disk Resource Limits DISKINODES="4560000:4800000" DISKSPACE="39845888:41943040" # Quota Resource Limits QUOTATIME="0" QUOTAUGIDLIMIT="3000" # CPU Resource Limits CPUUNITS="1000" #RATE="eth0:1:6000" # IPTables config IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ip_conntrack ip_conntrack_ftp ipt_LOG ipt_conntrack ipt_helper ipt_state iptable_nat ip_nat_ftp ipt_TOS ipt_REDIRECT" IP_ADDRESS="x.x.x.x" HOSTNAME="x.x.x.x" VE_ROOT="/vz/root/$VEID" VE_PRIVATE="/vz/private/$VEID" OSTEMPLATE="centos-5-x86_64" ORIGIN_SAMPLE="512mb" NAMESERVER="x.x.x.x" NAME="x.x.x.x" Any help would be greatly appreciated! Thanks in advance.
Re: Container networking broken after upgrade to 2.6.18-308.8.2.el5.028stab101.1, stange kernel erro
Did you ever get a response to this? I'm seeing this today: Mar 16 10:47:34 MyVMName kernel: CPT ERR: ffff8101fe2c5000,252224 :NLMERR: -17 [root@MyVMName1 ~]# uname -a Linux MyVMName.someplace.com 2.6.18-371.3.1.el5.028stab110.1 #1 SMP Fri Dec 13 15:01:51 MSK 2013 x86_64 x86_64 x86_64 GNU/Linux My biggest complaint with OpenVZ is just the forums don't get a lot of traffic so when you find problems it's like a ghost town. CJ
Hi, Good to know I'm not the only person who experiences this. I ended up posting on OpenVZ's bugzilla and did indeed get a reply. I can't seem to find the bug now, it's been closed or removed maybe? It's not resolved for me, I have to restart the containers after a reboot by hand to get them online. Out of curiosity, how many network cards do you have on this server? I have many OpenVZ servers and this one has 3 physical network cards, all the others have 2 network cards and do not experience this issue. Could be totally unrelated...