OpenVZ Forum


Home » General » Support » OpenVZ 6.3 - No VPS have Connectivity until "service network restart"
OpenVZ 6.3 - No VPS have Connectivity until "service network restart" [message #48758] Sun, 18 November 2012 00:53 Go to next message
MikeDVB is currently offline  MikeDVB
Messages: 12
Registered: April 2010
Junior Member
I just set up a brand new OpenVZ 6.3 server, disabled SELINUX, and migrated a single VPS over to the node using vzmigrate. That went fine, but once the VPS was migrated over the VPS has no connectivity to the outside world.

I can ping the VPS' IP from inside the node, and from another server on the same switch, but I cannot reach it from outside of our network nor can the VPS reach anything but the node itself. It can't ping google, or even a direct IP.

If I run 'service network restart' on the node, the VPS will suddenly have connectivity. This is without making *any* changes to *anything* after the reboot and before doing 'service network restart'. If I do 'service network restart' and the VPS has connectivity - on reboot it no longer has connectivity until I restart the network manually again.
# service network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down interface eth1:                              [  OK  ]
Shutting down interface venet0:  Shutting down interface venet0:
                                                           [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:                                [  OK  ]
Bringing up interface eth1:                                [  OK  ]
Bringing up interface venet0:  Bringing up interface venet0:
Configuring interface venet0:
net.ipv4.conf.venet0.send_redirects = 0
Configuring ipv6 venet0:
                                                           [  OK  ]


I do see from the boot.log that it is doing the network start-up using the same /etc/init.d/network script:
cat boot.log
                Welcome to CentOS
Starting udev:                                             [  OK  ]
Setting hostname redacted.redacted.com:                  [  OK  ]
Checking filesystems
/dev/sdb2: clean, 63294/182255616 files, 18157450/729021952 blocks
/dev/sda1: clean, 50/131072 files, 85436/523264 blocks
                                                           [  OK  ]
Remounting root filesystem in read-write mode:             [  OK  ]
Mounting local filesystems:                                [  OK  ]
Enabling /etc/fstab swaps:                                 [  OK  ]
Entering non-interactive startup
Calling the system activity data collector (sadc):
ip6tables: Applying firewall rules:                        [  OK  ]
iptables: Applying firewall rules:                         [  OK  ]
iptables: Loading additional modules: ipt_REJECT ipt_tos ipt_TOS ipt_LOG ip_conntrack ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ipt_state ip[  OK  ]t ip_nat_ftp ipt_owner ipt_REDIRECT
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:                                [  OK  ]
Bringing up interface eth1:                                [  OK  ]
Bringing up interface venet0:  Bringing up interface venet0:
Configuring interface venet0:
net.ipv4.conf.venet0.send_redirects = 0
Configuring ipv6 venet0:
                                                           [  OK  ]
Starting auditd:                                           [  OK  ]
Starting portreserve:                                      [  OK  ]
Starting system logger:                                    [  OK  ]
Mounting other filesystems:                                [  OK  ]
Retrigger failed udev events                               [  OK  ]
Starting sshd:                                             [  OK  ]
Starting postfix:                                          [  OK  ]
Starting lighttpd:                                         [  OK  ]
Starting...
Starting...
Starting crond:                                            [  OK  ]
Checking vzevent kernel module ...                         [  OK  ]
Starting vzeventd:                                         [  OK  ]
Starting OpenVZ:                                           [  OK  ]
Applying OOM adjustments:                                  [  OK  ]
Starting CT 258:                                           [  OK  ]


The VPS is indeed running after the boot:
# vzlist -a
      CTID      NPROC STATUS    IP_ADDR         HOSTNAME
       258         36 running   re.da.ct.ed     redacted.redacted.com


ipv4.ip_forward is enabled:
# sysctl -e net.ipv4.ip_forward
net.ipv4.ip_forward = 1


I haven't touched the /etc/vz/vz.conf:
# cat /etc/vz/vz.conf
## Global parameters
VIRTUOZZO=yes
LOCKDIR=/vz/lock
DUMPDIR=/vz/dump
VE0CPUUNITS=1000

## Logging parameters
LOGGING=yes
LOGFILE=/var/log/vzctl.log
LOG_LEVEL=0
VERBOSE=0

## Disk quota parameters
DISK_QUOTA=yes
VZFASTBOOT=no

# Disable module loading. If set, vz initscript does not load any modules.
#MODULES_DISABLED=yes

# The name of the device whose IP address will be used as source IP for CT.
# By default automatically assigned.
#VE_ROUTE_SRC_DEV="eth0"

# Controls which interfaces to send ARP requests and modify APR tables on.
NEIGHBOUR_DEVS=all

## Fail if there is another machine in the network with the same IP
ERROR_ON_ARPFAIL="no"

## Template parameters
TEMPLATE=/vz/template

## Defaults for containers
VE_ROOT=/vz/root/$VEID
VE_PRIVATE=/vz/private/$VEID
CONFIGFILE="basic"
DEF_OSTEMPLATE="centos-5"

## Load vzwdog module
VZWDOG="no"

## IPv4 iptables kernel modules to be enabled in CTs by default
IPTABLES="ipt_REJECT ipt_tos ipt_TOS ipt_LOG ip_conntrack ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ipt_state iptable_nat ip_nat_ftp ipt_owner ipt_REDIRECT"
## IPv4 iptables kernel modules to be loaded by init.d/vz script
IPTABLES_MODULES="$IPTABLES"

## Enable IPv6
IPV6="yes"

## IPv6 ip6tables kernel modules
IP6TABLES="ip6_tables ip6table_filter ip6table_mangle ip6t_REJECT"


I flushed iptables rules on the main node, which didn't help/solve the issue.

I'm really at a loss as to what would cause this. I thought it may be the arp table, but if that were the case - a reboot (takes less than 2 minutes) should not cause failures - especially since the ARP entries don't time out that fast on our network

Traceroutes from external sources and other servers show that the traffic is hitting the node before timing out:
# traceroute re.da.ct.ed
traceroute to re.da.ct.ed (re.da.ct.ed), 30 hops max, 40 byte packets
 1  the-node.redacted.com (the-node.primary-ip.com)  0.150 ms  0.133 ms  0.121 ms
 2  * * *
 3  * * * 
etc...


The VPS can ping the node's IP, and the node can ping the VPS' IP before and after 'service network restart'.

[Updated on: Sun, 18 November 2012 01:10]

Report message to a moderator

Re: OpenVZ 6.3 - No VPS have Connectivity until "service network restart" [message #48817 is a reply to message #48758] Sun, 09 December 2012 05:51 Go to previous messageGo to next message
MikeDVB is currently offline  MikeDVB
Messages: 12
Registered: April 2010
Junior Member
Still haven't gotten anywhere with this.

Installed a new server with CentOS 6.3, installed the OpenVZ Kernel, downloaded the CentOS 6 x86_64 template from openvz.org and then proceeded to create a brand new VPS from the command line.

The behavior is exactly the same - the VPS has no connectivity until I do "service network restart" on the node.

I'm starting to think that, perhaps, OpenVZ 6 isn't compatible with this hardware. It's an older server, but not extremely old - Dual Intel 5620, 24gb ram.
Re: OpenVZ 6.3 - No VPS have Connectivity until "service network restart" [message #48818 is a reply to message #48817] Sun, 09 December 2012 09:03 Go to previous messageGo to next message
maka
Messages: 3
Registered: December 2012
Junior Member
Hi!

What network card do you have/use?

I have had similar problems as you describe. This was because my NIC was a slow starter and wasn't fully initialized when my containers was started.

Run dmesg and see if your NIC is ready ("eth0: Link is up...") before or after your containers are started ("CT: xxx: started").
Re: OpenVZ 6.3 - No VPS have Connectivity until "service network restart" [message #48819 is a reply to message #48758] Sun, 09 December 2012 18:03 Go to previous messageGo to next message
MikeDVB is currently offline  MikeDVB
Messages: 12
Registered: April 2010
Junior Member
I installed OpenVZ 6.3 on newer hardware - Dual e2620 - Linux redacted 2.6.32-042stab065.3 #1 SMP Mon Nov 12 21:59:14 MSK 2012 x86_64 x86_64 x86_64 GNU/Linux

It's just a vanilla CentOS 6.3 minimal with vzkernel-2.6.32-042stab065.3 installed. I haven't customized /etc/vz/vz.conf or anything else on the system at this point beyond installing some things like screen via yum.

When I migrate in an existing VPS from another server, or I create a brand new VPS right on the server - the VPS has no network connectivity in and out to the world until I do "service network restart" on the node.

If I reboot, I have to wait for OpenVZ to fully start before doing service network restart to get them all working.

I'm really not sure where to troubleshoot this further - I know it's not a 'slow to initialize networking card' because this exact server was running OpenVZ5 for over a year previously with no issues.

I have experienced this issue on this hardware, and other hardware, on the latest version of OpenVZ5 as well as OpenVZ6.

If I restart a VPS, it has no network connectivity until 'service network restart' happens on the node itself as well. I can ping the IP of the VPS from the node, I can ping the node from the VPS.

The VPS just can't seem to reach the internet and the internet can't seem to reach the VPS until I restart networking on the host.

[Updated on: Mon, 10 December 2012 11:00]

Report message to a moderator

Re: OpenVZ 6.3 - No VPS have Connectivity until "service network restart" [message #48821 is a reply to message #48758] Mon, 10 December 2012 13:06 Go to previous messageGo to next message
MikeDVB is currently offline  MikeDVB
Messages: 12
Registered: April 2010
Junior Member
No Message Body

[Updated on: Mon, 10 December 2012 16:05]

Report message to a moderator

Re: OpenVZ 6.3 - No VPS have Connectivity until "service network restart" [message #48822 is a reply to message #48758] Mon, 10 December 2012 19:29 Go to previous messageGo to next message
Ales is currently offline  Ales
Messages: 330
Registered: May 2009
Senior Member
If you closely inspect the network state a) after the reboot b) after the network restart, are there any differences at all?

Routing table perhaps? Or iptables rules?

There must be something Confused
Re: OpenVZ 6.3 - No VPS have Connectivity until "service network restart" [message #48823 is a reply to message #48758] Tue, 11 December 2012 01:41 Go to previous messageGo to next message
MikeDVB is currently offline  MikeDVB
Messages: 12
Registered: April 2010
Junior Member
I looked at everything I could, but what eventually gave it away for me was pulling the ARP entries from the core. The secondary NIC was ARPing for IPs along with the primary nic resulting in the network routing public packets to the private interface.

We're working on upgrading some networking gear and, in the meantime, our private network which is used for backups is on the same vlan. Once we get our new gear installed our private ports will have their own vlan and the possibility of this issue wouldn't even exist.

It turned out not to be vzctl at all.

The thing is, we were aware of this possibility and had set these accordingly:
net.ipv4.conf.eth0.arp_ignore
net.ipv4.conf.eth1.arp_ignore

They work on OpenVZ5 but don't seem to have effect on OpenVZ6.

In the meantime, we just racked up an extra switch to provide internal networking for backups and other tasks.
Re: OpenVZ 6.3 - No VPS have Connectivity until "service network restart" [message #48825 is a reply to message #48758] Tue, 11 December 2012 17:23 Go to previous message
Ales is currently offline  Ales
Messages: 330
Registered: May 2009
Senior Member
Good catch. It's also very valuable to have these kind of borderline cases, rhel5 / rhel6 differences, etc. documented either here or in the wiki, thanks for reporting.
Previous Topic: call trace on 2.6.32-042stab065.3
Next Topic: IP_conntrack limit per VPS
Goto Forum:
  


Current Time: Wed Jul 17 06:27:52 GMT 2024

Total time taken to generate the page: 0.02887 seconds