| *SOLVED* VZ fails to start cleanly on reboot [message #9518] |
Mon, 08 January 2007 15:06  |
tchipman
Messages: 28
Registered: June 2006
|
Junior Member |
|
|
I'm curious if anyone else has seen this behaviour, which has hit me twice now in the past ~6 weeks (power failues, yay)
-machine shuts itself down gracefully via UPS monitoring software
-when power is restored, machine boots up automatically
-machine boots normally, but OpenVZ virtual hosts fail to come online
-when I connect to machine, I can then manually use
"/etc/init.d/vz restart" and vzhosts come up cleanly this time, no problems at all.
-machine was patched to latest OpenVZ using OpenVZ YUM repos last month. See below for exact details on machine,platform,versions and logged errors.
Many thanks for any assistance,
--Tim
-------------------
MACHINE INFO:
=============
-AMD AthlonX2 2000mhz machine running CentOS 32-bit
-using stock OpenVZ kernel installed from Yum Repo from OpenVZ:
-vzctl version is 3.0.13
[root@hydra log]# uname -a
Linux hydra......ca 2.6.9-023stab037.3-smp #1 SMP Wed Dec 13 19:33:33 MSK 2006 i686 athlon i386 GNU/Linux
[root@hydra log]# vzctl
vzctl version 3.0.13
Copyright (C) 2000-2006 SWsoft.
...
SYMPTOM AFTER FIRST CONNECT AND THEN RESTART:
=============================================
--paste--
[root@hydra ~]# vzlist
VE not found
[root@hydra ~]# cd /etc/init.d
[root@hydra init.d]# ./vz status
OpenVZ is running...
[root@hydra init.d]# ./vz restart
Stopping OpenVZ: [ OK ]
Starting OpenVZ: [ OK ]
Bringing up interface venet0: [ OK ]
Configuring interface venet0: [ OK ]
Starting VE 236: [ OK ]
Starting VE 237: [ OK ]
Starting VE 238: [ OK ]
Starting VE 239: [ OK ]
[root@hydra init.d]# vzlist
VEID NPROC STATUS IP_ADDR HOSTNAME
236 36 running 129.X.23.236 polyp.phys.....ca
237 7 running 129.X.23.237 urchin.phys.....ca
238 16 running 129.X.23.238 seahorse.phys....ca
239 11 running 129.X.23.239 nema.phys.....ca
[root@hydra init.d]#
--endpaste--
OUTPUT CAPTURED IN /etc/log/vz.log:
[next bit is for controlled shutdown...]
2007-01-07T10:28:16-0400 vzctl : VE 237 : Stopping VE ...
2007-01-07T10:28:17-0400 vzctl : VE 236 : Setting CPU limit: 0
2007-01-07T10:28:17-0400 vzctl : VE 236 : Setting CPU units: 2000
2007-01-07T10:28:17-0400 vzctl : VE 236 : WARNING: Settings were not saved and will be resetted to original values on next start (use --save flag)
2007-01-07T10:28:17-0400 vzctl : VE 236 : Stopping VE ...
2007-01-07T10:28:22-0400 vzctl : VE 237 : VE was stopped
2007-01-07T10:28:22-0400 vzctl : VE 238 : VE was stopped
2007-01-07T10:28:22-0400 vzctl : VE 238 : VE is unmounted
2007-01-07T10:28:22-0400 vzctl : VE 237 : VE is unmounted
2007-01-07T10:28:24-0400 vzctl : VE 239 : VE was stopped
2007-01-07T10:28:24-0400 vzctl : VE 239 : VE is unmounted
2007-01-07T10:28:25-0400 vzctl : VE 236 : VE was stopped
2007-01-07T10:28:25-0400 vzctl : VE 236 : VE is unmounted
[power is restored and system is coming up]
2007-01-07T11:45:01-0400 venetclean : VE 236 : VE died, clear IPs:
2007-01-07T11:45:01-0400 venetclean : VE 237 : VE died, clear IPs:
2007-01-07T11:45:02-0400 venetclean : VE 238 : VE died, clear IPs:
2007-01-07T11:45:02-0400 venetclean : VE 239 : VE died, clear IPs:
2007-01-08T09:02:53-0400 vzctl : Setting CPU units: 1000
2007-01-08T09:02:53-0400 vzctl : WARNING: Settings were not saved and will be resetted to original values on next start (use --save flag)
[manually start via /etc/init.d/vz restart command:]
2007-01-08T09:02:53-0400 vzctl : VE 236 : Starting VE ...
..etc... all good...
[Updated on: Mon, 05 February 2007 10:54] by Moderator Report message to a moderator |
|
|
|
|
|
| Re: VZ fails to start cleanly on reboot [message #9553 is a reply to message #9534] |
Tue, 09 January 2007 14:08  |
tchipman
Messages: 28
Registered: June 2006
|
Junior Member |
|
|
Hi,
Yes - this is consistent behaviour. we had another power fail last night (insane!) and the behaviour is identical. Virtual hosts didn't auto-start, but login as root I was able to issue a "restart" with the init script which brought up the virtual hosts just fine.
I'm trolling the logfile as per your request, and I have trouble finding anything which cites specifically OpenVZ. best hits I've got are shown below:
Jan 8 18:25:57 hydra kernel: Virtuozzo Fair CPU scheduler
Jan 8 18:25:57 hydra kernel: Built 1 zonelists
I can't see anything in my /var/log/boot.log either.
The best thing I can find, as per my original post, is logged in /var/log/vzctl.log, where I see:
CONTROLLED SHUTDOWN UNDERWAY - ENDING:
2007-01-08T18:06:52-0400 vzctl : VE 236 : VE is unmounted
POWER IS RESTORED, MACHINE COMES UP ON ITS OWN:
2007-01-08T18:30:01-0400 venetclean : VE 236 : VE died, clear IPs:
2007-01-08T18:30:01-0400 venetclean : VE 237 : VE died, clear IPs:
2007-01-08T18:30:01-0400 venetclean : VE 238 : VE died, clear IPs:
2007-01-08T18:30:01-0400 venetclean : VE 239 : VE died, clear IPs:
I CONNECT AS ROOT REMOTELY AND RESTART VZ SERVICE:
2007-01-08T18:55:42-0400 vzctl : Setting CPU units: 1000
2007-01-08T18:55:42-0400 vzctl : WARNING: Settings were not saved and will be resetted to original values on next start (use --save flag)
2007-01-08T18:55:42-0400 vzctl : VE 236 : Starting VE ...
2007-01-08T18:55:42-0400 vzctl : VE 236 : VE is mounted
2007-01-08T18:55:42-0400 vzctl : VE 236 : Adding IP address(es): 129.XXX.XXX.236
2007-01-08T18:55:46-0400 vzctl : VE 236 : Setting CPU units: 22327
2007-01-08T18:55:46-0400 vzctl : VE 236 : Set hostname: polyp.phys.XX.XY.ca
2007-01-08T18:55:46-0400 vzctl : VE 236 : File resolv.conf was modified
2007-01-08T18:55:46-0400 vzctl : VE 236 : VE start in progress...
OTHER VZHOSTS COME UP FINE AS PER THIS ONE SHOWN ABOVE (there are 5 vzhosts on this machine in total)
It appears from the vzctl.log that an attempt is made to bring up all of the virtual hosts, but then errors result ending with "VE Died Clear IPs" for each host, one after the other...
If you can make suggestions about other logging I should be able to observe, it would be appreciated. Hopefully (?) what is above is of some use (?)
Thanks for your help with this,
Tim Chipman
|
|
|
|
| Re: VZ fails to start cleanly on reboot [message #9997 is a reply to message #9553] |
Sun, 04 February 2007 01:19 |
tchipman
Messages: 28
Registered: June 2006
|
Junior Member |
|
|
Hi,
Just wanted to update this thread, since things have evolved recently somewhat.
I just rolled yesterday a yum-update on the machine in question, which pulled automagically from the OpenVZ Yum Repo the newest (Jan-07) offical kernel build, (2.6.9-023stab040.1 I believe)
After this yum update and a reboot using the new kernel, I was very happy to see that all my virtual hosts started correctly while the physical host was booting up.
So - I'm still not precisely certain what the problem was, but it seems to be resolved for now.
--Tim Chipman
|
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
