OpenVZ Forum - RDF feed
type: channel
items: IBM RAID controller re-mount lock
items: Re: IBM RAID controller re-mount lock

IBM RAID controller re-mount lock
type: item
description: Dear OpenVZ community, We have a system from IBM (System x3400) with an IBM ServeRAID 8K-I controller on which we are running Debian-Lenny 4.1.2-25. The kernel is openvz-kernel 2.6.26-2-openvz-686. The containers are located on a separate ext3-partition and are mounted like this: /dev/drbd0 on /mnt/hq-drbd-stor01 type ext3 (rw) In particular, we experience a lockup when we try to un-mount and remount the container's partition. The following steps work: vzctl chkpnt 77 vzctl stop 77 /etc/init.d/openvz stop /etc/init.d/openvz start vzctl restore 77 But these don't work: vzctl chkpnt 77 vzctl stop 77 /etc/init.d/openvz stop umount /dev/drbd0 mount /dev/drbd0 /etc/init.d/openvz start vzctl restore 77 The restore produces the lockup. The output until it hangs: Restoring VE ... Starting VE ... VE is mounted undump... Adding IP address(es): 10.20.60.20 Setting CPU units: 1000 Configure meminfo: 65536 Nothing more is displayed after this. The host shows 100% CPU (system-)usage, load goes up to about 10. From our understanding, a container's processes run inside a kernel-process on the host. The container's currently active process (as far as we can tell, it's a different one on any try) takes all the CPU from the host. OpenVZ-version is 036test001. We already tried changing the RAID-controller driver in use from aacraid-1.1.5.2449 to aacraid-1.1.5.2467 but without any luck. Then we tried using USB-disks instead of the dev/drbd0 where the restore worked even after re-mounting, so the issue must lie within the RAID controller. We also tried to run a DRBD device on two USB-disks instead of using ServeRAID attached disks this constellation also works, so we can assume that DRDB is working correctly Is this a known issue? If yes, (how) can it be solved? Is there a way to debug this and provide you and us with more information? Thank you for your time., Maximilian Csuk ipcenter.at
creator: maximilian.csuk
date: 2009-08-05T14:35:44-00:00

Re: IBM RAID controller re-mount lock
type: item
description: Hello, - let's increase the verbose level of vzctl utility. In /etc/vz/vz.conf please set LOG_LEVEL and VERBOSE variables both to 10. - Quote: OpenVZ-version is 036test001. Sorry, I didn't catch this statement. - Quote: openvz-kernel 2.6.26-2-openvz-686. Do you have a chance to check the stable rhel5-2.6.18 or at least 2.6.18 kernels? I'm not sure that you do, but nevertheless ... http://download.openvz.org/kernel/ - Are there any suspicious messages in dmesg or in logs? - Quote: The host shows 100% CPU (system-)usage, load goes up to about 10. From our understanding, a container's processes run inside a kernel-process on the host. The container's currently active process (as far as we can tell, it's a different one on any try) takes all the CPU from the host. Do you have a chance to get the pid of the process that consumes 100% cpu? What is the output of "/proc/pid/wchan"? - It may be useful to gather alt-sysrq-* outputs: alt-sysrq- 1) "m" - for memory info dump 2) "p" - for registers - several times, please, twice the number of CPUs 3) "a" - for scheduler stat - 3 times 4) "w" - another scheduler info - 3 times 5) "t" - for all processes calltraces. Warning - this is a resource consuming operation. At least - twice.
creator: maratrus
date: 2009-08-06T09:00:33-00:00

Re: IBM RAID controller re-mount lock
type: item
description: Quote: let's increase the verbose level of vzctl utility. In /etc/vz/vz.conf please set LOG_LEVEL and VERBOSE variables both to 10. Done. Quote: Do you have a chance to get the pid of the process that consumes 100% cpu? What is the output of "/proc/pid/wchan"? Yes. The output is "0". Quote: Are there any suspicious messages in dmesg or in logs? I am not sure, but I guess not. Please have a look: /var/log/vzctl: 2009-08-07T09:40:01+0200 vzctl : VE 77 : VE start in progress... 2009-08-07T09:40:01+0200 vzctl : VE 77 : Restoring completed succesfully 2009-08-07T09:41:28+0200 vzctl : VE 77 : Setting up checkpoint... 2009-08-07T11:41:28+0400 vzctl : VE 77 : suspend... 2009-08-07T11:41:28+0400 vzctl : VE 77 : dump... 2009-08-07T11:41:29+0400 vzctl : VE 77 : kill... 2009-08-07T09:41:29+0200 vzctl : VE 77 : Running: /usr/lib/vzctl/scripts/vps-net_del 2009-08-07T09:41:29+0200 vzctl : VE 77 : Running: /usr/sbin/vzquota stat 77 -f 2009-08-07T09:41:29+0200 vzctl : VE 77 : Running: /usr/sbin/vzquota off 77 2009-08-07T09:41:29+0200 vzctl : VE 77 : VE is unmounted 2009-08-07T09:41:29+0200 vzctl : VE 77 : Checkpointing completed succesfully 2009-08-07T09:41:36+0200 vzctl : VE 77 : Restoring VE ... 2009-08-07T09:41:36+0200 vzctl : VE 77 : Starting VE ... 2009-08-07T09:41:36+0200 vzctl : VE 77 : Running: /usr/sbin/vzquota show 77 2009-08-07T09:41:36+0200 vzctl : VE 77 : Running: /usr/sbin/vzquota on 77 -r 0 -b 1048676 -B 1153124 -i 200100 -I 220100 -e 0 -n 0 -s 0 2009-08-07T09:41:36+0200 vzctl : VE 77 : Mounting root: /var/lib/vz/root/77 /var/lib/vz/private/77 2009-08-07T09:41:36+0200 vzctl : VE 77 : VE is mounted 2009-08-07T09:41:36+0200 vzctl : VE 77 : Adding IP address(es): 10.20.60.20 2009-08-07T09:41:36+0200 vzctl : VE 77 : Running: /usr/lib/vzctl/scripts/vps-net_add 2009-08-07T09:41:38+0200 vzctl : VE 77 : Setting CPU units: 1000 2009-08-07T09:41:38+0200 vzctl : VE 77 : Configure meminfo: 65536 2009-08-07T09:41:38+0200 vzctl : VE 77 : Running: /usr/sbin/vzquota stat 77 -f 2009-0…
creator: maximilian.csuk
date: 2009-08-07T09:07:26-00:00

Re: IBM RAID controller re-mount lock
type: item
description: Thank you very much for the provided information and sorry for delay. I'd recommend you to file a bug into bugzilla. http://bugzilla.openvz.org/
creator: maratrus
date: 2009-08-13T07:55:45-00:00