OpenVZ Forum: Support » Container stall leads to server hang

Home » General » Support » Container stall leads to server hang

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Container stall leads to server hang [message #47544]

Wed, 15 August 2012 16:59

greg
Messages: 13
Registered: March 2012

Junior Member

I just had a strange problem:

- a container was working (answering web requests)
- trying to connect to this container via ssh or "vzctl enter" leads the connection to hang
- trying to restart the container leads to a total hang of the server (not even a ping)

I cannot find anything in the logs

Any idea?

TIA

greg

Report message to a moderator

Re: Container stall leads to server hang [message #47571 is a reply to message #47544]

Thu, 16 August 2012 15:18

Ales
Messages: 330
Registered: May 2009

Senior Member

None whatsoever, I'm afraid. Still, to help anyone else answer you, I'd strongly recommend following the advice written in red on the first page of the forum - please report your kernel and tools version.

Please add any additional details you think might help, such as your networking setup, OS and OS version (on hardware node and on the virtual machine) and any oddities you might think of...

Seems like an odd problem, though. Can you reproduce it? Does it happen every time or does it seem random?

Report message to a moderator

Re: Container stall leads to server hang [message #47582 is a reply to message #47571]

Fri, 17 August 2012 01:43

greg
Messages: 13
Registered: March 2012

Junior Member

Thanks for your reply and sorry I didn't see the rules.

It's the first time I see this problem (I've been using vservers/vz for years) ; after a reboot, the problem is gone.

I found this link, but I cannot know if it's really relevant:

forum.soluslabs.com /archive/index.php/t-238.html

Report message to a moderator

Re: Container stall leads to server hang [message #48726 is a reply to message #47544]

Sun, 11 November 2012 03:37

seanfulton
Messages: 105
Registered: May 2007

Senior Member

I had a problem like this once. I can't for the life of me remember what caused it, but it was with the rhel5 kernel and was years ago. Happily, it hasn't happened since.

What kernel are you using?

In retrospect, I believe I may have over-committed the HN and so locking up the VE was actually locking up the node itself. So imagine have a 24G HN and 30G assigned the VEs, then maxing them all out. The last one to max would hang the machine. I'm thinking this may have been what happened in my case, but I don't think at the time I knew what the problem was.

Check your configs and make sure you are not over-committing RAM. Which is a good reason to run the rhel6 kernel (much, much simpler RAM control).

If you are using the RHEL5 kernel let me know, I have some ideas on that.

sean

Report message to a moderator

Re: Container stall leads to server hang [message #48730 is a reply to message #48726]

Mon, 12 November 2012 13:27

greg
Messages: 13
Registered: March 2012

Junior Member

Thanks for your answer. The problem happened only once, monthes ago, but there are a few things I'm sure of:

- I don't use RHEL; host is a pre-installed debian+openvz, guests are either fedora or ubuntu
- I don't think I over-allocated memory; the summary says: Physical Memory (1.94GB/662MB)

The total allocated max memory is 671+235+343 which is about 1.2GB. How much should I leave for the host system?

TIA

greg

Report message to a moderator

Re: Container stall leads to server hang [message #48735 is a reply to message #48730]

Mon, 12 November 2012 21:47

seanfulton
Messages: 105
Registered: May 2007

Senior Member

What kernel version?

I'm not sure which summary that data is from. Can you do free -m on the host node?

sean

Report message to a moderator

Re: Container stall leads to server hang [message #48741 is a reply to message #48735]

Tue, 13 November 2012 14:44

greg
Messages: 13
Registered: March 2012

Junior Member

uname -a
Linux ****** 2.6.32-6-pve #1 SMP Mon Jan 23 08:27:52 CET 2012 x86_64 GNU/Linux

free -m
total used free shared buffers cached
Mem: 1987 1909 77 0 346 895
-/+ buffers/cache: 668 1318
Swap: 0 0 0

I'm not sure it was the same version when the problem occurred, I may have updated since.

Now I notice there is no swap?!?

Report message to a moderator

Re: Container stall leads to server hang [message #48742 is a reply to message #48741]

Tue, 13 November 2012 14:56

seanfulton
Messages: 105
Registered: May 2007

Senior Member

The lack of swap could be the problem, especially with limited RAM like you have. With the 2.6.32 kernel you should check the SWAPPAGES variable in /etc/vz/conf/*.conf and see if you told any of your VEs to swap. That could do it, or it could just be that the HN ran out of RAM and couldn't swap.

Easy way to add swap (and keep you from getting hung up again) is to create a swap file have the system swap to it. Here is a how-to. Just to be safe, if you have the room, make your swap extra big. The system will get really, really slow when it starts to swap a lot, so you will know it.

http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

uses
dd if=/dev/zero of=/swapfile1 bs=1024 count=524288

for a 512M swap file. I would do count=2097152 for a 2G swap, or if you have the room,
count=4194304

Like I said, the system will get really, really slow when it starts to swap so you know know you are having a problem.

sean

Report message to a moderator

Re: Container stall leads to server hang [message #48743 is a reply to message #48742]

Tue, 13 November 2012 15:05

greg
Messages: 13
Registered: March 2012

Junior Member

Thanks for the tip.

The 2G-memory system was pre-configured with a 512M swap partion, but not set up in /etc/fstab, I just asked the technical support if they did it on purpose.

Is there a clean way to get a warning when the memory overflows to swap? I can cron+grep but I'm not sure it would be reliable...

Report message to a moderator

Re: Container stall leads to server hang [message #48744 is a reply to message #48743]

Tue, 13 November 2012 15:27

seanfulton
Messages: 105
Registered: May 2007

Senior Member

FREE=`free -m | grep Mem | awk '{print $4}'`
if [ $FREE -lt 256 ] ; then
echo "ONLY $FREE RAM LEFT ON $HN" | mail -s "RAM SHORTAGE ON $HN" you@you.com
fi

Should work fine if you run it from cron every 5 minutes or so.

Or you could try munin (http://munin-monitoring.org/) which is a great way to monitor resource usage. Setting up alerts in munin is a bit complicated though.

sean

Report message to a moderator

Previous Topic:	CT starts in vz init, even if stopped before and disabled.
Next Topic:	Gentoo template creation - stat(/vz/root/1001): No such file or directory

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun May 12 13:46:01 GMT 2024

Total time taken to generate the page: 0.01459 seconds