OpenVZ Forum: Support » HN hangs with no reason. Memory shortage?

Home » General » Support » HN hangs with no reason. Memory shortage?

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

HN hangs with no reason. Memory shortage? [message #37365]

Fri, 04 September 2009 15:42

piplite
Messages: 27
Registered: March 2008

Junior Member

So i have a host node running 20 VEs.
From time to time HN just hangs, no ping, no ssh response. i have to reboot it by powercycling to bring it back to work.
Its 2.6.18-128.2.1.el5.028stab064.4PAE running centos 5.2

and as you can see from /var/log/messages from 10.49 to 11.25 (reboot time) server doesnt respond.

Where do i look? I have same thing happening from time to time with 2-3 of my servers.

Sep  4 10:31:05 perun kernel: Fatal resource shortage: privvmpages, UB 6640.
Sep  4 10:31:05 perun kernel: Fatal resource shortage: privvmpages, UB 6640.
Sep  4 10:32:59 perun kernel: CT: 6830: stopped
Sep  4 10:33:23 perun kernel: CT: 6830: started
Sep  4 10:49:03 perun kernel: CT: 6830: stopped
Sep  4 10:49:12 perun kernel: CT: 6830: started
Sep  4 11:02:52 perun kernel: CT: 6830: stopped
Sep  4 11:03:23 perun kernel: CT: 6830: started
Sep  4 11:25:05 perun syslogd 1.4.1: restart.
Sep  4 11:25:05 perun kernel: klogd 1.4.1, log source = /proc/kmsg started.
Sep  4 11:25:05 perun kernel: Linux version 2.6.18-128.2.1.el5.028stab064.4PAE (root@rh5-build-x64) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jul 22 00:38:32 MSD 2009
Sep  4 11:25:05 perun kernel: BIOS-provided physical RAM map:
Sep  4 11:25:05 perun kernel: **********************************************************

Report message to a moderator

Re: HN hangs with no reason. Memory shortage? [message #37366 is a reply to message #37365]

Fri, 04 September 2009 15:45

kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia

Senior Member

Show us your vzmemcheck -v output please.

Kir Kolyshkin

http://static.openvz.org/userbars/openvz-developer.png

Report message to a moderator

Re: HN hangs with no reason. Memory shortage? [message #37368 is a reply to message #37366]

Fri, 04 September 2009 15:47

piplite
Messages: 27
Registered: March 2008

Junior Member

Here its:

# vzmemcheck -v
Output values in %
veid        LowMem  LowMem     RAM MemSwap MemSwap   Alloc   Alloc   Alloc
              util  commit    util    util  commit    util  commit   limit
6880          0.75  852.96    0.16    0.10   22.68    0.15   22.68   23.19
6860          0.59  852.96    0.28    0.17   22.68    0.29   22.68   23.19
6840          0.61  852.96    0.31    0.19   22.68    0.62   22.68   23.19
6820          0.16  852.96    0.03    0.02   22.68    0.02   22.68   23.19
6810          0.40  852.96    0.05    0.03   22.68    0.07   22.68   23.19
6800          0.54  852.96    0.20    0.12   22.68    0.25   22.68   23.19
6790          0.79  852.96    0.19    0.12   22.68    0.17   22.68   23.19
6780          1.85  852.96    0.55    0.33   22.68    0.61   22.68   23.19
6770          0.19  852.96    0.03    0.02   22.68    0.03   22.68   23.19
6740          0.31  852.96    0.08    0.05   22.68    0.07   22.68   23.19
6690          0.16  852.96    0.03    0.02   22.68    0.02   22.68   23.19
6650          0.35  852.96    1.32    0.80   22.68    0.93   22.68   23.19
6640          0.42  852.96    0.23    0.14   22.68    0.50   22.68   23.19
6590          0.32  852.96    0.06    0.04   22.68    0.05   22.68   23.19
6570          0.37  852.96    0.06    0.04   22.68    0.05   22.68   23.19
6530          0.37  852.96    0.05    0.03   22.68    0.08   22.68   23.19
6520          0.84  868.59    0.45    0.27   23.59    0.47   23.59   24.61
6450          0.18  852.96    0.03    0.02   22.68    0.03   22.68   23.19
6420          0.53  852.96    0.20    0.12   22.68    0.25   22.68   23.19
2690          0.28  852.96    0.07    0.04   22.68    0.21   22.68   23.19
6400          0.41  868.59    0.17    0.11   23.59    0.35   23.59   24.61
5890          0.30  899.86    0.08    0.05   25.42    0.07   25.42   27.45
4250          0.46  884.23    0.21    0.12   24.51    0.17   24.51   26.03
4090          0.31  868.59    0.08    0.05   23.59    0.07   23.59   24.61
4060          0.21  868.59    0.03    0.02   23.59    0.03   23.59   24.61
3970          0.20  899.86    0.04    0.02   25.42    0.04   25.42   27.45
3780          0.59  868.59    0.26    0.15   23.59    0.30   23.59   24.61
3760          0.54  868.59    0.21    0.12   23.59    0.25   23.59   24.61
3750          0.66  962.39    0.24    0.14   29.08    0.22   29.08   33.14
3740          0.94  868.59    1.13    0.68   23.59    1.24   23.59   24.61
3500          1.42  899.86    1.69    1.02   25.42    1.30   25.42   27.45
3450          0.95  868.59    0.23    0.14   23.59    0.20   23.59   24.61
2770          0.25  852.96    0.04    0.03   22.68    0.04   22.68   23.19
2630          0.54  868.59    0.20    0.12   23.59    0.25   23.59   24.61
2170          0.45  852.96    0.14    0.08   22.68    0.15   22.68   23.19
2160          0.98  884.23    0.92    0.56   24.51    0.69   24.51   26.03
2120          0.57  868.59    0.32    0.20   23.59    0.95   23.59   23.85
2090          1.33  852.96    0.53    0.32   22.68    0.54   22.68   23.19
2070          0.28  852.96    0.11    0.07   22.68    0.15   22.68   23.19
1980          0.27  868.59    0.07    0.04   23.59    0.06   23.59   24.61
1960          0.85  868.59    0.64    0.39   23.59    0.56   23.59   24.61
1950          0.19  845.14    0.03    0.02   22.22    0.03   22.22   22.48
1930          0.39  852.96    0.05    0.03   22.68    0.08   22.68   23.19
1920          0.49  884.23    0.48    0.29   24.51    0.35   24.51   26.03
1900          0.99  868.59    0.49    0.29   23.59    0.75   23.59   24.61
1890          0.99  868.59    0.97    0.58   23.59    0.62   23.59   24.61
1850          0.68  899.86    0.42    0.25   25.42    0.83   25.42   27.45
6870          0.40  852.96    0.12    0.07   22.68    0.09   22.68   23.19
1750          1.26  852.96    0.51    0.31   22.68    0.42   22.68   23.19
1740          0.77  852.96    0.16    0.10   22.68    0.15   22.68   23.19
1710          0.53  868.59    0.24    0.14   23.59    0.24   23.59   24.61
6830          0.74  852.96    0.33    0.20   22.68    0.34   22.68   23.19
1690          0.54  845.14    0.26    0.16   22.22    0.18   22.22   22.48
1680          0.19  845.14    0.03    0.02   22.22    0.03   22.22   22.48
1620          0.65  915.49    0.24    0.14   26.34    1.10   26.34   28.88
1550          0.37  868.59    0.10    0.06   23.59    0.63   23.59   24.61
1440          0.33  852.96    0.52    0.31   22.68    0.33   22.68   23.19
1430          0.38  852.96    0.05    0.03   22.68    0.08   22.68   23.19
-------------------------------------------------------------------------
Summary:     32.42 50151.66   16.73   10.09 1355.22   18.71 1355.22 1406.01

Report message to a moderator

Re: HN hangs with no reason. Memory shortage? [message #37370 is a reply to message #37368]

Fri, 04 September 2009 16:15

kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia

Senior Member

Oh. You overcommitment very very very much. See, for LowMem you have a value of 50000%, while normal values for your number of containers is about 100 to 120%.

See much more detailed description here: http://wiki.openvz.org/UBC_systemwide_configuration. Learn to use UBCs.

Kir Kolyshkin

Report message to a moderator

Re: HN hangs with no reason. Memory shortage? [message #37384 is a reply to message #37370]

Sat, 05 September 2009 15:45

piplite
Messages: 27
Registered: March 2008

Junior Member

And why doesnt memory use swap space? I mean system memory. I know that individual VE has no swap.
Lets say system memory usage reaches its top, what happens next?
In my situation system just hangs and needs a reboot.

Thanks.

Report message to a moderator

Re: HN hangs with no reason. Memory shortage? [message #37386 is a reply to message #37365]

Sat, 05 September 2009 18:24

divB
Messages: 79
Registered: April 2009

Member

AFAIK, low memory is kernel memory (kmemsize + socket buffers) and should be related to vmguarpages and numprocs. Look up in the wiki! And validate your config with vzcfgvalidate.

However, IMO your overall mem commitment is very high too. The MemSwap commit is 1355% (!) and this should be somewhere at 100% as far as I understand. Because if it is not, you promise your containers more memory than you have in your HN.

Regards,
divB

[Updated on: Sat, 05 September 2009 18:39]

Report message to a moderator

Re: HN hangs with no reason. Memory shortage? [message #38257 is a reply to message #37365]

Tue, 01 December 2009 20:32

piplite
Messages: 27
Registered: March 2008

Junior Member

Im getting to this question again as im confused. I have the following
What parameter am i missing and overselling here?

# vzmemcheck -v
Output values in %
veid        LowMem  LowMem     RAM MemSwap MemSwap   Alloc   Alloc   Alloc
              util  commit    util    util  commit    util  commit   limit
36789         0.26   23.22    0.07    0.04    1.76    0.06    1.76    4.15
36787         0.37   23.22    0.33    0.21    1.76    1.04    1.76    4.15
36786         0.27   23.22    0.06    0.04    1.76    0.06    1.76    4.15
36785         0.43   23.22    0.31    0.19    1.76    0.47    1.76    4.15
36784         0.56   29.16    0.27    0.17    3.60    0.59    3.60   10.75
36783         0.30   23.22    0.12    0.08    1.76    0.10    1.76    4.15
36782         0.87   38.65    0.62    0.39    4.79    1.40    4.79   14.32
36781         0.24   26.78    0.06    0.04    2.71    0.05    2.71    7.47
36780         0.46   26.78    0.48    0.30    2.71    1.07    2.71    7.47
36779         0.72   29.16    1.05    0.66    3.60    1.01    3.60   10.75
36778         0.21   38.65    0.06    0.04    4.79    0.06    4.79   14.32
36762         0.40   23.22    0.26    0.16    1.76    0.20    1.76    4.15
36758         0.21   23.22    0.05    0.03    1.76    0.04    1.76    4.15
36720         1.37   23.22    0.87    0.55    1.76    0.89    1.76    4.15
36672         0.57   23.22    0.51    0.32    1.76    0.49    1.76    4.15
36641         1.57   38.65    1.64    1.03    6.58    1.86    6.58   21.46
36639         0.70   25.60    0.57    0.36    2.26    1.67    2.26    5.83
6710          1.23  140.78    0.19    0.12    6.68    0.30    6.68    9.06
6580          0.24  144.34    0.06    0.04    7.62    0.05    7.62   12.38
6380          0.95  144.34    0.82    0.51    7.62    0.78    7.62   12.38
4300          0.30  135.56    0.29    0.18    6.86    0.32    6.86   10.43
3590          1.41  144.34    0.69    0.43    7.62    0.72    7.62   12.38
1470          0.31  473.10    0.12    0.08   26.13    0.09   26.13   45.18
1010          0.81  133.18    0.35    0.22    6.36    0.39    6.36    8.74
900           0.21  144.34    0.05    0.03    7.62    0.04    7.62   12.38
430           1.19  141.97    0.59    0.37    6.93    0.86    6.93    9.90
340           1.42  144.34    1.23    0.77    7.62    1.08    7.62   12.38
-------------------------------------------------------------------------
Summary:     17.60 2208.72   11.72    7.36  137.97   15.71  137.97  274.91

On a server i have 27 vps running with config files like this:

64Mb vps:

# UBC parameters (in form of barrier:limit)
KMEMSIZE="67108864:67108864"
LOCKEDPAGES="256:256"
PRIVVMPAGES="32768:65536"
SHMPAGES="21504:21504"
NUMPROC="192:192"
PHYSPAGES="0:2147483647"
VMGUARPAGES="16384:16384"
OOMGUARPAGES="16384:16384"
NUMTCPSOCK="1536:1536"
NUMFLOCK="188:206"
NUMPTY="16:16"
NUMSIGINFO="256:256"
TCPSNDBUF="6291456:6291456"
TCPRCVBUF="6291456:6291456"
OTHERSOCKBUF="1126080:2097152"
DGRAMRCVBUF="262144:262144"
NUMOTHERSOCK="1536:1536"
DCACHESIZE="3409920:3624960"
NUMFILE="1536:1536"
AVNUMPROC="180:180"
NUMIPTENT="128:128"

# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="3072000:3072000"
DISKINODES="384000:384000"
QUOTATIME="0"

# CPU fair sheduler parameter
CPUUNITS="1000"


VE_ROOT="/vz/root/$VEID"
VE_PRIVATE="/vps/$VEID"
OSTEMPLATE="fedora-10-i386-default"
ORIGIN_SAMPLE="vps.basic"
QUOTAUGIDLIMIT="2048"
CPULIMIT="10"
CPUS="1"
MEMINFO="pages:16384"

128 vps:

# UBC parameters (in form of barrier:limit)
KMEMSIZE="67108864:67108864"
LOCKEDPAGES="256:256"
PRIVVMPAGES="98304:131072"
SHMPAGES="21504:21504"
NUMPROC="512:512"
PHYSPAGES="0:2147483647"
VMGUARPAGES="32768:32768"
OOMGUARPAGES="32768:32768"
NUMTCPSOCK="3072:3072"
NUMFLOCK="188:206"
NUMPTY="16:16"
NUMSIGINFO="256:256"
TCPSNDBUF="12582912:12582912"
TCPRCVBUF="12582912:12582912"
OTHERSOCKBUF="1126080:2097152"
DGRAMRCVBUF="262144:262144"
NUMOTHERSOCK="3072:3072"
DCACHESIZE="3409920:3624960"
NUMFILE="3072:3072"
AVNUMPROC="180:180"
NUMIPTENT="128:128"

# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="4608000:4608000"
DISKINODES="576000:576000"
QUOTATIME="0"

# CPU fair sheduler parameter
CPUUNITS="1000"


VE_ROOT="/vz/root/$VEID"
VE_PRIVATE="/vps/$VEID"
OSTEMPLATE="debian-5.0.3-i386-minimal"
ORIGIN_SAMPLE="vps.basic"
QUOTAUGIDLIMIT="2048"
CPULIMIT="20"
CPUS="1"
MEMINFO="pages:32768"

Report message to a moderator

Re: HN hangs with no reason. Memory shortage? [message #38284 is a reply to message #38257]

Thu, 03 December 2009 19:08

unxs
Messages: 21
Registered: September 2009
Location: Oregon, USA

Junior Member

You will need to read up on this. See the above posts.

Most expert forum users will ignore you and other newbies -unless you prove yourself and/or it helps the paid tools guys.

You were lucky so far Wink

. Use a webgui to run your VZ nodes, if you do not have the time to learn and tinker on your own. Some vz manager software will make sure you do not overcommit resources and even adjust your UBCs for you.

Just remember there is a war in progress: vz/xen/vmware among other virtualization tech. (and of course the paid vs. free tools.)

Report message to a moderator

Previous Topic:	CentOS 5 template
Next Topic:	download.openvz.org broken

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Thu Nov 07 20:03:49 GMT 2024

Total time taken to generate the page: 0.03361 seconds