OpenVZ Forum


Home » General » Support » HN hangs with no reason. Memory shortage?
HN hangs with no reason. Memory shortage? [message #37365] Fri, 04 September 2009 15:42 Go to next message
piplite is currently offline  piplite
Messages: 27
Registered: March 2008
Junior Member
So i have a host node running 20 VEs.
From time to time HN just hangs, no ping, no ssh response. i have to reboot it by powercycling to bring it back to work.
Its 2.6.18-128.2.1.el5.028stab064.4PAE running centos 5.2

and as you can see from /var/log/messages from 10.49 to 11.25 (reboot time) server doesnt respond.

Where do i look? I have same thing happening from time to time with 2-3 of my servers.

Sep  4 10:31:05 perun kernel: Fatal resource shortage: privvmpages, UB 6640.
Sep  4 10:31:05 perun kernel: Fatal resource shortage: privvmpages, UB 6640.
Sep  4 10:32:59 perun kernel: CT: 6830: stopped
Sep  4 10:33:23 perun kernel: CT: 6830: started
Sep  4 10:49:03 perun kernel: CT: 6830: stopped
Sep  4 10:49:12 perun kernel: CT: 6830: started
Sep  4 11:02:52 perun kernel: CT: 6830: stopped
Sep  4 11:03:23 perun kernel: CT: 6830: started
Sep  4 11:25:05 perun syslogd 1.4.1: restart.
Sep  4 11:25:05 perun kernel: klogd 1.4.1, log source = /proc/kmsg started.
Sep  4 11:25:05 perun kernel: Linux version 2.6.18-128.2.1.el5.028stab064.4PAE (root@rh5-build-x64) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jul 22 00:38:32 MSD 2009
Sep  4 11:25:05 perun kernel: BIOS-provided physical RAM map:
Sep  4 11:25:05 perun kernel: **********************************************************

Re: HN hangs with no reason. Memory shortage? [message #37366 is a reply to message #37365] Fri, 04 September 2009 15:45 Go to previous messageGo to next message
kir is currently offline  kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia
Senior Member

Show us your vzmemcheck -v output please.

Kir Kolyshkin
http://static.openvz.org/userbars/openvz-developer.png
Re: HN hangs with no reason. Memory shortage? [message #37368 is a reply to message #37366] Fri, 04 September 2009 15:47 Go to previous messageGo to next message
piplite is currently offline  piplite
Messages: 27
Registered: March 2008
Junior Member
Here its:
# vzmemcheck -v
Output values in %
veid        LowMem  LowMem     RAM MemSwap MemSwap   Alloc   Alloc   Alloc
              util  commit    util    util  commit    util  commit   limit
6880          0.75  852.96    0.16    0.10   22.68    0.15   22.68   23.19
6860          0.59  852.96    0.28    0.17   22.68    0.29   22.68   23.19
6840          0.61  852.96    0.31    0.19   22.68    0.62   22.68   23.19
6820          0.16  852.96    0.03    0.02   22.68    0.02   22.68   23.19
6810          0.40  852.96    0.05    0.03   22.68    0.07   22.68   23.19
6800          0.54  852.96    0.20    0.12   22.68    0.25   22.68   23.19
6790          0.79  852.96    0.19    0.12   22.68    0.17   22.68   23.19
6780          1.85  852.96    0.55    0.33   22.68    0.61   22.68   23.19
6770          0.19  852.96    0.03    0.02   22.68    0.03   22.68   23.19
6740          0.31  852.96    0.08    0.05   22.68    0.07   22.68   23.19
6690          0.16  852.96    0.03    0.02   22.68    0.02   22.68   23.19
6650          0.35  852.96    1.32    0.80   22.68    0.93   22.68   23.19
6640          0.42  852.96    0.23    0.14   22.68    0.50   22.68   23.19
6590          0.32  852.96    0.06    0.04   22.68    0.05   22.68   23.19
6570          0.37  852.96    0.06    0.04   22.68    0.05   22.68   23.19
6530          0.37  852.96    0.05    0.03   22.68    0.08   22.68   23.19
6520          0.84  868.59    0.45    0.27   23.59    0.47   23.59   24.61
6450          0.18  852.96    0.03    0.02   22.68    0.03   22.68   23.19
6420          0.53  852.96    0.20    0.12   22.68    0.25   22.68   23.19
2690          0.28  852.96    0.07    0.04   22.68    0.21   22.68   23.19
6400          0.41  868.59    0.17    0.11   23.59    0.35   23.59   24.61
5890          0.30  899.86    0.08    0.05   25.42    0.07   25.42   27.45
4250          0.46  884.23    0.21    0.12   24.51    0.17   24.51   26.03
4090          0.31  868.59    0.08    0.05   23.59    0.07   23.59   24.61
4060          0.21  868.59    0.03    0.02   23.59    0.03   23.59   24.61
3970          0.20  899.86    0.04    0.02   25.42    0.04   25.42   27.45
3780          0.59  868.59    0.26    0.15   23.59    0.30   23.59   24.61
3760          0.54  868.59    0.21    0.12   23.59    0.25   23.59   24.61
3750          0.66  962.39    0.24    0.14   29.08    0.22   29.08   33.14
3740          0.94  868.59    1.13    0.68   23.59    1.24   23.59   24.61
3500          1.42  899.86    1.69    1.02   25.42    1.30   25.42   27.45
3450          0.95  868.59    0.23    0.14   23.59    0.20   23.59   24.61
2770          0.25  852.96    0.04    0.03   22.68    0.04   22.68   23.19
2630          0.54  868.59    0.20    0.12   23.59    0.25   23.59   24.61
2170          0.45  852.96    0.14    0.08   22.68    0.15   22.68   23.19
2160          0.98  884.23    0.92    0.56   24.51    0.69   24.51   26.03
2120          0.57  868.59    0.32    0.20   23.59    0.95   23.59   23.85
2090          1.33  852.96    0.53    0.32   22.68    0.54   22.68   23.19
2070          0.28  852.96    0.11    0.07   22.68    0.15   22.68   23.19
1980          0.27  868.59    0.07    0.04   23.59    0.06   23.59   24.61
1960          0.85  868.59    0.64    0.39   23.59    0.56   23.59   24.61
1950          0.19  845.14    0.03    0.02   22.22    0.03   22.22   22.48
1930          0.39  852.96    0.05    0.03   22.68    0.08   22.68   23.19
1920          0.49  884.23    0.48    0.29   24.51    0.35   24.51   26.03
1900          0.99  868.59    0.49    0.29   23.59    0.75   23.59   24.61
1890          0.99  868.59    0.97    0.58   23.59    0.62   23.59   24.61
1850          0.68  899.86    0.42    0.25   25.42    0.83   25.42   27.45
6870          0.40  852.96    0.12    0.07   22.68    0.09   22.68   23.19
1750          1.26  852.96    0.51    0.31   22.68    0.42   22.68   23.19
1740          0.77  852.96    0.16    0.10   22.68    0.15   22.68   23.19
1710          0.53  868.59    0.24    0.14   23.59    0.24   23.59   24.61
6830          0.74  852.96    0.33    0.20   22.68    0.34   22.68   23.19
1690          0.54  845.14    0.26    0.16   22.22    0.18   22.22   22.48
1680          0.19  845.14    0.03    0.02   22.22    0.03   22.22   22.48
1620          0.65  915.49    0.24    0.14   26.34    1.10   26.34   28.88
1550          0.37  868.59    0.10    0.06   23.59    0.63   23.59   24.61
1440          0.33  852.96    0.52    0.31   22.68    0.33   22.68   23.19
1430          0.38  852.96    0.05    0.03   22.68    0.08   22.68   23.19
-------------------------------------------------------------------------
Summary:     32.42 50151.66   16.73   10.09 1355.22   18.71 1355.22 1406.01

Re: HN hangs with no reason. Memory shortage? [message #37370 is a reply to message #37368] Fri, 04 September 2009 16:15 Go to previous messageGo to next message
kir is currently offline  kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia
Senior Member

Oh. You overcommitment very very very much. See, for LowMem you have a value of 50000%, while normal values for your number of containers is about 100 to 120%.

See much more detailed description here: http://wiki.openvz.org/UBC_systemwide_configuration. Learn to use UBCs.


Kir Kolyshkin
http://static.openvz.org/userbars/openvz-developer.png
Re: HN hangs with no reason. Memory shortage? [message #37384 is a reply to message #37370] Sat, 05 September 2009 15:45 Go to previous messageGo to next message
piplite is currently offline  piplite
Messages: 27
Registered: March 2008
Junior Member
And why doesnt memory use swap space? I mean system memory. I know that individual VE has no swap.
Lets say system memory usage reaches its top, what happens next?
In my situation system just hangs and needs a reboot.

Thanks.
Re: HN hangs with no reason. Memory shortage? [message #37386 is a reply to message #37365] Sat, 05 September 2009 18:24 Go to previous messageGo to next message
divB is currently offline  divB
Messages: 79
Registered: April 2009
Member
AFAIK, low memory is kernel memory (kmemsize + socket buffers) and should be related to vmguarpages and numprocs. Look up in the wiki! And validate your config with vzcfgvalidate.

However, IMO your overall mem commitment is very high too. The MemSwap commit is 1355% (!) and this should be somewhere at 100% as far as I understand. Because if it is not, you promise your containers more memory than you have in your HN.

Regards,
divB

[Updated on: Sat, 05 September 2009 18:39]

Report message to a moderator

Re: HN hangs with no reason. Memory shortage? [message #38257 is a reply to message #37365] Tue, 01 December 2009 20:32 Go to previous messageGo to next message
piplite is currently offline  piplite
Messages: 27
Registered: March 2008
Junior Member
Im getting to this question again as im confused. I have the following
What parameter am i missing and overselling here?
# vzmemcheck -v
Output values in %
veid        LowMem  LowMem     RAM MemSwap MemSwap   Alloc   Alloc   Alloc
              util  commit    util    util  commit    util  commit   limit
36789         0.26   23.22    0.07    0.04    1.76    0.06    1.76    4.15
36787         0.37   23.22    0.33    0.21    1.76    1.04    1.76    4.15
36786         0.27   23.22    0.06    0.04    1.76    0.06    1.76    4.15
36785         0.43   23.22    0.31    0.19    1.76    0.47    1.76    4.15
36784         0.56   29.16    0.27    0.17    3.60    0.59    3.60   10.75
36783         0.30   23.22    0.12    0.08    1.76    0.10    1.76    4.15
36782         0.87   38.65    0.62    0.39    4.79    1.40    4.79   14.32
36781         0.24   26.78    0.06    0.04    2.71    0.05    2.71    7.47
36780         0.46   26.78    0.48    0.30    2.71    1.07    2.71    7.47
36779         0.72   29.16    1.05    0.66    3.60    1.01    3.60   10.75
36778         0.21   38.65    0.06    0.04    4.79    0.06    4.79   14.32
36762         0.40   23.22    0.26    0.16    1.76    0.20    1.76    4.15
36758         0.21   23.22    0.05    0.03    1.76    0.04    1.76    4.15
36720         1.37   23.22    0.87    0.55    1.76    0.89    1.76    4.15
36672         0.57   23.22    0.51    0.32    1.76    0.49    1.76    4.15
36641         1.57   38.65    1.64    1.03    6.58    1.86    6.58   21.46
36639         0.70   25.60    0.57    0.36    2.26    1.67    2.26    5.83
6710          1.23  140.78    0.19    0.12    6.68    0.30    6.68    9.06
6580          0.24  144.34    0.06    0.04    7.62    0.05    7.62   12.38
6380          0.95  144.34    0.82    0.51    7.62    0.78    7.62   12.38
4300          0.30  135.56    0.29    0.18    6.86    0.32    6.86   10.43
3590          1.41  144.34    0.69    0.43    7.62    0.72    7.62   12.38
1470          0.31  473.10    0.12    0.08   26.13    0.09   26.13   45.18
1010          0.81  133.18    0.35    0.22    6.36    0.39    6.36    8.74
900           0.21  144.34    0.05    0.03    7.62    0.04    7.62   12.38
430           1.19  141.97    0.59    0.37    6.93    0.86    6.93    9.90
340           1.42  144.34    1.23    0.77    7.62    1.08    7.62   12.38
-------------------------------------------------------------------------
Summary:     17.60 2208.72   11.72    7.36  137.97   15.71  137.97  274.91


On a server i have 27 vps running with config files like this:

64Mb vps:
# UBC parameters (in form of barrier:limit)
KMEMSIZE="67108864:67108864"
LOCKEDPAGES="256:256"
PRIVVMPAGES="32768:65536"
SHMPAGES="21504:21504"
NUMPROC="192:192"
PHYSPAGES="0:2147483647"
VMGUARPAGES="16384:16384"
OOMGUARPAGES="16384:16384"
NUMTCPSOCK="1536:1536"
NUMFLOCK="188:206"
NUMPTY="16:16"
NUMSIGINFO="256:256"
TCPSNDBUF="6291456:6291456"
TCPRCVBUF="6291456:6291456"
OTHERSOCKBUF="1126080:2097152"
DGRAMRCVBUF="262144:262144"
NUMOTHERSOCK="1536:1536"
DCACHESIZE="3409920:3624960"
NUMFILE="1536:1536"
AVNUMPROC="180:180"
NUMIPTENT="128:128"

# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="3072000:3072000"
DISKINODES="384000:384000"
QUOTATIME="0"

# CPU fair sheduler parameter
CPUUNITS="1000"


VE_ROOT="/vz/root/$VEID"
VE_PRIVATE="/vps/$VEID"
OSTEMPLATE="fedora-10-i386-default"
ORIGIN_SAMPLE="vps.basic"
QUOTAUGIDLIMIT="2048"
CPULIMIT="10"
CPUS="1"
MEMINFO="pages:16384"


128 vps:
# UBC parameters (in form of barrier:limit)
KMEMSIZE="67108864:67108864"
LOCKEDPAGES="256:256"
PRIVVMPAGES="98304:131072"
SHMPAGES="21504:21504"
NUMPROC="512:512"
PHYSPAGES="0:2147483647"
VMGUARPAGES="32768:32768"
OOMGUARPAGES="32768:32768"
NUMTCPSOCK="3072:3072"
NUMFLOCK="188:206"
NUMPTY="16:16"
NUMSIGINFO="256:256"
TCPSNDBUF="12582912:12582912"
TCPRCVBUF="12582912:12582912"
OTHERSOCKBUF="1126080:2097152"
DGRAMRCVBUF="262144:262144"
NUMOTHERSOCK="3072:3072"
DCACHESIZE="3409920:3624960"
NUMFILE="3072:3072"
AVNUMPROC="180:180"
NUMIPTENT="128:128"

# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="4608000:4608000"
DISKINODES="576000:576000"
QUOTATIME="0"

# CPU fair sheduler parameter
CPUUNITS="1000"


VE_ROOT="/vz/root/$VEID"
VE_PRIVATE="/vps/$VEID"
OSTEMPLATE="debian-5.0.3-i386-minimal"
ORIGIN_SAMPLE="vps.basic"
QUOTAUGIDLIMIT="2048"
CPULIMIT="20"
CPUS="1"
MEMINFO="pages:32768"

Re: HN hangs with no reason. Memory shortage? [message #38284 is a reply to message #38257] Thu, 03 December 2009 19:08 Go to previous message
unxs is currently offline  unxs
Messages: 21
Registered: September 2009
Location: Oregon, USA
Junior Member
You will need to read up on this. See the above posts.

Most expert forum users will ignore you and other newbies -unless you prove yourself and/or it helps the paid tools guys.

You were lucky so far Wink. Use a webgui to run your VZ nodes, if you do not have the time to learn and tinker on your own. Some vz manager software will make sure you do not overcommit resources and even adjust your UBCs for you.

Just remember there is a war in progress: vz/xen/vmware among other virtualization tech. (and of course the paid vs. free tools.)
Previous Topic: CentOS 5 template
Next Topic: download.openvz.org broken
Goto Forum:
  


Current Time: Sat Apr 13 08:51:22 GMT 2024

Total time taken to generate the page: 0.01991 seconds