OpenVZ Forum


Home » General » Support » -bash: fork: Cannot allocate memory
-bash: fork: Cannot allocate memory [message #29109] Sat, 05 April 2008 12:11 Go to next message
gnutered is currently offline  gnutered
Messages: 12
Registered: April 2008
Junior Member
Trying to dd a 1G file in a Debian VE on a Debian HN causes the above error, and then the VE cannot be successfully restarted.

Background:
Kernel is official Debian Etch + openvz. Guests are two Debian minimals (vanilla, no change). HN has 512MB RAM.

I try to create a large file:

root@oneoneoneoneoneone:/# time dd if=/dev/zero of=~/zero bs=1024k count=1000
dd: writing `//zero': Disk quota exceeded
818+0 records in
817+0 records out
857153536 bytes (857 MB) copied, 14.4335 seconds, 59.4 MB/s


This fills the disk quota (no big deal). Second time I get my error:

root@oneoneoneoneoneone:/# time dd if=/dev/zero of=~/zero bs=1024k count=1000
-bash: fork: Cannot allocate memory
root@oneoneoneoneoneone:/# free -m
-bash: fork: Cannot allocate memory
root@oneoneoneoneoneone:/# logout
exited from VPS 111111


I deleted the zero file and even increased the disk quota, on the long shot that this was related to exceeding the disk quota. Did not fix.

I cannot restart the VE:

tony@sodium:/proc/bc/111111$ sudo vzctl restart 111111
Restarting VPS
Stopping VPS ...
VPS was stopped
vps-net_del WARNING: Function proxy_arp for eth0 is set to 0. Enable with 'sysctl -w net.ipv4.conf.eth0.proxy_arp=1'. See /usr/share/doc/vzctl/README.Debian.
VPS is unmounted
Starting VPS ...
VPS is mounted
Unable to fork: Cannot allocate memory
Not enough resources to start environment
VPS start failed
VPS is unmounted


I have enough free memory (or so it appears):

tony@sodium:/proc/bc/111111$ free -m
             total       used       free     shared    buffers     cached
Mem:           502        147        355          0          3        112
-/+ buffers/cache:         31        471
Swap:         2047          0       2047

I haven't yet restarted the server. I'd like to try and recover from this broken situation first.

Any clues?

Tony
Re: -bash: fork: Cannot allocate memory [message #29110 is a reply to message #29109] Sat, 05 April 2008 12:40 Go to previous messageGo to next message
swindmill is currently offline  swindmill
Messages: 57
Registered: April 2007
Member
Can you paste the contents of /proc/user_beancounters for the affected VE?

It sounds like you're hitting up against a resource limit.

Also, what versions of the kernel, vzctl and vzquota are you using?
Re: -bash: fork: Cannot allocate memory [message #29111 is a reply to message #29110] Sat, 05 April 2008 12:54 Go to previous messageGo to next message
gnutered is currently offline  gnutered
Messages: 12
Registered: April 2008
Junior Member
tony@sodium:/proc/bc/111111$ sudo cat /proc/user_beancounters
Version: 2.5
       uid  resource           held    maxheld    barrier      limit    failcnt
   222222:  kmemsize              0     507534    2752512    2936012          0
            lockedpages           0          0         32         32          0
            privvmpages       51200      52436      49152      53575          3
            shmpages              0          0       8192       8192          0
            dummy                 0          0          0          0          0
            numproc               0          9         65         65          0
            physpages             0        878          0 2147483647          0
            vmguarpages           0          0       6144 2147483647          0
            oomguarpages          0        878       6144 2147483647          0
            numtcpsock            0          2         80         80          0
            numflock              0          2        100        110          0
            numpty                0          1         16         16          0
            numsiginfo            0          3        256        256          0
            tcpsndbuf             0          0     319488     524288          0
            tcprcvbuf             0          0     319488     524288          0
            othersockbuf          0       2928     132096     336896          0
            dgramrcvbuf           0          0     132096     132096          0
            numothersock          0          4         80         80          0
            dcachesize            0          0    1048576    1097728          0
            numfile               0        208       2048       2048          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            numiptent             0         10        128        128          0
   111111:  kmemsize              0     922339    2752512    2936012          0
            lockedpages           0          8         32         32          0
            privvmpages      235008     236610      49152      53575          9
            shmpages              0          0       8192       8192          0
            dummy                 0          0          0          0          0
            numproc               0         15         65         65          0
            physpages             0       5431          0 2147483647          0
            vmguarpages           0          0       6144 2147483647          0
            oomguarpages          0       5431       6144 2147483647          0
            numtcpsock            0          2         80         80          0
            numflock              0          7        100        110          0
            numpty                0          1         16         16          0
            numsiginfo            0          3        256        256          0
            tcpsndbuf             0       6696     319488     524288          0
            tcprcvbuf             0     325872     319488     524288          0
            othersockbuf          0       6696     132096     336896          0
            dgramrcvbuf           0       8376     132096     132096          0
            numothersock          0          4         80         80          0
            dcachesize            0          0    1048576    1097728          0
            numfile               0        323       2048       2048          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            dummy                 0          0          0          0          0
            numiptent             0         10        128        128          0
        0:  kmemsize        2719902    3336457 2147483647 2147483647          0
            lockedpages           0          0 2147483647 2147483647          0
            privvmpages        5660      11610 2147483647 2147483647          0
            shmpages           1292       1308 2147483647 2147483647          0
            dummy                 0          0 2147483647 2147483647          0
            numproc              58         72 2147483647 2147483647          0
            physpages          2658       5432 2147483647 2147483647          0
            vmguarpages           0          0 2147483647 2147483647          0
            oomguarpages       2670       5432 2147483647 2147483647          0
            numtcpsock            7          8 2147483647 2147483647          0
            numflock              2          4 2147483647 2147483647          0
            numpty                2          2 2147483647 2147483647          0
            numsiginfo            0          2 2147483647 2147483647          0
            tcpsndbuf         75888      75888 2147483647 2147483647          0
            tcprcvbuf        114688      81920 2147483647 2147483647          0
            othersockbuf      20088      24776 2147483647 2147483647          0
            dgramrcvbuf           0       8376 2147483647 2147483647          0
            numothersock         32         36 2147483647 2147483647          0
            dcachesize            0          0 2147483647 2147483647          0
            numfile            1149       1368 2147483647 2147483647          0
            dummy                 0          0 2147483647 2147483647          0
            dummy                 0          0 2147483647 2147483647          0
            dummy                 0          0 2147483647 2147483647          0
            numiptent            10         10 2147483647 2147483647          0


Kernel is from http://download.openvz.org debian repository; vzctl and vzquota are from etch repository:

tony@sodium:/proc/bc/111111$ uname -a
Linux sodium 2.6.18-openvz-686 #1 SMP Tue Apr 10 20:28:40 CEST 2007 i686 GNU/Linux
tony@sodium:/proc/bc/111111$ dpkg -l linux-image-`uname -r`
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                   Version                Description
+++-======================-======================-============================================================
ii  linux-image-2.6.18-ope 028.18.1-2.6.18-12-1   Linux kernel binary image for version 2.6.18-openvz-686

tony@sodium:/proc/bc/111111$ dpkg -l vzctl vzquota
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name                   Version                Description
+++-======================-======================-============================================================
ii  vzctl                  3.0.11-13              server virtualization solution - control tools
ii  vzquota                3.0.8-2                server virtualization solution - quota tools


[Updated on: Sat, 05 April 2008 12:54]

Report message to a moderator

Re: -bash: fork: Cannot allocate memory [message #29112 is a reply to message #29111] Sat, 05 April 2008 12:57 Go to previous messageGo to next message
swindmill is currently offline  swindmill
Messages: 57
Registered: April 2007
Member
On both VEs you have hit up against the privvmpages resource limit.

The last column shows a 9 for one VE and a 3 for the other, which means you've tried to exceed the memory allocation limit that number of times.

Try increasing the privvmpages value for the VE in question and try again.
Re: -bash: fork: Cannot allocate memory [message #29114 is a reply to message #29112] Sat, 05 April 2008 13:17 Go to previous messageGo to next message
gnutered is currently offline  gnutered
Messages: 12
Registered: April 2008
Junior Member
Cool, thanks for that. It works - kinda.

The privvmpages count persists across restarts of the VE. Does this mean it's a cumulative count of *all* memory allocations, ever. for that VE? That seems wrong, unless this parameter is not meant be used except in unusual circumstances.

I edited the conf file, and she would start. But subsequent dd commands show that the privvmpages count is still going up, and, sure enough, I ran out of resources again.

The failcnt column is a good idea.

Thanks for your help.

Tony
Re: -bash: fork: Cannot allocate memory [message #29148 is a reply to message #29109] Mon, 07 April 2008 10:41 Go to previous messageGo to next message
kir is currently offline  kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia
Senior Member

Quote:

The privvmpages count persists across restarts of the VE. Does this mean it's a cumulative count of *all* memory allocations, ever. for that VE? That seems wrong, unless this parameter is not meant be used except in unusual circumstances.


privvmpages.held (as well as all other UBC parameters) shows the current usage of this resource. If a container is stopped and any parameter's held value is more than zero, this means either a resource leak or a bug in accounting -- and in both cases should be reported straight to http://bugzilla.openvz.org/

Speaking of fail counters, see http://wiki.openvz.org/UBC_failcnt_reset

For more info, see http://wiki.openvz.org/privvmpages and in general http://wiki.openvz.org/UBC


Kir Kolyshkin
http://static.openvz.org/userbars/openvz-developer.png
Re: -bash: fork: Cannot allocate memory [message #29153 is a reply to message #29148] Mon, 07 April 2008 12:00 Go to previous messageGo to next message
gnutered is currently offline  gnutered
Messages: 12
Registered: April 2008
Junior Member
Bug: http://bugzilla.openvz.org/show_bug.cgi?id=870

Thanks to swindmill for the pointer to the privvmpages.

Tony
Re: -bash: fork: Cannot allocate memory [message #29154 is a reply to message #29153] Mon, 07 April 2008 12:44 Go to previous messageGo to next message
swindmill is currently offline  swindmill
Messages: 57
Registered: April 2007
Member
Perhaps you should try updating your vzctl and vzquota to the newest versions available.

These can be downloaded and installed from the openvz mirror of the debian-systs repository by adding the following to your sources.list:

deb http://download.openvz.org/debian-systs etch openvz
Re: -bash: fork: Cannot allocate memory [message #29158 is a reply to message #29154] Mon, 07 April 2008 14:22 Go to previous message
gnutered is currently offline  gnutered
Messages: 12
Registered: April 2008
Junior Member
Sure, can do, but this looks very much like a kernel bug IMO

Tony
Previous Topic: startproc issue in a VPS
Next Topic: SOLVED: Hang on vzctl start/stop in futex() in the ve.
Goto Forum:
  


Current Time: Tue Nov 05 20:06:00 GMT 2024

Total time taken to generate the page: 0.05147 seconds