OpenVZ Forum: Support » OpenVZ - allocation failed: out of vmalloc space

Home » General » Support » OpenVZ - allocation failed: out of vmalloc space

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Switch to threaded view of this topic

Create a new topic

Submit Reply

OpenVZ - allocation failed: out of vmalloc space [message #24460]

Wed, 05 December 2007 10:16

stuartornum is currently offline

stuartornum
Messages: 21
Registered: February 2007

Junior Member

Hi,

Got a slight problem on my development OpenVZ server, it has been working 100% up until 2 days ago when I did a restart.

I have 5 VE's (id 201 - 205), I was able to run all 5 will ample resourses spare.

Now I can only run one at a time...?

I checked the logs and get this
Code:

Dec  3 18:16:17 localhost kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to increase size.
Dec  3 18:16:17 localhost kernel: VE: 202: stopped
Dec  3 18:16:17 localhost kernel: VE: 202: failed to start with err=-12

This may help as well
Code:

# cat /proc/meminfo | grep Vmalloc
VmallocTotal:   110584 kB
VmallocUsed:    110200 kB
VmallocChunk:      136 kB

Thanks in advanced

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24496 is a reply to message #24460]

Wed, 05 December 2007 15:47

maratrus is currently offline

maratrus
Messages: 1495
Registered: August 2007
Location: Moscow

Senior Member

Hi,

please give to us a little more information:

1. What kernel do you use?
2. What OS do you use on HN?
3. What kind of templates do you use?

Thank You!

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24498 is a reply to message #24496]

Wed, 05 December 2007 15:52

stuartornum is currently offline

stuartornum
Messages: 21
Registered: February 2007

Junior Member

Hi,

I am running on CentOS 4.5 kernel 2.6.18-8.1.14.el5.028stab045.1

I am using the standard OpenVZ templates of CentOS default and minimal

Some more info

# free -mt
             total       used       free     shared    buffers     cached
Mem:          1007        608        398          0         47        515
-/+ buffers/cache:         46        960
Swap:         2055          0       2054
Total:        3062        609       2453

Thanks for you help.

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24513 is a reply to message #24498]

Wed, 05 December 2007 17:45

maratrus is currently offline

maratrus
Messages: 1495
Registered: August 2007
Location: Moscow

Senior Member

Hi,

Sorry for the inconvenience but can you also attach lsmod output?

Thank You!

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24516 is a reply to message #24513]

Wed, 05 December 2007 18:04

stuartornum is currently offline

stuartornum
Messages: 21
Registered: February 2007

Junior Member

Hi,

# lsmod
Module                  Size  Used by
simfs                   9068  1
vzcpt                 111524  0
vzdquota               44180  1 [permanent]
xt_tcpudp               7040  0
xt_length               6016  0
ipt_ttl                 5888  0
xt_tcpmss               6272  0
ipt_TCPMSS              8064  0
iptable_mangle          8576  1
iptable_nat            13188  2
ip_nat                 22288  1 iptable_nat
ip_conntrack           60356  4 vzcpt,iptable_nat,ip_nat
nfnetlink              10648  2 ip_nat,ip_conntrack
iptable_filter          8576  1
xt_multiport            7168  0
xt_limit                6656  0
ipt_tos                 5760  0
ipt_REJECT              9344  0
ip_tables              18760  3 iptable_mangle,iptable_nat,iptable_filter
x_tables               19204  11 xt_tcpudp,xt_length,ipt_ttl,xt_tcpmss,ipt_TCPMS                                   S,iptable_nat,xt_multiport,xt_limit,ipt_tos,ipt_REJECT,ip_tables
parport_pc             29092  0
lp                     16168  0
parport                37960  2 parport_pc,lp
autofs4                24324  0
sunrpc                145084  1
vznetdev               21764  2
vzmon                  46088  3 vzcpt,vznetdev
ipv6                  261280  16 vzcpt,vzmon
vzdev                   7556  3 vzdquota,vznetdev,vzmon
joydev                 13888  0
dm_mirror              29648  0
dm_mod                 56984  1 dm_mirror
video                  19588  0
sbs                    18468  0
i2c_ec                  8960  1 sbs
button                 10512  0
battery                13700  0
asus_acpi              19480  0
ac                      9092  0
usb_storage            76512  0
uhci_hcd               25356  0
ehci_hcd               32780  0
intelfb                34348  1
cfbcopyarea             7552  1 intelfb
cfbimgblt               6912  1 intelfb
cfbfillrect             7552  1 intelfb
i2c_i801               11276  0
i2c_core               23552  2 i2c_ec,i2c_i801
snd_hda_intel          21268  0
snd_hda_codec         144128  1 snd_hda_intel
snd_pcm_oss            43168  0
snd_mixer_oss          19200  1 snd_pcm_oss
snd_pcm                72964  3 snd_hda_intel,snd_hda_codec,snd_pcm_oss
snd_timer              25604  1 snd_pcm
snd                    52740  6 snd_hda_intel,snd_hda_codec,snd_pcm_oss,snd_mixe                                   r_oss,snd_pcm,snd_timer
soundcore              13152  1 snd
snd_page_alloc         13832  2 snd_hda_intel,snd_pcm
r8169                  31496  0
e100                   36744  0
mii                     9216  1 e100
ide_cd                 39968  0
cdrom                  38048  1 ide_cd
ext3                  124040  3
jbd                    61736  1 ext3
ata_piix               17416  4
libata                 97176  1 ata_piix
sd_mod                 22912  6
scsi_mod              131212  3 usb_storage,libata,sd_mod

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24583 is a reply to message #24516]

Thu, 06 December 2007 14:07

vaverin is currently offline

vaverin
Messages: 708
Registered: September 2005

Senior Member

I would like to prepare the following experiment:
please remove some modules and wacth Vmalloc in /proc/meminfo.

First of all I suppose intelfb driver.

thank You,
Vasily Averin

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24586 is a reply to message #24583]

Thu, 06 December 2007 14:18

stuartornum is currently offline

stuartornum
Messages: 21
Registered: February 2007

Junior Member

I have managed to find what the problem was.

in /etc/vz/vz.conf

I originally had

## IPv4 iptables kernel modules
IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length"

However I wanted iptable_nat, so I added it to the list, and thats when I started getting this Vmalloc error, so I just removed it and all VPS' are now running !!

However why does iptable_nat cause his problem?

Thanks

[Updated on: Thu, 06 December 2007 14:19]

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24587 is a reply to message #24586]

Thu, 06 December 2007 14:25

vaverin is currently offline

vaverin
Messages: 708
Registered: September 2005

Senior Member

iptable_nat really consumes vmalloc, it creates some big hashes.
But I do not think that it causes this issue, I believe some other driver eats too many vmalloc.
Could you please show /proc/meminfo before and after starting of vz service?

thank you,
Vasily Averin

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24588 is a reply to message #24587]

Thu, 06 December 2007 14:29

stuartornum is currently offline

stuartornum
Messages: 21
Registered: February 2007

Junior Member

Hi,

Thanks for your help.

Here is meminfo with OpenVZ NOT running

# cat /proc/meminfo
MemTotal:      1031776 kB
MemFree:        878100 kB
Buffers:         15200 kB
Cached:         106688 kB
SwapCached:          0 kB
Active:          71332 kB
Inactive:        55164 kB
HighTotal:      128544 kB
HighFree:        31764 kB
LowTotal:       903232 kB
LowFree:        846336 kB
SwapTotal:     2104504 kB
SwapFree:      2104504 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:        4644 kB
Mapped:           4096 kB
Slab:            13576 kB
PageTables:        512 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   2620392 kB
Committed_AS:    18460 kB
VmallocTotal:   110584 kB
VmallocUsed:    109536 kB
VmallocChunk:      624 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     4096 kB

And here it is with OpenVZ running:

# cat /proc/meminfo
MemTotal:      1031776 kB
MemFree:        832564 kB
Buffers:         15456 kB
Cached:         106952 kB
SwapCached:          0 kB
Active:         118128 kB
Inactive:        49088 kB
HighTotal:      128544 kB
HighFree:          688 kB
LowTotal:       903232 kB
LowFree:        831876 kB
SwapTotal:     2104504 kB
SwapFree:      2104504 kB
Dirty:            2464 kB
Writeback:           0 kB
AnonPages:       44828 kB
Mapped:          22912 kB
Slab:            16444 kB
PageTables:       1544 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   2620392 kB
Committed_AS:   268868 kB
VmallocTotal:   110584 kB
VmallocUsed:    110336 kB
VmallocChunk:      100 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     4096 kB

Thanks again

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #24631 is a reply to message #24588]

Fri, 07 December 2007 05:15

vaverin is currently offline

vaverin
Messages: 708
Registered: September 2005

Senior Member

Hi Stuart,
thank You for cooperation.

As You can see VmallocUsed (109536 kB) on Your node is too high even without vz running.

Usual consuption should be like the following

VmallocTotal: 114680 kB
VmallocUsed: 5748 kB
VmallocChunk: 108532 kB

It is important to find the cause of this issue.
I believe that vmalloc was used by some module. Could You please remove kernel modules and watch /proc/meminfo?

thank You,
Vasily Averin

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #27226 is a reply to message #24460]

Tue, 12 February 2008 11:14

stuartornum is currently offline

stuartornum
Messages: 21
Registered: February 2007

Junior Member

Hi Vasily,

Sorry for the delay, I still have the prolem with Vmalloc.

However I am unsure how to remove kernel modules.

Here is /etc/modprobe.conf

alias eth0 e100
alias eth1 r8169
alias scsi_hostadapter ata_piix
alias snd-card-0 snd-azx
options snd-card-0 index=0
install snd-azx /sbin/modprobe --ignore-install snd-azx && /usr/sbin/alsactl restore >/dev/null 2>&1 || :
remove snd-azx { /usr/sbin/alsactl store >/dev/null 2>&1 || : ; }; /sbin/modprobe -r --ignore-remove snd-azx
alias usb-controller ehci-hcd
alias usb-controller1 uhci-hcd

Thanks again

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #27228 is a reply to message #27226]

Tue, 12 February 2008 11:28

vaverin is currently offline

vaverin
Messages: 708
Registered: September 2005

Senior Member

Hi, Stuart!

Please try to stop services (i.e. vz, iptables, network, sound) -- it can lead to remove some modules.

Then please use rmmod for removing unused modules (it have '0' in 'Used by' clomn showed by lsmod).

After each attempt please look at current Vmalloc consumption (in /proc/meminfo file).

thank you,
Vasily Averin

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #27229 is a reply to message #24460]

Tue, 12 February 2008 11:39

stuartornum is currently offline

stuartornum
Messages: 21
Registered: February 2007

Junior Member

Hi Vasily,

Thanks again for you help;

So far I have done this ( the numbers are Vmalloc used, VmallocTotal is 110584 kB )

VZ On - 110272 KB
VZ Off - 109896 KB

IPtables Off - 109588 KB

rmmod snd_hda_intel
rmmod snd_hda_codec - 109392 kB

Here is lsmod again:

# lsmod
Module                  Size  Used by
ipt_LOG                10240  0
ipt_TOS                 6272  0
x_tables               19204  2 ipt_LOG,ipt_TOS
vzdquota               44180  0 [permanent]
parport_pc             29092  0
lp                     16168  0
parport                37960  2 parport_pc,lp
autofs4                24324  0
sunrpc                145084  1
ipv6                  261280  16
vzdev                   7556  1 vzdquota
joydev                 13888  0
dm_mirror              29648  0
dm_mod                 56984  1 dm_mirror
video                  19588  0
sbs                    18468  0
i2c_ec                  8960  1 sbs
button                 10512  0
battery                13700  0
asus_acpi              19480  0
ac                      9092  0
usb_storage            76512  0
uhci_hcd               25356  0
ehci_hcd               32780  0
intelfb                34348  1
cfbcopyarea             7552  1 intelfb
cfbimgblt               6912  1 intelfb
cfbfillrect             7552  1 intelfb
i2c_i801               11276  0
i2c_core               23552  2 i2c_ec,i2c_i801
snd_pcm_oss            43168  0
snd_mixer_oss          19200  1 snd_pcm_oss
snd_pcm                72964  1 snd_pcm_oss
snd_timer              25604  1 snd_pcm
snd                    52740  4 snd_pcm_oss,snd_mixer_oss,snd_pcm,snd_timer
soundcore              13152  1 snd
snd_page_alloc         13832  1 snd_pcm
r8169                  31496  0
e100                   36744  0
mii                     9216  1 e100
ide_cd                 39968  0
cdrom                  38048  1 ide_cd
ext3                  124040  3
jbd                    61736  1 ext3
ata_piix               17416  4
libata                 97176  1 ata_piix
sd_mod                 22912  6
scsi_mod              131212  3 usb_storage,libata,sd_mod

Thanks

[Updated on: Tue, 12 February 2008 11:39]

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #27233 is a reply to message #27229]

Tue, 12 February 2008 12:35

vaverin is currently offline

vaverin
Messages: 708
Registered: September 2005

Senior Member

It is still abnormal high vmalloc consumption.
please continue to remove the unused modules.
You need to found module that eats ~100 Mb vmalloc.

Report message to a moderator

Send a private message to this user

Re: OpenVZ - allocation failed: out of vmalloc space [message #27234 is a reply to message #24460]

Tue, 12 February 2008 13:51

stuartornum is currently offline

stuartornum
Messages: 21
Registered: February 2007

Junior Member

Hi Vasily,

Ive got it!! Actually, bit of a work around, I install the enterprise kernel, and here is meminfo:

cat /proc/meminfo
MemTotal:      1029844 kB
MemFree:        934252 kB
Buffers:         10608 kB
Cached:          47484 kB
SwapCached:          0 kB
Active:          48900 kB
Inactive:        25604 kB
HighTotal:      128544 kB
HighFree:        58476 kB
LowTotal:       901300 kB
LowFree:        875776 kB
SwapTotal:     2104504 kB
SwapFree:      2104504 kB
Dirty:               8 kB
Writeback:           0 kB
AnonPages:       16424 kB
Mapped:          10300 kB
Slab:            11540 kB
PageTables:        876 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   2619424 kB
Committed_AS:    40264 kB
VmallocTotal:   118776 kB
VmallocUsed:      5488 kB
VmallocChunk:   113064 kB

Thank you for all your help, still dont know which module it was, I removed a hell of a lot too..?

Oh well, thanks again.

Report message to a moderator

Send a private message to this user

Switch to threaded view of this topic

Create a new topic

Submit Reply

Previous Topic:	Backup Strategy for Multiple VEs
Next Topic:	How to share a template

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

PDF

]

Current Time: Sat Nov 16 19:04:40 GMT 2024

Total time taken to generate the page: 0.03051 seconds