OpenVZ Forum - RDF feed
type: channel
items: max number of VE
items: Re: max number of VE
items: SOLVED Re: max number of VE
items: Re: SOLVED Re: max number of VE

max number of VE
type: item
description: Hello Gents, i'm new here and like to ask where are the limits of VEs on a host. I try to start 100 on an 16GB dual quad core Dell2950. I build the .conf with vzsplit -n100 but i get at max 49 VEs running. I guess that kmemsize is my problem. How much kemsize is usefull for a verry small VE? thx yahbluez
creator: yahbluez
date: 2007-08-27T21:01:58-00:00

Re: max number of VE
type: item
description: vzsplit is probably not working well enough in your case. Can you tell us what vzmemcheck -v shows? I recommend to stick to shipped configs. I guess you can run a few thousand VEs on such a machine, using the default ve-light.conf.
creator: kir
date: 2007-08-28T00:05:55-00:00

Re: max number of VE
type: item
description: Same Problem. I put together all informations: Dell PE2950, 16GB RAM, 750GB HD, dual E5310 Debian ETCH AMD64 (aka x86_64) The VE OS is a minimal debian etch. I try to run 100 instances. The conf file: # Primary parameters AVNUMPROC="40:50" NUMPROC="80:80" NUMTCPSOCK="60:80" NUMOTHERSOCK="60:80" VMGUARPAGES="1725:9223372036854775807" # Secondary parameters KMEMSIZE="524288:589824" TCPSNDBUF="32768:65536" TCPRCVBUF="32768:65536" OTHERSOCKBUF="32768:65536" DGRAMRCVBUF="32768:32768" OOMGUARPAGES="1725:9223372036854775807" # Auxiliary parameters LOCKEDPAGES="4:4" SHMPAGES="1152:1152" PRIVVMPAGES="20480:40960" NUMFILE="512:512" NUMFLOCK="50:60" NUMPTY="4:4" NUMSIGINFO="256:256" DCACHESIZE="196608:202752" PHYSPAGES="0:9223372036854775807" NUMIPTENT="16:16" # Disk quota parameters (in form of softlimit:hardlimit) DISKSPACE="204800:204800" DISKINODES="80000:88000" QUOTATIME="0" # CPU fair sheduler parameter CPUUNITS="4000" ncl0:~/bin# vzmemcheck -v Output values in % veid LowMem LowMem RAM MemSwap MemSwap Alloc Alloc Alloc util commit util util commit util commit limit 148 0.00 0.01 0.00 0.00 0.02 0.00 0.02 0.50 147 0.00 0.01 0.00 0.00 0.02 0.00 0.02 0.50 % and so on all the same 103 0.00 0.01 0.00 0.00 0.02 0.00 0.02 0.50 102 0.00 0.01 0.00 0.00 0.02 0.00 0.02 0.50 101 0.00 0.01 0.00 0.00 0.02 0.00 0.02 0.50 100 0.00 0.01 0.00 0.00 0.02 0.00 0.02 0.50 ------------------------------------------------------------ ------------- Summary: 0.10 0.60 0.25 0.12 1.14 0.16 1.14 24.31 user_beans: 0: kmemsize 15036287 16002019 9223372036854775807 9223372036854775807 0 lockedpages 0 5205 9223372036854775807 9223372036854775807 0 privvmpages 4387 14309 9223372036854775807 9223372036854775807 0 shmpages 1292 1308 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 numproc 474 483 9223372036854775807 9223372036854775807 0 physpages 2076 3635 9223372036854775807 9223372036854775807 0 vmguarpages 0 0 9223372036854775807 9223372036854775807 0 oomguarp…
creator: yahbluez
date: 2007-08-28T12:31:52-00:00

Re: max number of VE
type: item
description: I see. I got you wrong for the first time. The problem is not with number of VEs that you have -- the problem is kmemsize limits are way too small for those VEs. Just increase those; something like this should do the job: for VE in `vzlist -a -H -oveid`; do vzctl set $VE --kmemsize 1740800:1847296 --save done
creator: kir
date: 2007-08-28T12:40:05-00:00

Re: max number of VE
type: item
description: Same effect as befor. A question, is this typicall: ncl0:/proc/sys# cat /proc/user_beancounters Version: 2.5 uid resource held maxheld barrier limit failcnt 0: kmemsize 4949907 9474391 9223372036854775807 9223372036854775807 0 lockedpages 0 5203 9223372036854775807 9223372036854775807 0 privvmpages 3190 14306 9223372036854775807 9223372036854775807 0 shmpages 652 668 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 numproc 79 140 9223372036854775807 9223372036854775807 0 physpages 1667 3628 9223372036854775807 9223372036854775807 0 vmguarpages 0 0 9223372036854775807 9223372036854775807 0 oomguarpages 1667 3628 9223372036854775807 9223372036854775807 0 numtcpsock 2 2 9223372036854775807 9223372036854775807 0 numflock 1 6 9223372036854775807 9223372036854775807 0 numpty 1 1 9223372036854775807 9223372036854775807 0 numsiginfo 0 3 9223372036854775807 9223372036854775807 0 tcpsndbuf 43336 43336 9223372036854775807 9223372036854775807 0 tcprcvbuf 32768 0 9223372036854775807 9223372036854775807 0 othersockbuf 9248 17536 9223372036854775807 9223372036854775807 0 dgramrcvbuf 0 8456 9223372036854775807 9223372036854775807 0 numothersock 31 37 9223372036854775807 9223372036854775807 0 dcachesize 0 0 9223372036854775807 9223372036854775807 0 numfile 1354 2259 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 numiptent 10 10 9223372036854775807 9223372036854775807 0 I wonder about the barrier and limit values they look verry high? thx
creator: yahbluez
date: 2007-08-29T11:35:03-00:00

Re: max number of VE
type: item
description: 0 beancounter is a host system - it should be unlimited, so high values are ok.
creator: dev
date: 2007-08-29T14:16:06-00:00

Re: max number of VE
type: item
description: what do you mean by got only 49 VEs to run? vzctl start failed on other 51 VEs? or machine started to be slow? or what?
creator: dev
date: 2007-08-29T14:17:01-00:00

Re: max number of VE
type: item
description: yep, these configs have terribly small limits. try to use config after vzsplit -n 20 or 10. It will be a config with overcommitment, but it's ok for your test.
creator: dev
date: 2007-08-29T14:18:37-00:00

Re: max number of VE
type: item
description: I mean the openvz did not start more than 49 instances. Please looke here: I first run 10 Ves than add 30 add up to 49 and than when i try to strat VE nummber 50 it will not do that. I got an error message at syslog. I have 100 identical Ves atthe stoarge so not working ist not based on the expliziet number of the instance. When i have 49 running i can't start more VEs. I still don't see which limit i oversize, maybe someone here sees it: no VE started Version: 2.5 uid resource held maxheld barrier limit failcnt 0: kmemsize 5148172 9474391 9223372036854775807 9223372036854775807 0 lockedpages 0 5203 9223372036854775807 9223372036854775807 0 privvmpages 3190 14306 9223372036854775807 9223372036854775807 0 shmpages 652 668 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 numproc 79 140 9223372036854775807 9223372036854775807 0 physpages 1667 3628 9223372036854775807 9223372036854775807 0 vmguarpages 0 0 9223372036854775807 9223372036854775807 0 oomguarpages 1667 3628 9223372036854775807 9223372036854775807 0 numtcpsock 2 2 9223372036854775807 9223372036854775807 0 numflock 1 6 9223372036854775807 9223372036854775807 0 numpty 1 1 9223372036854775807 9223372036854775807 0 numsiginfo 0 3 9223372036854775807 9223372036854775807 0 tcpsndbuf 43336 43336 9223372036854775807 9223372036854775807 0 tcprcvbuf 32768 0 9223372036854775807 9223372036854775807 0 othersockbuf 9248 27856 9223372036854775807 9223372036854775807 0 dgramrcvbuf 0 8456 9223372036854775807 9223372036854775807 0 numothersock 31 37 9223372036854775807 9223372036854775807 0 dcachesize 0 0 9223372036854775807 9223372036854775807 0 numfile 1352 2259 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 numiptent 10 10 9223372036854775807 9223372036854775807 0 10 VEs running 0: kmemsize 6690269 9474391 9223372036854775807 92233…
creator: yahbluez
date: 2007-08-29T15:59:12-00:00

Re: max number of VE
type: item
description: Is there something bad in dmesg? The exact quote would be appreciated. Also it's not quite clear what OpenVZ kernel version do you use. Have you compiled that yourself? Then what was the version of linux-patch-openvz. Also — can you reproduce the problem with the latest 2.6.18 kernel from http://debian.systs.org/?
creator: kir
date: 2007-08-29T21:24:42-00:00

Re: max number of VE
type: item
description: Hi Kirr, kernel: Linux version 2.6.18-openvz-12-1etch1-amd64 () (ola@deb-build-amd64.openvz.org) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) Not self made. full dmesg: Bootdata ok (command line is root=/dev/mapper/vg-rootfs ro ) Linux version 2.6.18-openvz-12-1etch1-amd64 () (ola@deb-build-amd64.openvz.org) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Fri May 4 23:37:24 MSD 2007 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 00000000000a0000 (usable) BIOS-e820: 0000000000100000 - 00000000cffa8000 (usable) BIOS-e820: 00000000cffa8000 - 00000000cffb7c00 (ACPI data) BIOS-e820: 00000000cffb7c00 - 00000000d0000000 (reserved) BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved) BIOS-e820: 0000000100000000 - 0000000430000000 (usable) DMI 2.4 present. ACPI: RSDP (v002 DELL ) @ 0x00000000000f2620 ACPI: XSDT (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f26a0 ACPI: FADT (v003 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f27a8 ACPI: MADT (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f289c ACPI: SPCR (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f297d ACPI: HPET (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f29cd ACPI: MCFG (v001 DELL PE_SC3 0x00000001 DELL 0x00000001) @ 0x00000000000f2a05 ACPI: DSDT (v001 DELL PE_SC3 0x00000001 MSFT 0x0100000e) @ 0x0000000000000000 No NUMA configuration found Faking a node at 0000000000000000-0000000430000000 Bootmem setup node 0 0000000000000000-0000000430000000 On node 0 totalpages: 4124603 DMA zone: 3027 pages, LIFO batch:0 DMA32 zone: 831464 pages, LIFO batch:31 Normal zone: 3290112 pages, LIFO batch:31 ACPI: PM-Timer IO Port: 0x808 ACPI: Local APIC address 0xfee00000 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) Processor #0 6:15 APIC version 20 ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled) Processor #4 6:15 APIC version 20 ACPI: LAPIC (acpi_id[0…
creator: yahbluez
date: 2007-08-29T21:46:33-00:00

Re: max number of VE
type: item
description: yahbluez wrote on Wed, 29 August 2007 19:59 I mean the openvz did not start more than 49 instances. Please looke here: I first run 10 Ves than add 30 add up to 49 and than when i try to strat VE nummber 50 it will not do that. I got an error message at syslog. what message in syslog? please always report exact info from logs/output from commands when something fails. Otherwise it is impossible to help w/o knowing what fails.
creator: dev
date: 2007-08-30T08:10:16-00:00

Re: max number of VE
type: item
description: Sorry it was at the end of my post, i show you the output of vzmemcheck and vzcpucheck just befor i try to start the next VE. You see the error message and the paste from syslog. It says error -11. I try to move to centos 5.0 instead of etch, maybe that will make things easier? wbr yahbluez ncl0:~/bin# vzmemcheck Output values in % LowMem LowMem RAM MemSwap MemSwap Alloc Alloc Alloc util commit util util commit util commit limit 0.39 3.34 0.82 0.41 1.68 0.59 1.68 24.85 ncl0:~/bin# vzcpucheck Current CPU utilization: 197000 Power of the node: 657217 Try to start VE 49: Starting VPS ... VPS is mounted env_create error: Resource temporarily unavailable VPS start failed /var/log/syslog tells me: Aug 29 17:52:02 ncl0 kernel: VE: 149: stopped Aug 29 17:52:02 ncl0 kernel: VE: 149: failed to start with err=-11 ncl0:~/bin# uname -a Linux ncl0 2.6.18-openvz-12-1etch1-amd64 #1 SMP Fri May 4 23:37:24 MSD 2007 x86_64 GNU/Linux
creator: yahbluez
date: 2007-08-30T09:17:00-00:00

Re: max number of VE
type: item
description: hmm... -11 == EAGAIN... strange... Can you please first try the most recent kernel 012 is too old and I remember there were many fixes... if you can recompile the kernel I could prepare you debug patch which could help to resolve it...
creator: dev
date: 2007-08-30T09:21:55-00:00

Re: max number of VE
type: item
description: I can compile on this server. To do so i need the patch and the information for wich kernel version the patch is. Do you think that it is always a better idea to use a redhat based system for the host themself? I preffer to run debian inside the Ves but outside centos or even SLES will be finde for me too. thx yahbluez
creator: yahbluez
date: 2007-08-30T09:58:18-00:00

Re: max number of VE
type: item
description: Well, I guess you do not need to compile anything (yet). Just add debian.systs.org repository to your apt sources list and update the kernel from that repo. Reboot into that new kernel (should be 2.6.18-028stab039 as of now) and try to reproduce the problem.
creator: kir
date: 2007-08-30T11:26:56-00:00

SOLVED Re: max number of VE
type: item
description: Thx kir, the newest kernel in this rep is: 2.6.18-028stab035.1-ovz-smp #1 SMP Wed Jun 13 21:52:27 CEST 2007 x86_64 GNU/Linux With that kernel the bug did not apear. Im able to run 100 VEs. So no need for me to switch to centos on the host. where may i found the sources of this kernel tree? thanks again for your quick assistance. with best regards yahbluez
creator: yahbluez
date: 2007-08-30T12:53:59-00:00

Re: SOLVED Re: max number of VE
type: item
description: Quote: where may i found the sources of this kernel tree? Why do you need it? Want to rebuild? Why? Sources are at http://download.openvz.org/kernel/branches/2.6.18/028stab035 .1/ or you may use the latest released kernel: http://download.openvz.org/kernel/branches/2.6.18/current/
creator: kir
date: 2007-08-30T13:39:15-00:00

Re: SOLVED Re: max number of VE
type: item
description: to include squashfs. But its possibly to do that on an other machine. thx again for your help. yahbluez
creator: yahbluez
date: 2007-08-30T14:05:18-00:00