OpenVZ Forum: Support » Need help with CPU shortage

Home » General » Support » Need help with CPU shortage

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Need help with CPU shortage [message #36414]

Wed, 17 June 2009 09:08

dvazart
Messages: 37
Registered: October 2008
Location: France

Member

Hi !

I'm running a 2.6.18-14-ovz-686-enterprise kernel under Debian Etch. I have about 150 VEs running in 2 quad core Intel Xeon prosessors.

My problem is that the processors reach 100% for several minutes and very random, with htop i can see that is the kernel which saturates the CPU (CPU utilization by the kernel is marked in red):

http://lh6.ggpht.com/_8Lj7RDSnhB4/Sjiqwz3umYI/AAAAAAAAEdg/LF5M5MEeumM/s144/openvz-100cpu.png

http://picasaweb.google.com/lh/photo/9npqR8JyIvoXy4r2z-4Adw? feat=directlink

I also graph the CPU utilization with Munin, but the only value that reaches 100% is the iowait:

http://lh4.ggpht.com/_8Lj7RDSnhB4/Sjiv49d0pKI/AAAAAAAAEdk/aAbCnaSNc-Y/s144/munin-iowait.png

http://picasaweb.google.com/lh/photo/jUNQXmiO0x7JMkxJSLUv6w? feat=directlink

I have tried to reduce the VE's parameters to:

CPUUNITS="2000"
CPULIMIT="2"
CPUS="1"

... But still happening!

I really need help with this.

Thaks.

----------- Daniel Vazart ------------
"Knowledge is power, Sharing is human"
------- http://www.vazart.net --------

Report message to a moderator

Re: Need help with CPU shortage [message #36417 is a reply to message #36414]

Wed, 17 June 2009 13:14

maratrus
Messages: 1495
Registered: August 2007
Location: Moscow

Senior Member

Hello,

what is the output of vzcpucheck utility?

Report message to a moderator

Re: Need help with CPU shortage [message #36418 is a reply to message #36414]

Wed, 17 June 2009 13:21

dvazart
Messages: 37
Registered: October 2008
Location: France

Member

Hi !

Thanks for your reply... there is my vzcpucheck output :

Current CPU utilization: 404876
Power of the node: 799850

----------- Daniel Vazart ------------
"Knowledge is power, Sharing is human"
------- http://www.vazart.net --------

Report message to a moderator

Re: Need help with CPU shortage [message #36419 is a reply to message #36418]

Wed, 17 June 2009 13:57

maratrus
Messages: 1495
Registered: August 2007
Location: Moscow

Senior Member

Hi,
a few words about CPULIMITs.
What purpose do we pursue restricting VEs CPULIMIT? Not allow a particular VE to use a free CPU? So, in case of intensive CPU consumption on the part of VEs, CPUUNITS are the main restriction that balances the system load. In case of low CPU consumption we would be likely restricted by CPULIMITS (if they don't equal to zero) while actually CPU would be idle. Thus, it's not clear what kind of benefit we'll obtain by setting up CPULIMIT parameters. Moreover, the situation may occur when a particular VE holds some kind of locks (and other VEs wait these locks too) but cannot release them because of the CPULIMIT restriction. But it worth saying that in case of CPULIMITS=0 this situation may occur too (hence, with less probability). So, if somebody wants to set up CPULIMIT parameters it's of course not prohibited but it worth bearing in mind the above arguments. May be worth disabling CPULIMITs at all?

The first provided output of htop utility shows rather big load average value. There are either processes that couldn't obtain CPU (e.g. because of the CPULIMIT) or processes in D-state producing a lot of i/o operations. BTW, does htop takes into account processes that are in D-state? Is it worth either tuning it or comparing htop output with 'top' or/and 'ps' outputs?

htop output indicates that "minicom-cron", "php script" and "getty" consume 100% CPU. Is it their permanent behavior? Why do they consume so much CPU?

Report message to a moderator

Re: Need help with CPU shortage [message #36420 is a reply to message #36414]

Wed, 17 June 2009 15:02

dvazart
Messages: 37
Registered: October 2008
Location: France

Member

Thanks for the explanation.

Finally, noting the times at which the CPU is saturated, I find a crontab from ISPConfig (check_services.php) we have about 100 VEs with ISPconfig installed, so every 30 minutes the CPU reaches the 100% and all the ISPconfig crontabs are running... (now i have to find how to edit the crontab with a shell script in debian).

But your explanation about CPULIMITs its very interesting... so you think we can have 200 VEs running in the same server without CPULIMITS ? We want to have 4 diferents plans of VPS with diferent resources, in our case do you think its possible without CPULIMITS ? and what about the number of CPUS for each VE?

Quote:

The first provided output of htop utility shows rather big load average value. There are either processes that couldn't obtain CPU (e.g. because of the CPULIMIT) or processes in D-state producing a lot of i/o operations. BTW, does htop takes into account processes that are in D-state? Is it worth either tuning it or comparing htop output with 'top' or/and 'ps' outputs?

So in this case its probably that the script (check_services.php) causes the iowait that we can see in the second image ?
( http://picasaweb.google.com/lh/photo/jUNQXmiO0x7JMkxJSLUv6w? feat=directlink)

thaks..

----------- Daniel Vazart ------------
"Knowledge is power, Sharing is human"
------- http://www.vazart.net --------

Report message to a moderator

Re: Need help with CPU shortage [message #36426 is a reply to message #36420]

Thu, 18 June 2009 08:13

maratrus
Messages: 1495
Registered: August 2007
Location: Moscow

Senior Member

Hi,

Quote:

so you think we can have 200 VEs running in the same server without CPULIMITS ? We want to have 4 diferents plans of VPS with diferent resources, in our case do you think its possible without CPULIMITS ? and what about the number of CPUS for each VE?

If you're going to set up a particular configuration nobody can prevent you to realize your plans. Smile

It's not recommended using such little values as it was shown in a first post.

Quote:

CPULIMIT="2"

The problems you might bump into described in my previous message. Just keep them in mind.
To avoid the potential problems it has been recommended you to switch off CPULIMITS at all but you might just increase them using cut-and-try method. The same may be said about CPUS variable except for the additional note - CPULIMIT=100 is worse than CPUS=1 (without CPULIMIT at all)
It's also recommended not to use CPUUNITS inside VE which can differ more that in times.

Report message to a moderator

Re: Need help with CPU shortage [message #36427 is a reply to message #36414]

Thu, 18 June 2009 08:43

dvazart
Messages: 37
Registered: October 2008
Location: France

Member

OK....

So CPUS override the CPULIMIT variable ? and if i have something like:

CPULIMIT="40"
CPUS="1"

Thats mean the 40% of 1 CPU ?

And another question... i have installed vzsplit (but I do not know the version), you think i can trust in it ?

Thanks !

----------- Daniel Vazart ------------
"Knowledge is power, Sharing is human"
------- http://www.vazart.net --------

Report message to a moderator

Re: Need help with CPU shortage [message #36428 is a reply to message #36427]

Thu, 18 June 2009 08:59

maratrus
Messages: 1495
Registered: August 2007
Location: Moscow

Senior Member

Quote:

So CPUS override the CPULIMIT variable ?

Not sure what you have in mind. I'd rather give an example. Let's assume that HN has 2 CPUs.
The first configuration:
- CPULIMIT=100, CPUS is not set.
In this case a CPU-intensive task may assume no more than 100% of a single CPU but CPU itself is not specified so the task might be migrated from one CPU to another quite frequent. Migration from one to another CPU is a very expensive operation when talking about performance.

- CPUS=1, CPULIMIT is not set.
In this case a CPU-intensive task may assume no more than 100% of a single CPU which is specified quite well with CPUS variable so the task is not migrated between different CPUS and this behavior eliminates a lot of performance losses.

Quote:

i have installed vzsplit (but I do not know the version), you think i can trust in it ?

I think yes. It constructs a basic configuration which you can tune afterward.

Report message to a moderator

Re: Need help with CPU shortage [message #36431 is a reply to message #36414]

Thu, 18 June 2009 11:25

dvazart
Messages: 37
Registered: October 2008
Location: France

Member

Ok, that clarifies the difference between CPU and CPULIMIT.

But at this point i dont know which of the two should I use...

I have 2 quad core Xeon Processors for a total of 8 CPUs, and i want to run at least 200 VEs :

If i set CPUS="1" for each VE, i'm affraid the server will saturate.
CPULIMIT seems to be the better solution for me, but i'm affraid to lose performances.
And finally i dont know if leave the VEs without limits its a good idea.

Confused

confusing...

----------- Daniel Vazart ------------
"Knowledge is power, Sharing is human"
------- http://www.vazart.net --------

Report message to a moderator

Re: Need help with CPU shortage [message #36432 is a reply to message #36431]

Thu, 18 June 2009 11:48

maratrus
Messages: 1495
Registered: August 2007
Location: Moscow

Senior Member

I think you'd better
- tune CPUUNITS in an appropriate way so that HN receives at least 5%-10% ( OpenVZ-Users-Guilde page 53)
- remove all CPULIMITS (or make them not to be too small).
- think about decreasing the number of VEs Smile

Report message to a moderator

Re: Need help with CPU shortage [message #36434 is a reply to message #36414]

Thu, 18 June 2009 11:55

dvazart
Messages: 37
Registered: October 2008
Location: France

Member

Thank you very much for your help!

----------- Daniel Vazart ------------
"Knowledge is power, Sharing is human"
------- http://www.vazart.net --------

Report message to a moderator

Re: Need help with CPU shortage [message #36437 is a reply to message #36414]

Thu, 18 June 2009 13:33

dvazart
Messages: 37
Registered: October 2008
Location: France

Member

Hi !

The last questions :

I'm going to make a shell script with "for" to change the parameters of all VEs, there are some command to disable the CPU variable?

Something like :

vzctl set $VE --cpulimit 0 --cpus 0 --save

The manpage of vzctl only shows :

Quote:

--cpus num
sets number of CPUs available in the container.

And, there are some vzctl command to set the VE0CPUUNITS variable? or i have to set it editing the vz.conf file? I guess i have to restart the daemon after edit VE0CPUUNITS ?

----------- Daniel Vazart ------------
"Knowledge is power, Sharing is human"
------- http://www.vazart.net --------

Report message to a moderator

Previous Topic:	Getting OpenVZ container to run at 1000MB (Network speed) instead of 100MB
Next Topic:	Best bandwidth monitoring method?

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Mon Nov 11 13:52:08 GMT 2024

Total time taken to generate the page: 0.03622 seconds