| Limiting CPU for tasks on HN ? [message #33188] |
Sat, 27 September 2008 10:09  |
geejay
Messages: 11
Registered: May 2008
|
Junior Member |
|
|
Hello,
It appears that CPU intensive tasks on the hardware node do kill completely the performance on the VEs.
E.g. Simply copying a big directory like a VE private directory on the hardware node, bringing CPU load to 20-25% or so, completely kills responsiveness of my VEs, making them basically unusuable.
Starting tasks with nice -9 doesnt help either.
Is there any way to limit the CPU usage of the Hardware node so that VEs can continue running ? Or can one guarantee minimum CPU time in VEs ?
As it stands I am unable to do essential maintenance tasks on the HN because VEs get effectively killed.
TIA for any input on the matter
I am running Centos 5
2.6.18-92.1.1.el5.028stab057.2
Geejay
|
|
|
|
|
|
| Re: Limiting CPU for tasks on HN ? [message #33202 is a reply to message #33199] |
Sun, 28 September 2008 06:24  |
geejay
Messages: 11
Registered: May 2008
|
Junior Member |
|
|
Maratrus,
vzcpucheck -v
VEID CPUUNITS
-------------------------
0 1000
102 83333
Current CPU utilization: 84333
Power of the node: 159866
there is only one VPS on the HN which has been allocated 80000 CPUUNITS
If I do a copy of several GB on the HN the VPS becomes unresponsive. The VPS is running a VoIP application, Asterisk.
SIP clients cannot initiate new calls while the copying on the HN happens. Typically the VPS would use maximum 2% CPU to initiate a call. So there should not be any problem with resources. Beancounters also show that everything is running fine. Also I can see that the VPS receives the SIP messages over the network, but Asterisk stalls and does not process the call. IMVHO the CPU scheduler does not work.
Something is definitely not working as suggested in the Openvz manual.
How can debug this problem ?
I should perhaps mention that the VPS was converted from a physical machine. On startup it throws the following error:
Sep 28 08:26:49 onl3 sysctl: error: unknown error 1 setting key 'kernel.sysrq'
Sep 28 08:26:49 onl3 sysctl: error: unknown error 1 setting key 'kernel.core_uses_pid'
Sep 28 08:26:49 online rc.sysinit: Configuring kernel parameters: failed
However on the HN systcl shows no errors:
sysctl -p
net.ipv4.ip_forward = 1
kernel.shmmax = 4294967295
kernel.msgmax = 65536
kernel.msgmnb = 65536
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.default.proxy_arp = 0
kernel.sysrq = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.send_redirects = 0
kernel.shmall = 268435456
net.ipv4.conf.default.send_redirects = 1
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
102: kmemsize 3265249 4588138 11055923 11377049 0
lockedpages 0 0 3000 3000 0
privvmpages 18487 34351 250000 250000 0
shmpages 1293 1309 21504 21504 0
dummy 0 0 0 0 0
numproc 57 67 240 240 0
physpages 9821 24879 0 2147483647 0
vmguarpages 0 0 33792 2147483647 0
oomguarpages 9821 24879 200000 2147483647 0
numtcpsock 9 23 360 360 0
numflock 4 10 188 206 0
numpty 2 2 16 16 0
numsiginfo 0 9 256 256 0
tcpsndbuf 87204 229160 1720320 2703360 0
tcprcvbuf 147456 326676 1720320 2703360 0
othersockbuf 34240 174280 1126080 2097152 0
dgramrcvbuf 0 32412 500000 500000 0
numothersock 22 32 360 360 0
dcachesize 0 0 3409920 3624960 0
numfile 1389 1546 9312 9312 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 26 26 128 128 0
[Updated on: Sun, 28 September 2008 06:36] Report message to a moderator |
|
|
|
|
|
| Re: Limiting CPU for tasks on HN ? [message #33217 is a reply to message #33188] |
Mon, 29 September 2008 16:13 |
kenjy
Messages: 50
Registered: July 2008
Location: Mexico
|
Member |
|
|
OpenVz works with the kernel scheduler, you can set cpuunits for giving priority's like this:
root@windu:~# vzcpucheck -v
VEID CPUUNITS
-------------------------
0 1000
101 4000
102 2000
103 3012
Current CPU utilization: 13024
Power of the node: 221180
a highest value means more power.
http://kenjy.net |Live net !!!
|
|
|
|
|
|
| Re: Limiting CPU for tasks on HN ? [message #33714 is a reply to message #33188] |
Sun, 02 November 2008 19:05 |
seanfulton
Messages: 105
Registered: May 2007
|
Senior Member |
|
|
I am experiencing a similar problem. We moved our mail server from a physical machine with 2 G of RAM and 2 G of SWAP into a virtual machine with essentially the same resources.
Nightly backup causes the mail server activity to drop to nothing, causing back-ups and major issues. I have spent about a month trouble-shooting this and verified that there is nothing within the VE that is causing the hang up.
When I turn off the nightly backup, no problems. When I run one, everything hangs.
Two things I have noticed which are strange: there are no errors inside the VE, just more and more sendmail processes launching, but the old ones don't close.
Second, it is possible to allocate more RAM to a VE than the system has, causing the HN to crash. I opened up all of the memory settings for the VE until the HN crashed. Had to back off the settings to keep it up. I know when the machine was running on a physical machine, it would kill off processes as the system ran out of ram (and swap). This activity happens within the VE itself. But if you accidentally tell the VE it has 8 G available but it only has 6G, it will cause a kernel panic the HN once it exhausts the 6 G.
BTW: Nightly backup is vzdump to an NFS mount. THis is how I am set (2202 is the sendmail VE):
[root@gcn16 ~]# vzcpucheck -v
VEID CPUUNITS
-------------------------
0 1000
2202 100000
6005 1000
Current CPU utilization: 102000
Power of the node: 232811
[root@gcn16 ~]#
sean
[Updated on: Sun, 02 November 2008 19:07] Report message to a moderator |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Limiting CPU for tasks on HN ? [message #33794 is a reply to message #33745] |
Mon, 10 November 2008 16:06 |
seanfulton
Messages: 105
Registered: May 2007
|
Senior Member |
|
|
| seanfulton wrote on Thu, 06 November 2008 11:19 |
We have four machines running OpenVZ, various kernels. The one in question is running:
2.6.18-53.1.19.el5.028stab053.14PAE
I can't say for sure if this happens on our other machines because those are mostly web sites and we run the backups at night when traffic is low, so it's possible the problem just isn't affecting anything. The mail server is obvious because that tends to churn 24/7 so whenever I run my backups (using vzdump) everything hangs and backs up, causing it to eventually crash in the AM.
One thought I did have though is that even though the crash is during a period of high IO activity, the sendmail processes hang, stack up and eventually lock up the VE. When the backup completes, the sendmail activity doesn't seem to go back to normal--it seems to be permanently locked and just keeps spawning new processes until it crashes (usually at about 1800 processes).
This weekend I will upgrade the machine to the latest kernel and see if that makes a difference. I will report back.
sean
|
Update: I upgraded to the 57.2 kernel and am having the same problems.
I was not able to update to the 59.3 kernel because there is a problem initializing the quota in that kernel.
Any ideas?
sean
|
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
