OpenVZ Forum


Home » Mailing lists » Users » Reduce Load in VZs
Reduce Load in VZs [message #43168] Wed, 03 August 2011 06:55 Go to next message
<me is currently offline  <me
Messages: 4
Registered: August 2011
Junior Member
From: *parallels.com
Hello,
We have on our Server about 17VZs installed and running. Most of them run
simmilar cron jobs, e.g. apticron and this causes very high load at night.

Is there any best practise that we can use to reduce the load with these
jobs? I tought about running these cronjobs from outside with vzctl exec
(so make a cron at the host and just start the programms manually) and run
only 1 job at once.

In general: Is there a best practise to run things in every VZ and not
produce high load?

Thanks!

greetings
Re: Reduce Load in VZs [message #43169 is a reply to message #43168] Wed, 03 August 2011 07:11 Go to previous messageGo to next message
Tim Small is currently offline  Tim Small
Messages: 24
Registered: April 2011
Junior Member
From: *parallels.com
On 03/08/11 07:55, me@free-minds.net wrote:
> Hello,
> We have on our Server about 17VZs installed and running. Most of them run
> simmilar cron jobs, e.g. apticron and this causes very high load at night.
>
> Is there any best practise that we can use to reduce the load with these
> jobs? I tought about running these cronjobs from outside with vzctl exec
> (so make a cron at the host and just start the programms manually) and run
> only 1 job at once.
>
> In general: Is there a best practise to run things in every VZ and not
> produce high load?
>
>

I patched the openvz scripts to randomise the start-time of these
scripts across containers. One of my clients had the same problem - it
showed up quite clearly using the openvzcpu munin plugin which I wrote...

I'll dig out the patch later. Bug me if I haven't within 24 hours.

Tim.

--
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53 http://seoss.co.uk/ +44-(0)1273-808309
Re: Reduce Load in VZs [message #43170 is a reply to message #43169] Wed, 03 August 2011 07:47 Go to previous messageGo to next message
kir is currently offline  kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia
Senior Member

From: *parallels.com
On 08/03/2011 11:11 AM, Tim Small wrote:
> On 03/08/11 07:55, me@free-minds.net wrote:
>> Hello,
>> We have on our Server about 17VZs installed and running. Most of them run
>> simmilar cron jobs, e.g. apticron and this causes very high load at night.
>>
>> Is there any best practise that we can use to reduce the load with these
>> jobs? I tought about running these cronjobs from outside with vzctl exec
>> (so make a cron at the host and just start the programms manually) and run
>> only 1 job at once.
>>
>> In general: Is there a best practise to run things in every VZ and not
>> produce high load?
>>
>>
> I patched the openvz scripts to randomise the start-time of these
> scripts across containers. One of my clients had the same problem - it
> showed up quite clearly using the openvzcpu munin plugin which I wrote...
>
> I'll dig out the patch later. Bug me if I haven't within 24 hours.
>
> Tim.
>
Interesting. vzctl does crontab randomization, by means of
/etc/vz/dists/scripts/vps-postcreate.sh
script, so the question is why it is not working. I can see a few
reasons why it's not working as it should:

(1) CT was created not by vzctl
(2) CT has been upgraded from the inside, so /etc/crontab was being modified
(3) CT has a distro installed which has crontab in some unusual location
(4) Some other bug?


Kir Kolyshkin
http://static.openvz.org/userbars/openvz-developer.png
Re: Reduce Load in VZs [message #43171 is a reply to message #43170] Wed, 03 August 2011 08:10 Go to previous messageGo to next message
<me is currently offline  <me
Messages: 4
Registered: August 2011
Junior Member
From: *parallels.com
On Wed, 03 Aug 2011 11:47:05 +0400, Kir Kolyshkin <kir@openvz.org> wrote:
> Interesting. vzctl does crontab randomization, by means of
> /etc/vz/dists/scripts/vps-postcreate.sh
> script, so the question is why it is not working. I can see a few
> reasons why it's not working as it should:
>
> (1) CT was created not by vzctl
mh it was, all of them are created by this script
> (2) CT has been upgraded from the inside, so /etc/crontab was being
> modified
yes, the cronjobs are under /etc/cron.{daily,weeky,...}
> (3) CT has a distro installed which has crontab in some unusual location
no its running debian lenny or squeeze or sid
> (4) Some other bug?
mh could be, we try now different things to encounter the problem... seems
very complicated at the moment because we have really no idea whats going
on there

greetings!
Re: [PATCH] Reduce Load in VZs - stop cron.d thundering herd... [message #43172 is a reply to message #43170] Wed, 03 August 2011 08:29 Go to previous messageGo to next message
Tim Small is currently offline  Tim Small
Messages: 24
Registered: April 2011
Junior Member
From: *parallels.com
On 03/08/11 08:47, Kir Kolyshkin wrote:
>
> (3) CT has a distro installed which has crontab in some unusual location

Nearly (3)

/etc/cron.d is a standard location, but vps-postcreate.sh didn't deal
with it.

I thought I'd submitted this patch to debian, but I seem to have dropped
it :-(

Tim.

--- postcreate.sh.old 2010-09-14 06:07:31.000000000 +0100
+++ /etc/vz/dists/scripts/postcreate.sh 2011-08-03 09:27:22.490356610 +0100
@@ -24,12 +24,12 @@

function randcrontab()
{
- file=${VE_ROOT}"/etc/crontab"
+ for file in ${VE_ROOT}/etc/cron.d/* ${VE_ROOT}/etc/crontab
+ do
+ [ -f "${file}" ] || return 0

- [ -f "${file}" ] || return 0
-
- /bin/cp -fp ${file} ${file}.$$
- cat ${file} | awk '
+ /bin/cp -fp ${file} ${file}.$$
+ cat ${file} | awk '
BEGIN { srand(); }
{
if ($0 ~ /^[ \t]*#/ || $0 ~ /^[ \t]+*$/) {
@@ -61,8 +61,9 @@
}
print line;
}
-' > ${file}.$$ && /bin/mv -f ${file}.$$ ${file}
- /bin/rm -f ${file}.$$ 2>/dev/null
+ ' > ${file}.$$ && /bin/mv -f ${file}.$$ ${file}
+ /bin/rm -f ${file}.$$ 2>/dev/null
+ done
}

function disableroot()





--
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53 http://seoss.co.uk/ +44-(0)1273-808309
Re: Reduce Load in VZs [message #43173 is a reply to message #43171] Wed, 03 August 2011 08:32 Go to previous messageGo to next message
Tim Small is currently offline  Tim Small
Messages: 24
Registered: April 2011
Junior Member
From: *parallels.com
On 03/08/11 09:10, me@free-minds.net wrote:
>
>> (2) CT has been upgraded from the inside, so /etc/crontab was being
>> modified
>>
> yes, the cronjobs are under /etc/cron.{daily,weeky,...}
>

Compare /etc/crontab between containers - the entries should have
different (randomized) times:

# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts
--report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts
--report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts
--report /etc/cron.monthly )



... if they are, then perhaps it's /etc/cron.d/* tabs instead? This was
the problem on my client's system.

Tim.


--
South East Open Source Solutions Limited
Registered in England and Wales with company number 06134732.
Registered Office: 2 Powell Gardens, Redhill, Surrey, RH1 1TQ
VAT number: 900 6633 53 http://seoss.co.uk/ +44-(0)1273-808309
Re: [PATCH] Reduce Load in VZs - stop cron.d thundering herd... [message #43174 is a reply to message #43172] Wed, 03 August 2011 08:33 Go to previous messageGo to next message
&lt;me is currently offline  &lt;me
Messages: 4
Registered: August 2011
Junior Member
From: *parallels.com
awesome thanks, i'll try it. So best is i copy all the stuff from
cron.daily to cron.d and add 0 0 * * * ar @daily at the beginning, or is it
the same to leave them there and these things will allready called by cron?

On Wed, 03 Aug 2011 09:29:32 +0100, Tim Small <tim@seoss.co.uk> wrote:
> On 03/08/11 08:47, Kir Kolyshkin wrote:
>>
>> (3) CT has a distro installed which has crontab in some unusual
location
>
> Nearly (3)
>
> /etc/cron.d is a standard location, but vps-postcreate.sh didn't deal
> with it.
>
> I thought I'd submitted this patch to debian, but I seem to have dropped
> it :-(
>
> Tim.
>
> --- postcreate.sh.old 2010-09-14 06:07:31.000000000 +0100
> +++ /etc/vz/dists/scripts/postcreate.sh 2011-08-03 09:27:22.490356610
+0100
> @@ -24,12 +24,12 @@
>
> function randcrontab()
> {
> - file=${VE_ROOT}"/etc/crontab"
> + for file in ${VE_ROOT}/etc/cron.d/* ${VE_ROOT}/etc/crontab
> + do
> + [ -f "${file}" ] || return 0
>
> - [ -f "${file}" ] || return 0
> -
> - /bin/cp -fp ${file} ${file}.$$
> - cat ${file} | awk '
> + /bin/cp -fp ${file} ${file}.$$
> + cat ${file} | awk '
> BEGIN { srand(); }
> {
> if ($0 ~ /^[ \t]*#/ || $0 ~ /^[ \t]+*$/) {
> @@ -61,8 +61,9 @@
> }
> print line;
> }
> -' > ${file}.$$ && /bin/mv -f ${file}.$$ ${file}
> - /bin/rm -f ${file}.$$ 2>/dev/null
> + ' > ${file}.$$ && /bin/mv -f ${file}.$$ ${file}
> + /bin/rm -f ${file}.$$ 2>/dev/null
> + done
> }
>
> function disableroot()
Re: Reduce Load in VZs [message #43175 is a reply to message #43173] Wed, 03 August 2011 08:35 Go to previous message
&lt;me is currently offline  &lt;me
Messages: 4
Registered: August 2011
Junior Member
From: *parallels.com
Hi,
Ah yes, Im a fool :D - forget the mail from before...

i'll see what i can get there and will try the patch!

thank you both!

greetings

On Wed, 03 Aug 2011 09:32:47 +0100, Tim Small <tim@seoss.co.uk> wrote:
> On 03/08/11 09:10, me@free-minds.net wrote:
>>
>>> (2) CT has been upgraded from the inside, so /etc/crontab was being
>>> modified
>>>
>> yes, the cronjobs are under /etc/cron.{daily,weeky,...}
>>
>
> Compare /etc/crontab between containers - the entries should have
> different (randomized) times:
>
> # m h dom mon dow user command
> 17 * * * * root cd / && run-parts --report /etc/cron.hourly
> 25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts
> --report /etc/cron.daily )
> 47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts
> --report /etc/cron.weekly )
> 52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts
> --report /etc/cron.monthly )
>
>
>
> ... if they are, then perhaps it's /etc/cron.d/* tabs instead? This was
> the problem on my client's system.
>
> Tim.
Previous Topic: Puppet module for OpenVZ management
Next Topic: Question About /proc file system in VMs.
Goto Forum:
  


Current Time: Mon Dec 09 05:38:09 GMT 2019