OpenVZ Forum - RDF feed
type: channel
items: Is there a stable OpenVZ kernel, and which should be fit for production
items: Re: Is there a stable OpenVZ kernel, and which should be fit for production

Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Hello, since 2.6.32 branch is no longer maintained: " Also, from now (30 August 2011) we no longer maintain the following kernel branches: * 2.6.27 * 2.6.32 " we have switched to RHEL6 branch, which seems to run fine, and solves some long-running problems with 2.6.32 ( vSwap, problem with accounting of mmaped files usage ). All was nice until some heavier loaded servers came online with RHEL6, and - they started crashing. And then came the upgrade train: stab036.1 => stab037.1 => stab039.10 => stab040.1 => stab042.1 etc With one of the problems we caught, we were told to switch from stable to testing kernels ( now I see that that testing kernel later became stable, so while confusing, it makes some sense ). All those kernels ( and stab039.11, which from description should be latest stable ) exhibit the same problem/class of problems - when put under stress, they crash. It's quite easy to recreate, now that we've spent some time tracking it down, just start the machine with for example: stress --cpu 12 --io 16 --vm 32 -d 24 --hdd-bytes 10G and maybe bonnie++ running in loop, and in few minutes/few hours you've got dead machines spewing something like: [ 1515.249585] BUG: scheduling while atomic: stress/2054/0xffff8800 [ 1515.250189] BUG: unable to handle kernel paging request at fffffffc047118e0 [ 1515.250189] IP: [<ffffffff8105620e>] account_system_time+0x9e/0x1f0 [ 1515.250189] PGD 1a27067 PUD 0 [ 1515.250189] Thread overran stack, or stack corrupted [ 1515.250189] Oops: 0000 [#1] SMP or maybe: [ 1876.747809] BUG: unable to handle kernel paging request at 00000006000000bd [ 1876.747815] IP: [<ffffffff8105a4fe>] select_task_rq_fair+0x32e/0xa20 [ 1876.747823] PGD 12d089067 PUD 0 [ 1876.747826] Oops: 0000 [#1] SMP or [38764.623677] BUG: unable to handle kernel paging request at 000000000001e440 [38764.623677] IP: [<ffffffff814c8efe>] _spin_lock+0xe/0x30 [38764.623677] PGD 12c7b4067 PUD 12c7b5067 PMD 0 [38764.623677] Oops: 0002 [#2] SMP [38764.623677] last sysfs file: …
creator: Dariush Pietrzak
date: 2011-11-22T08:52:18-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: On 11/22/2011 12:52 PM, Dariush Pietrzak wrote: > Hello, > since 2.6.32 branch is no longer maintained: > " > Also, from now (30 August 2011) we no longer maintain the following kernel branches: > > * 2.6.27 > * 2.6.32 > " > we have switched to RHEL6 branch, which seems to run fine, and solves some > long-running problems with 2.6.32 ( vSwap, problem with accounting of > mmaped files usage ). > All was nice until some heavier loaded servers came online with RHEL6, and > - they started crashing. And then came the upgrade train: > stab036.1 => stab037.1 => stab039.10 => stab040.1 => stab042.1 etc > > With one of the problems we caught, we were told to switch from stable to > testing kernels ( now I see that that testing kernel later became stable, > so while confusing, it makes some sense ). > All those kernels ( and stab039.11, which from description should be > latest stable ) exhibit the same problem/class of problems - when put under > stress, they crash. > It's quite easy to recreate, now that we've spent some time tracking it down, > just start the machine with for example: > > stress --cpu 12 --io 16 --vm 32 -d 24 --hdd-bytes 10G > and maybe bonnie++ running in loop, and in few minutes/few hours you've got > dead machines spewing something like: > > [ 1515.249585] BUG: scheduling while atomic: stress/2054/0xffff8800 > [ 1515.250189] BUG: unable to handle kernel paging request at fffffffc047118e0 > [ 1515.250189] IP: [<ffffffff8105620e>] account_system_time+0x9e/0x1f0 > [ 1515.250189] PGD 1a27067 PUD 0 > [ 1515.250189] Thread overran stack, or stack corrupted > [ 1515.250189] Oops: 0000 [#1] SMP > > or maybe: > [ 1876.747809] BUG: unable to handle kernel paging request at 00000006000000bd > [ 1876.747815] IP: [<ffffffff8105a4fe>] select_task_rq_fair+0x32e/0xa20 > [ 1876.747823] PGD 12d089067 PUD 0 > [ 1876.747826] Oops: 0000 [#1] SMP > > or > [38764.623677] BUG: unable to handle kernel paging request at 000000000001e440 > [38764.623677] IP: [<ffffffff814c8efe>] …
creator: kir
date: 2011-11-22T10:34:59-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Zitat von Kir Kolyshkin <kir@openvz.org>: > On 11/22/2011 12:52 PM, Dariush Pietrzak wrote: >> Hello, >> since 2.6.32 branch is no longer maintained: >> " >> Also, from now (30 August 2011) we no longer maintain the following >> kernel branches: >> >> * 2.6.27 >> * 2.6.32 >> " >> we have switched to RHEL6 branch, which seems to run fine, and solves some >> long-running problems with 2.6.32 ( vSwap, problem with accounting of >> mmaped files usage ). >> All was nice until some heavier loaded servers came online with RHEL6, and >> - they started crashing. And then came the upgrade train: >> stab036.1 => stab037.1 => stab039.10 => stab040.1 => stab042.1 etc >> >> With one of the problems we caught, we were told to switch from stable to >> testing kernels ( now I see that that testing kernel later became stable, >> so while confusing, it makes some sense ). >> All those kernels ( and stab039.11, which from description should be >> latest stable ) exhibit the same problem/class of problems - when put under >> stress, they crash. >> It's quite easy to recreate, now that we've spent some time >> tracking it down, >> just start the machine with for example: >> >> stress --cpu 12 --io 16 --vm 32 -d 24 --hdd-bytes 10G >> and maybe bonnie++ running in loop, and in few minutes/few hours you've got >> dead machines spewing something like: >> >> [ 1515.249585] BUG: scheduling while atomic: stress/2054/0xffff8800 >> [ 1515.250189] BUG: unable to handle kernel paging request at >> fffffffc047118e0 >> [ 1515.250189] IP: [<ffffffff8105620e>] account_system_time+0x9e/0x1f0 >> [ 1515.250189] PGD 1a27067 PUD 0 >> [ 1515.250189] Thread overran stack, or stack corrupted >> [ 1515.250189] Oops: 0000 [#1] SMP >> >> or maybe: >> [ 1876.747809] BUG: unable to handle kernel paging request at >> 00000006000000bd >> [ 1876.747815] IP: [<ffffffff8105a4fe>] select_task_rq_fair+0x32e/0xa20 >> [ 1876.747823] PGD 12d089067 PUD 0 >> [ 1876.747826] Oops: 0000 [#1] SMP >> >> or >> [38764.623677] BUG: un…
creator: MailingListe
date: 2011-11-23T10:52:11-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: > I am very sad to hear this. Could you please file a bug to > bugzilla.openvz.org so our kernel guys will start working on that? Looking at bugzilla there are many other similiar reports, one of mine has been closed as fixed, but then returned in exactly the same function after just 6 minutes of stress-testing new kernel. It's easy to reproduce, just put enough load on the system. It looks really troubling, both vSwap and 042.x branches look very nice feature-wise, even vzmigrate seems to work fine, which is no small feat, but it kinda feels like stability has been sacrificed to get there. best regards, Eyck -- Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9 Total Existance Failure
creator: Dariush Pietrzak
date: 2011-11-23T12:31:52-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: On 11/23/2011 04:31 PM, Dariush Pietrzak wrote: >> I am very sad to hear this. Could you please file a bug to >> bugzilla.openvz.org so our kernel guys will start working on that? > Looking at bugzilla there are many other similiar reports, one of mine has > been closed as fixed, but then returned in exactly the same function after > just 6 minutes of stress-testing new kernel. > It's easy to reproduce, just put enough load on the system. Have you reopened it already? Can you provide bug number? > > It looks really troubling, both vSwap and 042.x branches look very nice > feature-wise, even vzmigrate seems to work fine, which is no small feat, > but it kinda feels like stability has been sacrificed to get there. > > best regards, Eyck Guys, I do understand reasons for your frustration, but so far I have only seen one specific bug mentioned in this thread, namely http://bugzilla.openvz.org/2095 it was filed yesterday and there is a patch already available for testing. Any other statements like "there are many bugs", "this kernel is unstable" are just not specific enough for me to deal with. If there are bugs, they need to be reported and fixed, and we, OpenVZ team, partly rely on you, our users. We do have internal QA but can't possibly test all the use cases and scenarios. Specifically, we rely on having bug reports from you, with full kernel logs (see http://wiki.openvz.org/Remote_console_setup), test cases (as specific and reproducible as possible), and ideally your ability to test patches that developers provide and report your results back to bugzilla. We treat bug reports very seriously, and we do our best to reproduce your bugs locally and fix them. Again, please be specific and refer to exact bugs in bugzilla when you are having problems with kernel stability, otherwise it's not helpful and I can't do much about it. Kir.
creator: kir
date: 2011-11-23T14:42:25-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Zitat von Kir Kolyshkin <kir@openvz.org>: > On 11/23/2011 04:31 PM, Dariush Pietrzak wrote: >>> I am very sad to hear this. Could you please file a bug to >>> bugzilla.openvz.org so our kernel guys will start working on that? >> Looking at bugzilla there are many other similiar reports, one of mine has >> been closed as fixed, but then returned in exactly the same function after >> just 6 minutes of stress-testing new kernel. >> It's easy to reproduce, just put enough load on the system. > > Have you reopened it already? Can you provide bug number? > >> >> It looks really troubling, both vSwap and 042.x branches look very nice >> feature-wise, even vzmigrate seems to work fine, which is no small feat, >> but it kinda feels like stability has been sacrificed to get there. >> >> best regards, Eyck > > Guys, > > I do understand reasons for your frustration, but so far I have only > seen one specific bug mentioned in this thread, namely > http://bugzilla.openvz.org/2095 it was filed yesterday and there is > a patch already available for testing. Any other statements like > "there are many bugs", "this kernel is unstable" are just not > specific enough for me to deal with. > > If there are bugs, they need to be reported and fixed, and we, > OpenVZ team, partly rely on you, our users. We do have internal QA > but can't possibly test all the use cases and scenarios. > > Specifically, we rely on having bug reports from you, with full > kernel logs (see http://wiki.openvz.org/Remote_console_setup), test > cases (as specific and reproducible as possible), and ideally your > ability to test patches that developers provide and report your > results back to bugzilla. > > We treat bug reports very seriously, and we do our best to reproduce > your bugs locally and fix them. Again, please be specific and refer > to exact bugs in bugzilla when you are having problems with kernel > stability, otherwise it's not helpful and I can't do much about it. > > Kir. It was no offense intended…
creator: MailingListe
date: 2011-11-23T15:21:57-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Zitat von Kir Kolyshkin <kir@openvz.org>: > On 11/23/2011 04:31 PM, Dariush Pietrzak wrote: >>> I am very sad to hear this. Could you please file a bug to >>> bugzilla.openvz.org so our kernel guys will start working on that? >> Looking at bugzilla there are many other similiar reports, one of mine has >> been closed as fixed, but then returned in exactly the same function after >> just 6 minutes of stress-testing new kernel. >> It's easy to reproduce, just put enough load on the system. > > Have you reopened it already? Can you provide bug number? > >> >> It looks really troubling, both vSwap and 042.x branches look very nice >> feature-wise, even vzmigrate seems to work fine, which is no small feat, >> but it kinda feels like stability has been sacrificed to get there. >> >> best regards, Eyck > > Guys, > > I do understand reasons for your frustration, but so far I have only > seen one specific bug mentioned in this thread, namely > http://bugzilla.openvz.org/2095 it was filed yesterday and there is > a patch already available for testing. Any other statements like > "there are many bugs", "this kernel is unstable" are just not > specific enough for me to deal with. > > If there are bugs, they need to be reported and fixed, and we, > OpenVZ team, partly rely on you, our users. We do have internal QA > but can't possibly test all the use cases and scenarios. > > Specifically, we rely on having bug reports from you, with full > kernel logs (see http://wiki.openvz.org/Remote_console_setup), test > cases (as specific and reproducible as possible), and ideally your > ability to test patches that developers provide and report your > results back to bugzilla. Okay, can someone with a bugzilla account please confirm and create a bug with this one: Kernel (uname -a): 2.6.32-042stab039.11 as x86_64 installed on CentOS 6 HN steps to reproduce: - use a Ubuntu 10.04 i386 (32 bit) template and create a VE - load ipt_recent in vz.conf - start the VE and use something like this "…
creator: MailingListe
date: 2011-11-23T16:59:29-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: On 11/23/2011 08:59 PM, lst_hoe02@kwsoft.de wrote: > Zitat von Kir Kolyshkin<kir@openvz.org>: > >> On 11/23/2011 04:31 PM, Dariush Pietrzak wrote: >>>> I am very sad to hear this. Could you please file a bug to >>>> bugzilla.openvz.org so our kernel guys will start working on that? >>> Looking at bugzilla there are many other similiar reports, one of mine has >>> been closed as fixed, but then returned in exactly the same function after >>> just 6 minutes of stress-testing new kernel. >>> It's easy to reproduce, just put enough load on the system. >> Have you reopened it already? Can you provide bug number? >> >>> It looks really troubling, both vSwap and 042.x branches look very nice >>> feature-wise, even vzmigrate seems to work fine, which is no small feat, >>> but it kinda feels like stability has been sacrificed to get there. >>> >>> best regards, Eyck >> Guys, >> >> I do understand reasons for your frustration, but so far I have only >> seen one specific bug mentioned in this thread, namely >> http://bugzilla.openvz.org/2095 it was filed yesterday and there is >> a patch already available for testing. Any other statements like >> "there are many bugs", "this kernel is unstable" are just not >> specific enough for me to deal with. >> >> If there are bugs, they need to be reported and fixed, and we, >> OpenVZ team, partly rely on you, our users. We do have internal QA >> but can't possibly test all the use cases and scenarios. >> >> Specifically, we rely on having bug reports from you, with full >> kernel logs (see http://wiki.openvz.org/Remote_console_setup), test >> cases (as specific and reproducible as possible), and ideally your >> ability to test patches that developers provide and report your >> results back to bugzilla. > Okay, can someone with a bugzilla account please confirm and create a > bug with this one: Pardon my curiosity, but why you need someone to act as your proxy filing bugs into bugzilla? I mean, I could create a bug, then a developer will…
creator: kir
date: 2011-11-23T17:13:27-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: On Wed, Nov 23, 2011 at 8:59 PM, <lst_hoe02@kwsoft.de> wrote: > Okay, can someone with a bugzilla account please confirm and create a bug > with this one: You should sign up there and create the bug yourself. That's the whole point of bug reporting -- interaction with devs. They can ask you to provide some more details or they can give you a patch to test. They will need *you* directly to make things fast, not the middleman. -- SY, Ilya A. Otyutskiy aka Sharp
creator: Sharp
date: 2011-11-23T17:20:06-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Zitat von Kir Kolyshkin <kir@openvz.org>: > On 11/23/2011 08:59 PM, lst_hoe02@kwsoft.de wrote: >> Zitat von Kir Kolyshkin<kir@openvz.org>: >> >>> On 11/23/2011 04:31 PM, Dariush Pietrzak wrote: >>>>> I am very sad to hear this. Could you please file a bug to >>>>> bugzilla.openvz.org so our kernel guys will start working on that? >>>> Looking at bugzilla there are many other similiar reports, one >>>> of mine has >>>> been closed as fixed, but then returned in exactly the same function after >>>> just 6 minutes of stress-testing new kernel. >>>> It's easy to reproduce, just put enough load on the system. >>> Have you reopened it already? Can you provide bug number? >>> >>>> It looks really troubling, both vSwap and 042.x branches look very nice >>>> feature-wise, even vzmigrate seems to work fine, which is no small feat, >>>> but it kinda feels like stability has been sacrificed to get there. >>>> >>>> best regards, Eyck >>> Guys, >>> >>> I do understand reasons for your frustration, but so far I have only >>> seen one specific bug mentioned in this thread, namely >>> http://bugzilla.openvz.org/2095 it was filed yesterday and there is >>> a patch already available for testing. Any other statements like >>> "there are many bugs", "this kernel is unstable" are just not >>> specific enough for me to deal with. >>> >>> If there are bugs, they need to be reported and fixed, and we, >>> OpenVZ team, partly rely on you, our users. We do have internal QA >>> but can't possibly test all the use cases and scenarios. >>> >>> Specifically, we rely on having bug reports from you, with full >>> kernel logs (see http://wiki.openvz.org/Remote_console_setup), test >>> cases (as specific and reproducible as possible), and ideally your >>> ability to test patches that developers provide and report your >>> results back to bugzilla. >> Okay, can someone with a bugzilla account please confirm and create a >> bug with this one: > > Pardon my curiosity, but why you need someone to act as …
creator: MailingListe
date: 2011-11-23T17:25:13-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: On 11/23/2011 09:25 PM, lst_hoe02@kwsoft.de wrote: > I already have countless accounts at numerous > bugzilla/forums/whatever, so i try to avoid creating throw away > accounts (Karteileichen) as much as possible. As some of the > developers must confirm/reproduce the bug anyway it was may impression > it would be smart to avoid just-another-account-somewhere. I am with you here, and I always wanted to add openid support to our bugzilla instance. Unfortunately it looks like openid support for bugzilla is taking forever to be finished. You can see my name in votes for bugzilla.mozilla.org bug #294608 "Support OpenID as a an account source and login verification method" at https://bugzilla.mozilla.org/page.cgi?id=voting/bug.html&amp ;bug_id=294608 While this is still a work in progress, I'm afraid registering at our bugzilla is the only way to have the OpenVZ bugs you came across fixed. > But if it helps, so it be Thank you!
creator: kir
date: 2011-11-23T20:29:12-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: > Have you reopened it already? Can you provide bug number? BUG 2080. I reopened, got told that that is completely different issue, then encountered exactly the same issue on supposedly fixed kernel, so reopened again. > Any other statements like > "there are many bugs", "this kernel is unstable" are just not > specific enough for me to deal with. That's why I wanted to provide a way to reproduce the problem, I would imagine that overnight stresstest would already be a part of your internal QA. This got through our own QA probably because we were running only 'stress' app, only when we added parallel bonnie+ ( which we did, because production machines that were crashing all had significant IO on them as common thing ). > If there are bugs, they need to be reported and fixed, and we, > OpenVZ team, partly rely on you, our users. We do have internal QA > but can't possibly test all the use cases and scenarios. > > Specifically, we rely on having bug reports from you, with full > kernel logs (see http://wiki.openvz.org/Remote_console_setup), test > cases (as specific and reproducible as possible), and ideally your > ability to test patches that developers provide and report your > results back to bugzilla. > > We treat bug reports very seriously, and we do our best to reproduce > your bugs locally and fix them. Again, please be specific and refer Thats very nice and correct, can you please tell me if the issue I can see here has been reproduced and is too hard to fix, or maybe it's not reproducible, and then, how can I help in reproducing them. As I said, from my point of view, the issue is trivially reproducible, results in crashes manifesting themselves in few different ways, so I assume that means multiple different bugs, or something more fundamental. best regards, Eyck -- Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9 Total Existance Failure
creator: Dariush Pietrzak
date: 2011-11-24T09:44:59-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Zitat von Dariush Pietrzak <ml-openvz-eyck@kuszelas.eu>: >> Have you reopened it already? Can you provide bug number? > > BUG 2080. > I reopened, got told that that is completely different issue, then > encountered exactly the same issue on supposedly fixed kernel, > so reopened again. > >> Any other statements like >> "there are many bugs", "this kernel is unstable" are just not >> specific enough for me to deal with. > > That's why I wanted to provide a way to reproduce the problem, I would > imagine that overnight stresstest would already be a part of your internal > QA. > This got through our own QA probably because we were running only > 'stress' app, only when we added parallel bonnie+ ( which we did, because > production machines that were crashing all had significant IO on them as > common thing ). Just out of curiousity i use my kernel crash-test setup to test with "stress" and "bonnie". I simply use the OpenVZ-Kernel with two container (ubuntu-10.04) and let one run stress and the other bonnie. The load is at 15 but the machine is humming along since around 4 hours... Is it possible that your problem arise from the io devices used? Regards Andreas
creator: MailingListe
date: 2011-11-24T11:27:23-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: > Just out of curiousity i use my kernel crash-test setup to test with > "stress" and "bonnie". I simply use the OpenVZ-Kernel with two > container (ubuntu-10.04) and let one run stress and the other > bonnie. The load is at 15 but the machine is humming along since > around 4 hours... With such low load we also couldn't crash it in timely matter. With lightly loaded machines we endured months without crash. I use this: stress -c 22 -i 24 -m 8 -d 20 --hdd-bytes 10G and this: while (true) do bonnie++ -d /fs/v/bonnie/ -c 8 -b -f -u root echo next done in parallel, I don't even have to run it inside containers. (test machine is single 4-core Xeon E5320, with 4G ram and two 146G raid 1s joined by lvm. With loadavg 50-80 we get crashes after few hours). > Is it possible that your problem arise from the io devices used? Possible, but unlikely, we first noticed crashed using FC devices, and then moved to testing on small P400i with 256M ram. One of the most affected machines used P410i controller, which is very similiar and the same generation as P400i. I can re-test on FC again. And while IO load seems to be neccessary to cause crash, resulting oops-es are similiar, very often account_system_time appears: [38766.228063] panic occurred, switching back to text console [38766.228063] BUG: scheduling while atomic: stress/1962/0x10000100 (this is identical to what we saw in production, only with 'java' instead of 'stress') [38766.227505] BUG: unable to handle kernel paging request at 0000000000021300 [38766.227509] IP: [<ffffffff81050ec4>] update_curr+0x154/0x200 [38766.227514] PGD 12c7b4067 PUD 12c7b5067 PMD 0 [38764.623677] BUG: unable to handle kernel paging request at 000000000001e440 [38764.623677] IP: [<ffffffff814c8efe>] _spin_lock+0xe/0x30 [38764.599189] BUG: unable to handle kernel paging request at 0000000000019550 [38764.599189] IP: [<ffffffff8105674f>] account_system_time+0xaf/0x1f0 [ 1876.747809] BUG: unable to handle kernel paging request at 00000006000000bd [ 18…
creator: Dariush Pietrzak
date: 2011-11-24T12:15:39-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Zitat von Dariush Pietrzak <ml-openvz-eyck@kuszelas.eu>: >> Just out of curiousity i use my kernel crash-test setup to test with >> "stress" and "bonnie". I simply use the OpenVZ-Kernel with two >> container (ubuntu-10.04) and let one run stress and the other >> bonnie. The load is at 15 but the machine is humming along since >> around 4 hours... > With such low load we also couldn't crash it in timely matter. > With lightly loaded machines we endured months without crash. > > I use this: > stress -c 22 -i 24 -m 8 -d 20 --hdd-bytes 10G > and this: > while (true) > do > bonnie++ -d /fs/v/bonnie/ -c 8 -b -f -u root > echo next > done > in parallel, I don't even have to run it inside containers. > (test machine is single 4-core Xeon E5320, with 4G ram and two 146G raid 1s > joined by lvm. With loadavg 50-80 we get crashes after few hours). So i tried loading the kernel harder. With a load of about 48 it was still stable, if i raise the number for c,i,m,d even more, the OOM killer jumps in and from that point the whole machine freeze, but does not panic??? Regards Andreas
creator: MailingListe
date: 2011-11-25T09:39:39-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: > >(test machine is single 4-core Xeon E5320, with 4G ram and two 146G raid 1s > >joined by lvm. With loadavg 50-80 we get crashes after few hours). > > So i tried loading the kernel harder. With a load of about 48 it was > still stable, if i raise the number for c,i,m,d even more, the OOM > killer jumps in and from that point the whole machine freeze, but > does not panic??? That would be good first sign, maybe the parameters I provided are specific for that DL360 machine, will test again on something larger. regards, Eyck -- Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9 Total Existance Failure
creator: Dariush Pietrzak
date: 2011-11-25T13:26:02-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: > still stable, if i raise the number for c,i,m,d even more, the OOM Just an update - with 042stab039.11 + bdi patch I was also unable to re-create the original problem, we also tried much higher loads ( memtester 52G 100 + stress -c 240 -i 24 -m 48 --vm-bytes 1024MB -d 20 --hdd-bytes 12G + bonnie++ ) and the problem does not return. We did encounter another 'hp smartarray disconnecting raids' problems on another test machine, but no kernel oops, and the problem seems hardware-related, thanks. best regards, Eyck -- Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9 Total Existance Failure
creator: Dariush Pietrzak
date: 2011-11-30T09:34:27-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: We're also seeing a big increase in instability since moving to the RHEL 6 kernels. Specifically, our typical platform consists of a Supermicro motherboard with dual 12-core AMD procs (ie. 24 in one system); The most frustrating part is that the symptom we're seeing is highly intermittent (sometimes it takes 10 minutes to trigger, sometimes several days), and doesn't result in a kernel panic or dump per se. Instead what we're seeing is an unresponsive system (still responding to ping, but all services on the box are unresponsive), with this scrolling by on the console: BUG: soft lockup - CPU#22 stuck for 67s! [node:585441] BUG: soft lockup - CPU#23 stuck for 68s! [node:585419] (multiple times per second, repeating all the different process numbers and many different processes running within containers). We're going to file a bug report on this, of course, but wondered if there was anything else we can do here to get any other information which can help the devs to come up with the cause and hopefully fix for the above? (Again, we're not getting a panic, and we're not able to do anything on the console.) Thanks, Stephen -- Stephen Balukoff Blue Box Group, LLC (800)613-4305 x807
creator: Stephen Balukoff
date: 2011-12-01T22:03:55-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Oh! And for what it's worth, we're seeing this on both the latest stable RHEL6 kernel, as well as the latest testing RHEL6 kernel available in the repositories for download. (That is, 042stab39.11 and 042stab044.1 respectively). Stephen On Thu, Dec 1, 2011 at 2:03 PM, Stephen Balukoff <sbalukoff@bluebox.net> wrote: > We're also seeing a big increase in instability since moving to the > RHEL 6 kernels. Specifically, our typical platform consists of a > Supermicro motherboard with dual 12-core AMD procs (ie. 24 in one > system); The most frustrating part is that the symptom we're seeing > is highly intermittent (sometimes it takes 10 minutes to trigger, > sometimes several days), and doesn't result in a kernel panic or dump > per se. Instead what we're seeing is an unresponsive system (still > responding to ping, but all services on the box are unresponsive), > with this scrolling by on the console: > > BUG: soft lockup - CPU#22 stuck for 67s! [node:585441] > BUG: soft lockup - CPU#23 stuck for 68s! [node:585419] > > (multiple times per second, repeating all the different process > numbers and many different processes running within containers). > > We're going to file a bug report on this, of course, but wondered if > there was anything else we can do here to get any other information > which can help the devs to come up with the cause and hopefully fix > for the above? (Again, we're not getting a panic, and we're not able > to do anything on the console.) > > Thanks, > Stephen > > > -- > Stephen Balukoff > Blue Box Group, LLC > (800)613-4305 x807 -- Stephen Balukoff Blue Box Group, LLC (800)613-4305 x807
creator: Stephen Balukoff
date: 2011-12-01T22:06:48-00:00

Re: Is there a stable OpenVZ kernel, and which should be fit for production
type: item
description: Ok, y'all: We managed to get a call trace. I've opened the following bug on this issue: http://bugzilla.openvz.org/show_bug.cgi?id=2110 Anything else I can do or provide to get traction on getting a developer to look at this? (This is a complete show-stopper for our Scientific Linux 6.1 OpenVZ roll-out.) Stephen On Thu, Dec 1, 2011 at 2:06 PM, Stephen Balukoff <sbalukoff@bluebox.net> wrote: > Oh! And for what it's worth, we're seeing this on both the latest > stable RHEL6 kernel, as well as the latest testing RHEL6 kernel > available in the repositories for download. (That is, 042stab39.11 > and 042stab044.1 respectively). > > Stephen > > On Thu, Dec 1, 2011 at 2:03 PM, Stephen Balukoff <sbalukoff@bluebox.net> wrote: >> We're also seeing a big increase in instability since moving to the >> RHEL 6 kernels. Specifically, our typical platform consists of a >> Supermicro motherboard with dual 12-core AMD procs (ie. 24 in one >> system); The most frustrating part is that the symptom we're seeing >> is highly intermittent (sometimes it takes 10 minutes to trigger, >> sometimes several days), and doesn't result in a kernel panic or dump >> per se. Instead what we're seeing is an unresponsive system (still >> responding to ping, but all services on the box are unresponsive), >> with this scrolling by on the console: >> >> BUG: soft lockup - CPU#22 stuck for 67s! [node:585441] >> BUG: soft lockup - CPU#23 stuck for 68s! [node:585419] >> >> (multiple times per second, repeating all the different process >> numbers and many different processes running within containers). >> >> We're going to file a bug report on this, of course, but wondered if >> there was anything else we can do here to get any other information >> which can help the devs to come up with the cause and hopefully fix >> for the above? (Again, we're not getting a panic, and we're not able >> to do anything on the console.) >> >> Thanks, >> Stephen >> >> >> -- >> Stephen Balukoff >> Blue Box Group, LLC >> (800)6…
creator: Stephen Balukoff
date: 2011-12-02T00:39:57-00:00