NOHZ: local_softirq_pending 100 - is there something to worry about?
Hello, Now again, after the node reboot because of network device freeze (e1000e driver bug? Still not updated driver version in openvz kernel) our node up for 22hours and we see this message in the logs: NOHZ: local_softirq_pending 100 Is this is something we have to worry about? For a while it is not caused any problems. Or we should just ignore this message. P.S. Well, I think we made a mistake to put RHEL6 openvz 2.6.32 kernel on the production to early... the more we use it the more bugs, locks, freezes and problems we have. It is far far away from "stable"... All other 2.6.18 centos 5 openvz nodes runs very good. Considering now to go back to RHEL 5 with 2.6.18 kernel for one more year, even if End-of-life for RHEL5 is alot shorter than RHEL6... Or maybe there is solution to use 2.6.18 kernel in Centos 6 and not to use 2.6.32 until it becomes really "stable" and then just switch to a 2.6.32? What you think about it? Thank you for any answers of suggestions.
Re: NOHZ: local_softirq_pending 100 - is there something to worry about?
I just wanted to say that I have small set of servers running centos6 servers with rhel6 based openvz kernels with running centos5 or centos6 CT's inside them and did not have any issues of kernel panicks or whatever. it works like a charm. I suggest that you use latest stable kernel, and if you still have issues and can provide why server crashes than submit a bug report. p.s. I use 2.6.32-042stab044.xx kernel versions
Paparaciz wrote on Tue, 31 January 2012 22:56 I just wanted to say that I have small set of servers running centos6 servers with rhel6 based openvz kernels with running centos5 or centos6 CT's inside them and did not have any issues of kernel panicks or whatever. it works like a charm. I suggest that you use latest stable kernel, and if you still have issues and can provide why server crashes than submit a bug report. p.s. I use 2.6.32-042stab044.xx kernel versions Yes, in production centos 6 nodes we are using latest stable 2.6.32 versions only. And I already submited three bug reports regarding our previous issues in bugzilla with this kernel, so I hope this will help to fix these bugs to make 2.6.32 kernel more and more stable. So, until 2.6.32 kernel becomes more stable, is there a reason to temporary use 2.6.18 stable kernel from centos 5 on the centos 6 nodes. And after some time, when 2.6.32 will be stable enough, switch back from 2.6.18 to a 2.6.32 kernel. The reason to use centos 6 OS (not centos 5) is longer support than for centos 5. Currently this is the single reason for us to start using centos 6, because all our centos 5 nodes runs stable, but there will be day when comes end-of-life for centos 5 support, so we will be forced to move all our centos 5 nodes to a centos 6 OS anyway, and if more centos 5 nodes we will use the more moving need to be done, more downtime, more work and more problems. Upgrade from centos 5 to centos 6 is not officially supported. So, to upgrade 5=>6 you'll have to completly reinstall centos 6 from zero and after that move containers. If there is just a few nodes, then this is not so big problem, but if there is a few tens or hunderds of nodes, then there will be a problem.
I also has network device freeze daily . This happen on Centos6, Hardware is Dxll R410 (Ram: 16G) I tested it with memtest86+ No any useful info. on log file & console .... There have 4 VPS only, and it configured by "vzsplit -n4" Then i switch to Debian6, but network device still freeze -___- The good news is it freeze around 2 month one time I think it is acceptable ^ ^
Same errors also here, around 2 times a day. 2 system hangs in about 1 month, dont no if it is related to this error, because i need to reset the server (not reponding on anything / blank screen / network down) Is it possible to add a fix in openvz kernel? perhaps: h_t_t_p_s://lkml.org/lkml/2007/5/22/35 is related ?? Using kernel 2.6.32-042stab065.3 Regards, Marco
I am still experiencing this, including the NOHZ message, and horrible instability in general. It's a nightmare! My uname -a: Linux vhost-amc-01 2.6.32-042stab081.5 #1 SMP Mon Sep 30 16:52:24 MSK 2013 x86_64 x86_64 x86_64 GNU/Linux I can't go back to 5, I have to remain on 6. Is there a known kernel version that works really well with CentOS 6 and OpenVZ?
Quote: Is there a known kernel version that works really well with CentOS 6 and OpenVZ? I'd say that the majority of CentOS 6 / OpenVZ servers work quite well, so that's the wrong question to be asking. Try filing a bug report with the information specific to your server and the problems you're having, that's probably the best course of action.
The problem is that when the machine crashes, there's nothing in the logs, no indication. It just happens, and I am left hanging with no data to hand to developers that might aid them in debugging and understanding the cause. I don't know how to reproduce it, it happens randomly. All I can see is those small "warnings" in the logs, such as the NOHZ error message, CPU locking messages, etc. For example: [42071.390012] hrtimer: interrupt took 13881 ns Or this one: [30754.200039] NOHZ: local_softirq_pending 100 Or this one which happened a few days ago and complete froze the machine: [249348.095995] BUG: soft lockup - CPU#4 stuck for 67s! [flush-8:0:866] (for all CPU's, not just CPU#4).
Perhaps if you post your HW configuration, someone will be able to provide more specific answers. Also, what steps did you take to get additional debug information? Perhaps someone can see a missing step...
This is my hardware configuration (output of 'lshw'): gist.github.com/skaag/7647697 And my 'dmesg' output: gist.github.com/skaag/7647719 It will be great if you can give me some ideas of other commands to run, other than 'lshw' and 'dmesg', which I can then post here.