Home » General » Support » Centos6 and OpenVZ memory problems (Node restarting due to excessive usage of Memory)
Centos6 and OpenVZ memory problems [message #43470] |
Wed, 14 September 2011 13:01 |
rshosting
Messages: 1 Registered: September 2011
|
Junior Member |
|
|
Hello,
First of all I would like to introduce myself here.. This is Sam here and have been in web hosting business since quite a long time now..
We have been using Openvz with SolusVM control panel to offer vitual servers to our clients. Our previous servers have Centos 5 and Openvz which runs like a charm.
Recently we had our new node ordered and I thought i better go with Centos 6. I had Openvz installed along with SolusVM on the server.
However, we have noticed, Openvz is not going along that well with Centos6, and is overusing the memory on the server. The node as a result gets rebooted frequently after every couple of days.
We suspected if it was the physical memory issue on the node, however it is good and not the main problem.
Existing Node details :
Server : Dell Quad Core Xeon E5620 2.40GHz
Physical RAM : 16 GB DDR3
SWAP : 16 GB
Drives : 4 x 1TB in RAID 10
OS : Centos 6 (64 bit)
Openvz Kernel : 2.6.32-042stab036.1
Listed below are few memory related mesages from dmesg :
Quote:
[92477.082356] Node 0 Normal scan:32 a_anon:34553 i_anon:1953 a_file:0 i_file:35 unevictable:0
[92477.082366] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[92477.082376] Node 1 DMA32 scan:0 a_anon:1527 i_anon:286 a_file:0 i_file:1 unevictable:0
[92477.082386] Node 1 Normal scan:32 a_anon:41958 i_anon:2419 a_file:1 i_file:4 unevictable:0
[92477.082397] Total 82737 anon:82696 file:41 a_anon:78038 i_anon:4658 a_file:1 i_file:40 unevictable:0
[92477.082409] UB: 183 RAM: 88891 / 98304 [193] SWAP: 21808 / 32768 [193] KMEM: 24547328 / 201326592 [0] Dirty 0
[92486.775157] php invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
[92486.775164] php cpuset=183 mems_allowed=0-1
[92486.775167] UB-183-Mem-Info:
[92486.775172] Node 0 Normal scan:1181 a_anon:40111 i_anon:4771 a_file:0 i_file:4 unevictable:0
[92486.775177] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[92486.775181] Node 1 DMA32 scan:1536 a_anon:557 i_anon:253 a_file:0 i_file:2 unevictable:0
[92486.775186] Node 1 Normal scan:1473 a_anon:42414 i_anon:4538 a_file:1 i_file:4 unevictable:0
[92486.775192] Total 92657 anon:92644 file:13 a_anon:83082 i_anon:9562 a_file:1 i_file:12 unevictable:0
[92486.775197] UB: 183 RAM: 98291 / 98304 [194] SWAP: 32793 / 32768 [194] KMEM: 22822912 / 201326592 [0] Dirty 0
[92486.775202] Out of memory in UB: kill process 364821 (mysqld) or a child
[92486.775490] Killed process 364821 (mysqld) vsz:272224kB, anon-rss:22536kB, file-rss:420kB
[92486.908373] OOM killed process mysqld (pid=364821, ve=183) exited, free=263440.
[92486.908543] UB-183-Mem-Info:
[92486.908549] Node 0 Normal scan:89444 a_anon:36719 i_anon:4781 a_file:1 i_file:127 unevictable:0
[92486.908556] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[92486.908563] Node 1 DMA32 scan:46915 a_anon:555 i_anon:261 a_file:0 i_file:0 unevictable:0
[92486.908570] Node 1 Normal scan:9272 a_anon:40212 i_anon:4538 a_file:0 i_file:5 unevictable:0
[92486.908579] Total 87199 anon:87066 file:133 a_anon:77486 i_anon:9580 a_file:1 i_file:132 unevictable:0
[92486.908587] UB: 183 RAM: 92852 / 98304 [208] SWAP: 32276 / 32768 [208] KMEM: 22646784 / 201326592 [0] Dirty 0
[92487.368083] mysqld invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
[92487.368090] mysqld cpuset=183 mems_allowed=0-1
[92487.368093] Out of memory in UB: kill process 364329 (php) or a child
[92487.368173] Killed process 364329 (php) vsz:128812kB, anon-rss:28076kB, file-rss:788kB
[92487.373705] OOM killed process php (pid=364329, ve=183) exited, free=274441.
[92487.793675] php invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
[92487.793682] php cpuset=183 mems_allowed=0-1
[92487.793685] Out of memory in UB: kill process 364914 (mysqld) or a child
[92487.793925] Killed process 364914 (mysqld) vsz:271892kB, anon-rss:13736kB, file-rss:432kB
[92487.795747] OOM killed process mysqld (pid=364914, ve=183) exited, free=270716.
[92487.924192] php invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
[92487.924197] php cpuset=183 mems_allowed=0-1
[92487.924200] Out of memory in UB: kill process 364684 (php) or a child
[92487.924279] Killed process 364684 (php) vsz:130040kB, anon-rss:36088kB, file-rss:1056kB
[92487.933825] OOM killed process php (pid=364684, ve=183) exited, free=274379.
[94157.964842] php invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
[94157.964847] php cpuset=188 mems_allowed=0-1
[94157.964850] UB-188-Mem-Info:
[94157.964853] Node 0 Normal scan:242 a_anon:56065 i_anon:7924 a_file:0 i_file:0 unevictable:0
[94157.964858] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[94157.964862] Node 1 DMA32 scan:0 a_anon:2847 i_anon:806 a_file:0 i_file:0 unevictable:0
[94157.964866] Node 1 Normal scan:30 a_anon:20771 i_anon:3489 a_file:9 i_file:11 unevictable:0
[94157.964871] Total 91922 anon:91902 file:20 a_anon:79683 i_anon:12219 a_file:9 i_file:11 unevictable:0
[94157.964876] UB: 188 RAM: 98145 / 98304 [12] SWAP: 32816 / 32768 [12] KMEM: 21098496 / 201326592 [0] Dirty 0
[94157.964880] Out of memory in UB: kill process 421327 (php) or a child
[94157.965144] Killed process 421327 (php) vsz:137724kB, anon-rss:37968kB, file-rss:376kB
[94157.971771] OOM killed process php (pid=421327, ve=188) exited, free=326718.
[94157.971776] UB-188-Mem-Info:
[94157.971781] Node 0 Normal scan:0 a_anon:49288 i_anon:7805 a_file:145 i_file:408 unevictable:0
[94157.971786] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[94157.971790] Node 1 DMA32 scan:0 a_anon:3019 i_anon:663 a_file:0 i_file:1 unevictable:0
[94157.971795] Node 1 Normal scan:0 a_anon:17992 i_anon:3434 a_file:133 i_file:182 unevictable:0
[94157.971801] Total 83081 anon:82201 file:880 a_anon:70299 i_anon:11902 a_file:278 i_file:602 unevictable:0
[94157.971807] UB: 188 RAM: 89295 / 98304 [12] SWAP: 31248 / 32768 [12] KMEM: 20742144 / 201326592 [0] Dirty 0
[96077.906840] php invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
[96077.906847] php cpuset=183 mems_allowed=0-1
[96077.906850] __ratelimit: 6 callbacks suppressed
[96077.906853] UB-183-Mem-Info:
[96077.906858] Node 0 Normal scan:5276 a_anon:53731 i_anon:6033 a_file:0 i_file:6 unevictable:0
[96077.906863] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[96077.906867] Node 1 DMA32 scan:9 a_anon:3187 i_anon:664 a_file:0 i_file:0 unevictable:0
[96077.906872] Node 1 Normal scan:9192 a_anon:24625 i_anon:3222 a_file:2 i_file:29 unevictable:0
[96077.906877] Total 91498 anon:91462 file:36 a_anon:81543 i_anon:9919 a_file:2 i_file:34 unevictable:0
[96077.906883] UB: 183 RAM: 98272 / 98304 [229] SWAP: 32803 / 32768 [229] KMEM: 21028864 / 201326592 [0] Dirty 0
[96077.906888] Out of memory in UB: kill process 365734 (mysqld_safe) or a child
[96077.907231] Killed process 365766 (mysqld) vsz:280488kB, anon-rss:31424kB, file-rss:488kB
[96077.920155] OOM killed process mysqld (pid=486348, ve=183) exited, free=255613.
[96077.920161] UB-183-Mem-Info:
[96077.920165] Node 0 Normal scan:5725 a_anon:50185 i_anon:4891 a_file:0 i_file:0 unevictable:0
[96077.920170] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[96077.920175] Node 1 DMA32 scan:9 a_anon:2505 i_anon:569 a_file:0 i_file:0 unevictable:0
[96077.920180] Node 1 Normal scan:9917 a_anon:22872 i_anon:2832 a_file:2 i_file:29 unevictable:0
[96077.920185] Total 83885 anon:83854 file:31 a_anon:75562 i_anon:8292 a_file:2 i_file:29 unevictable:0
[96077.920191] UB: 183 RAM: 90762 / 98304 [229] SWAP: 22422 / 32768 [229] KMEM: 20586496 / 201326592 [0] Dirty 0
[97492.758629] php invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
[97492.758636] php cpuset=183 mems_allowed=0-1
[97492.758640] UB-183-Mem-Info:
[97492.758644] Node 0 Normal scan:64 a_anon:42773 i_anon:4799 a_file:0 i_file:0 unevictable:0
[97492.758649] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[97492.758654] Node 1 DMA32 scan:27 a_anon:26398 i_anon:3277 a_file:0 i_file:0 unevictable:0
[97492.758658] Node 1 Normal scan:136 a_anon:13681 i_anon:2217 a_file:0 i_file:0 unevictable:0
[97492.758664] Total 93145 anon:93145 file:0 a_anon:82852 i_anon:10293 a_file:0 i_file:0 unevictable:0
[97492.758669] UB: 183 RAM: 98302 / 98304 [230] SWAP: 32788 / 32768 [230] KMEM: 19886080 / 201326592 [0] Dirty 0
[97492.758674] Out of memory in UB: kill process 365734 (mysqld_safe) or a child
[97492.759076] Killed process 486584 (mysqld) vsz:277728kB, anon-rss:23284kB, file-rss:528kB
[97492.769883] OOM killed process mysqld (pid=486668, ve=183) exited, free=266622.
[97492.770007] UB-183-Mem-Info:
[97492.770014] Node 0 Normal scan:32 a_anon:40343 i_anon:3050 a_file:2 i_file:225 unevictable:0
[97492.770021] Node 1 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
[97492.770028] Node 1 DMA32 scan:32 a_anon:25652 i_anon:3115 a_file:0 i_file:0 unevictable:0
[97492.770036] Node 1 Normal scan:32 a_anon:12981 i_anon:2116 a_file:1 i_file:8 unevictable:0
[97492.770044] Total 87493 anon:87257 file:236 a_anon:78976 i_anon:8281 a_file:3 i_file:233 unevictable:0
[97492.770050] UB: 183 RAM: 92719 / 98304 [238] SWAP: 24116 / 32768 [238] KMEM: 19759104 / 201326592 [0] Dirty 0
[108608.489396] CT: 188: stopped
[108609.672423] CT: 188: started
[113206.445641] Core dump to |/usr/libexec/abrt-hook-ccpp /var/spool/abrt 1579 11 503 0 pipe failed
[113206.445658] Core dump to |/usr/libexec/abrt-hook-ccpp /var/spool/abrt 1582 11 503 0 pipe failed
[113206.445671] Core dump to |/usr/libexec/abrt-hook-ccpp /var/spool/abrt 1580 11 503 0 pipe failed
[113206.445682] Core dump to |/usr/libexec/abrt-hook-ccpp /var/spool/abrt 1573 11 503 0 pipe failed
[113206.445690] Core dump to |/usr/libexec/abrt-hook-ccpp /var/spool/abrt 1581 11 503 0 pipe failed
[113206.445807] Core dump to |/usr/libexec/abrt-hook-ccpp /var/spool/abrt 1584 11 503 0 pipe failed
[113206.445815] Core dum
...
|
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43505 is a reply to message #43470] |
Sat, 17 September 2011 14:17 |
bjdea1
Messages: 39 Registered: February 2009
|
Member |
|
|
I agree with the OP. I too have seen a number of problems with the new Centos6 OpenVZ kernel, both memory related and scheduliing relating (soft lockups). I have had to reboot our servers approx every 2 weeks for the last 2 months. I am considering grabbing our servers from the Datacenter and installing the Centos 5 OpenVZ platform instead. It seems that the Centos 6 platform is just too unstable at the moment.
But perhaps I will give it a little longer, we've been removing as many unnecessary services as we can and adding every fix we can find. Nothing seems to have truly solved the regular crashing problem yet but I might hold out a little longer in the hope new releases and patches can finally solve whatever is causing the problems on our servers, which are standard Dell 1950 servers.
█Deasoft.com Hosting/Software
█AutoBillMe.com Billing Automation
|
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43512 is a reply to message #43511] |
Sun, 18 September 2011 02:09 |
bjdea1
Messages: 39 Registered: February 2009
|
Member |
|
|
Ales wrote on Sun, 18 September 2011 10:46There is no such thing as "Centos6 OpenVZ kernel". OpenVZ uses a RHEL 6 kernel, these are two different things.
OpenVZ kernel sources come directly from Red Hat and the kernel is fully up to date as of today. So if you are using OpenVZ RHEL 6 kernel, you're using the latest RHEL 6.1 version, not the outdated CentOS 6.0 version.
If anyone has problems with OpenVZ RHEL 6 kernel, please file bugs in the OpenVZ Bugzilla. Waiting for CentOS updates won't make any difference to OpenVZ as they are not connected in any way.
If I steer a bit into an off-topic area: CentOS project is (IMHO) in a bad state right now. It seems to be severely undermanned and poorly managed. Anyone using CentOS 6 in a live environment right now is very courageous (to use a polite term), considering you're not getting any updates at all. But this has nothing to do with OpenVZ.
IMHO, the current recipe for sane sysadmin life seems to be using Scientific Linux 6.x and OpenVZ RHEL 6 kernel. Unless something drastically changes at how CentOS project is run, I wouldn't use it even after/if they catch up with the updates.
I know centos6 kernel is the wrong name to give it but I'm sure people know what I meant. Yes its the RHEL6 OpenVZ Kernel on a Centos 6.0 platform. I'm not attacking OpenVZ Developers, I love what they are doing and OpenVZ. I'm just adding my experience so far with Centos 6 and OpenVZ RHEL 6 kernel.
Thanks for your input though. Perhaps we'll try Scientigic Linux 6.x instead. Are there many people running production servers on Scientific Linux 6.x here? Any feedback regarding Scientific Linux 6.x?
█Deasoft.com Hosting/Software
█AutoBillMe.com Billing Automation
[Updated on: Sun, 18 September 2011 02:10] Report message to a moderator
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43513 is a reply to message #43470] |
Sun, 18 September 2011 02:47 |
Ales
Messages: 330 Registered: May 2009
|
Senior Member |
|
|
bjdea1, I apologize if my post came across a bit harsh, that was not my intention. I was referring to both yours and cybernet2u's post when I was writing mine and just wanted to make clear to anyone else reading this thread that OpenVZ RHEL 6 kernel isn't affected by the lack of CentOS 6 updates.
To perhaps answer the other part of your post - we're currently deploying SL 6 on a bunch of new servers and promote it's VM templates instead of CentOS 6 to our customers. It's been nothing but a smooth ride for us.
Technically, it does all I expect it to do and the SL community seems very nice too. The developers seem to be on top of things, considering their response times when it comes to small SL-related bugs and, most importantly, upstream updates.
We have been using CentOS since it first came out and stuck with it as long as we could, despite all the problems. I have the highest regard for many of the community members there and I hope their developers manage to turn things around. But to be honest, it doesn't look good and at the moment, SL seems like a better choice. All IMHO, of course.
I'd also like to hear others who perhaps use SL / OpenVZ combo, so if anyone else is reading... do chip in.
|
|
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43518 is a reply to message #43470] |
Sun, 18 September 2011 13:26 |
Ales
Messages: 330 Registered: May 2009
|
Senior Member |
|
|
What the h... are you talking about? Again, CentOS speed of updates has no impact and makes no difference to OpenVZ. OpenVZ takes the freely available Red Hat Enterprise Linux kernel source, same as you or I could if we wanted. Please don't spread FUD out of ignorance.
You do know what CentOS is and how they create their binary packages? You do know that CentOS takes the RHEL sources without changing them (apart from few exceptions, like anaconda, etc.), same as SL, PUIAS, etc.?
The fact that CentOS 5 has catched up with the updates recently and the fact that OpenVZ now has the same version of the RHEL 5 kernel is a coincidence. OpenVZ did not wait for CentOS updates to catch up, same as SL or PUIAS didn't. They all separately took RHEL sources and worked with them.
Take a look at the OpenVZ RHEL 6 kernel and tell me which minor version is it - you'll find it in the release notes and/or in the source. Than look at the CentOS 6 kernel and tell me the minor version. See?
[Updated on: Sun, 18 September 2011 13:28] Report message to a moderator
|
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43533 is a reply to message #43515] |
Mon, 19 September 2011 19:00 |
mustardman
Messages: 91 Registered: October 2009
|
Member |
|
|
bjdea1 wrote on Sun, 18 September 2011 00:57I'm considering just switching repo's on my servers now (to Scientific Linux), and running a yum update. Seems simple enough and could solve the issue we're having. Thinking about it...
Please let us know how that works out. It should work. Now would be a good time since SL is at 6.1 and CentOS is still at 6.0 (although they are adding incremental updates now). So the conversion would update a lot of packages which lessens the chance of some wierd incompatibility coming up.
To the other poster asking about Scientific Linux stability. It is used by some of the largest Educational institutions in the world as well as CERN. I don't have any specific information but it's a forgone conclusion that there are large commercial organizations using it.
Regardless, because CentOS and SL are both just recompiled versions of RHEL they are almost identical and no reason they both should not be equally as stable.
[Updated on: Mon, 19 September 2011 19:03] Report message to a moderator
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43565 is a reply to message #43470] |
Sat, 24 September 2011 23:53 |
bjdea1
Messages: 39 Registered: February 2009
|
Member |
|
|
I switched to Scientific Linux on both my nodes (servers). Unfortunately it didn't solve the problem. Same issue, cpu soft_lockups occur randomly, every few days. I'm wondering if its a driver issue. I saw it happen last night but caught it too late. I was connected to node remotely via ssh and left top running. I saw last top output with node load at 1400+ and 2 processes running at 100%. The node was unresponsive by the time I got to see it, but I did see one last top screen update, the last one. So I know that this problem I'm having takes at least few minutes to build up the load. I am hoping I can simply kill the 100% processes and avoid node from locking up. I've written a small program to monitor load and send me an SMS when the load spikes over 15, that way the next time I will see if I can stop node lock up by just killing the process or see if I can issue reboot command before load gets too high.
█Deasoft.com Hosting/Software
█AutoBillMe.com Billing Automation
[Updated on: Sat, 24 September 2011 23:54] Report message to a moderator
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43566 is a reply to message #43470] |
Sat, 24 September 2011 23:58 |
bjdea1
Messages: 39 Registered: February 2009
|
Member |
|
|
Here's part of top output:
top - 20:33:41 up 19:04, 1 user, load average: 1484.78, 1294.08, 881.16
Tasks: 3168 total, 12 running, 3141 sleeping, 0 stopped, 15 zombie
Cpu(s): 0.1%us, 25.2%sy, 0.0%ni, 0.0%id, 74.8%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16425816k total, 16265468k used, 160348k free, 2705988k buffers
Swap: 16381944k total, 57692k used, 16324252k free, 4788064k cached
█Deasoft.com Hosting/Software
█AutoBillMe.com Billing Automation
|
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43601 is a reply to message #43515] |
Fri, 30 September 2011 08:13 |
Rene
Messages: 40 Registered: September 2006
|
Member |
|
|
bjdea1 wrote on Sun, 18 September 2011 00:57I'm considering just switching repo's on my servers now (to Scientific Linux), and running a yum update. Seems simple enough and could solve the issue we're having. Thinking about it...
I'm in the same situation and followed [blog.bradiceanu.net/2011/09/02/how-to-convert-centos-6-0-to -scientific-linux-6-x/] to convert my Centos 6.0 to SL 6.1.
Five minutes and it was done. If everything was just so easy.
FWIW.
Sorry I'm not trusted to post links yet.
[Updated on: Fri, 30 September 2011 08:14] Report message to a moderator
|
|
|
|
|
|
Re: Centos6 and OpenVZ memory problems [message #43788 is a reply to message #43786] |
Sat, 15 October 2011 23:23 |
bjdea1
Messages: 39 Registered: February 2009
|
Member |
|
|
I would like to follow up on my experience.
I originally installed Centos 6 on our server and applied the latest RHEL6 OpenVZ kernel. We had stability issues and the server was crashing randomly. I then tried upgrading from Centos 6.0 to Scientific Linux 6.1 and again updated the OpenVZ kernel to latest RHEL6. We still had stability issues however we got up to 11 days uptime at one point. Finally after too many crashes we scrapped the RHEL6 platform and installed the Centos 5 Operating system and installed the latest RHEL5 kernel. We now have complete stability, no outages, no crashes, constant stable platform.
This proves that the issues we were having were RHEL6 OpenVZ kernel related and not hardware related.
One key consideration with my server, which is a Dell 1950, is that it is a fully populated production server with many varied VPS's running. Gaming VPS's, wHM/cpanel VPS's, Proxy VPS's, etc. Seems that RHEL6 OpenVZ kernel has still some way to go before it can handle large complex platforms.
Our other server is also a Dell 1950 however it has only a few VPS's and its under low loads, pretty simple setup. Its still on the RHEL6 OpenVZ kernel platform and running fine. So in my eyes that makes it clear that RHEL6 OpenVZ kernel still does not yet cope with a large complex VPS environment, at least not in our experience.
But I want to commend the OpenVZ developers because the bug reports that I submitted were looked at quickly and they were working on the issues. It was good to see how active they were in working to resolve the bugs I reported and I'm sad that I had to revert back to the RHEL 5 platform before being able to give them more feedback. But I know they will get on top of these issues and probably in a month or 2 it'll all be cleared up.
█Deasoft.com Hosting/Software
█AutoBillMe.com Billing Automation
[Updated on: Sat, 15 October 2011 23:27] Report message to a moderator
|
|
|
|
Goto Forum:
Current Time: Fri Nov 08 21:01:26 GMT 2024
Total time taken to generate the page: 0.03362 seconds
|