| Server crash frequently [message #43738] |
Wed, 12 October 2011 05:08 |
nethubonline
Messages: 26
Registered: October 2007
|
Junior Member |
|
|
Hi all,
We have used RHEL5 with OpenVZ 2.6.18 for long time. And now we started to use RHEL6 with OpenVZ 2.6.32 2 months ago, during these 2 months the server restarted automatically over 10 times.
We are thinking whether the Memory/Swap are used out to cause the problem, however even we set the SWAPPAGES="0:0" (I guess this can disable SWAP), the server still restarts automatically.
Below are information that may help to find out what's wrong, you may see the server restarted on Oct 12 11:09:57 and there are some kernel log around 11:00 that relates to Out of memory to kill process, but I think Out of memory should break down then container, but not the hardware node.
Thanks in advance for any help
OS: RHEL 6.0
OpenVZ Versions:
vzquota-3.0.12-1.x86_64
vzkernel-firmware-2.6.32-042stab037.1.noarch
vzkernel-2.6.32-042stab037.1.x86_64
vzctl-3.0.29.3-1.x86_64
vzctl-lib-3.0.29.3-1.x86_64
/etc/sysconfig/vz-scripts/101.conf:
=========================================================
ONBOOT="yes"
# UBC parameters (in form of barrier:limit)
PHYSPAGES="0:2G"
SWAPPAGES="0:0"
KMEMSIZE="unlimited"
LOCKEDPAGES="unlimited"
PRIVVMPAGES="unlimited"
SHMPAGES="unlimited"
NUMPROC="unlimited"
VMGUARPAGES="2G:unlimited"
OOMGUARPAGES="2G:unlimited"
NUMTCPSOCK="unlimited"
NUMFLOCK="unlimited"
NUMPTY="unlimited"
NUMSIGINFO="unlimited"
TCPSNDBUF="unlimited"
TCPRCVBUF="unlimited"
OTHERSOCKBUF="unlimited"
DGRAMRCVBUF="unlimited"
NUMOTHERSOCK="unlimited"
DCACHESIZE="unlimited"
NUMFILE="unlimited"
NUMIPTENT="unlimited"
# Disk quota parameters (in form of softlimit:hardlimit)
DISKSPACE="500G:500G"
DISKINODES="5000000:5050000"
QUOTATIME="0"
QUOTAUGIDLIMIT="1000"
# CPU fair scheduler parameter
CPUUNITS="50000"
OSTEMPLATE="centos-4-i386-default"
NAMESERVER="xxxxxxxxxxxxxxxx"
CPULIMIT="50"
VE_ROOT="/vz/root/$VEID"
VE_PRIVATE="/vz/private/$VEID"
ORIGIN_SAMPLE="vps.centos4"
IP_ADDRESS="xxxxxxxxxxxxxxxx"
HOSTNAME="xxxxxxxxxxxxxxxx"
IPTABLES="ip_tables iptable_filter iptable_mangle ipt_limit ipt_multiport ipt_tos ipt_REJECT ipt_TCPMSS ipt
/var/log/message:
=========================================================
Oct 11 15:33:28 cdweb1 kernel: [193989.183733] httpd invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
Oct 11 15:33:28 cdweb1 kernel: [193989.183760] httpd cpuset=102 mems_allowed=0
Oct 11 15:33:28 cdweb1 kernel: [193989.183774] UB-102-Mem-Info:
Oct 11 15:33:28 cdweb1 kernel: [193989.183785] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Oct 11 15:33:28 cdweb1 kernel: [193989.183807] Node 0 DMA32 scan:1258 a_anon:166829 i_anon:29035 a_file:0 i_file:2 unevictable:0
Oct 11 15:33:28 cdweb1 kernel: [193989.183832] Node 0 Normal scan:110 a_anon:35590 i_anon:26766 a_file:0 i_file:0 unevictable:0
Oct 11 15:33:28 cdweb1 kernel: [193989.183858] Total 258222 anon:258220 file:2 a_anon:202419 i_anon:55801 a_file:0 i_file:2 unevictable:0
Oct 11 15:33:28 cdweb1 kernel: [193989.183885] UB: 102 RAM: 262142 / 262144 [1] SWAP: 7844 / 0 [1] KMEM: 15708160 / 9223372036854775807 [0] Dirty 1430
Oct 11 15:33:28 cdweb1 kernel: [193989.183916] Out of memory in UB: kill process 2767 (mysqld_safe) or a child
Oct 11 15:33:28 cdweb1 kernel: [193989.183956] Killed process 2913 (mysqld) vsz:658264kB, anon-rss:126564kB, file-rss:792kB
Oct 11 15:33:28 cdweb1 kernel: [193989.186170] OOM killed process mysqld (pid=241044, ve=102) exited, free=416955.
Oct 11 15:33:28 cdweb1 kernel: [193989.186197] UB-102-Mem-Info:
Oct 11 15:33:28 cdweb1 kernel: [193989.186207] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Oct 11 15:33:28 cdweb1 kernel: [193989.186228] Node 0 DMA32 scan:1258 a_anon:161089 i_anon:2821 a_file:0 i_file:2 unevictable:0
Oct 11 15:33:28 cdweb1 kernel: [193989.186252] Node 0 Normal scan:110 a_anon:35590 i_anon:26667 a_file:0 i_file:0 unevictable:0
Oct 11 15:33:28 cdweb1 kernel: [193989.186277] Total 226169 anon:226167 file:2 a_anon:196679 i_anon:29488 a_file:0 i_file:2 unevictable:0
Oct 11 15:33:28 cdweb1 kernel: [193989.186303] UB: 102 RAM: 230267 / 262144 [1] SWAP: 2240 / 0 [1] KMEM: 15052800 / 9223372036854775807 [0] Dirty 1430
Oct 12 11:00:11 cdweb1 kernel: [263979.364636] httpd invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
Oct 12 11:00:11 cdweb1 kernel: [263979.364663] httpd cpuset=102 mems_allowed=0
Oct 12 11:00:11 cdweb1 kernel: [263979.364677] UB-102-Mem-Info:
Oct 12 11:00:11 cdweb1 kernel: [263979.364687] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:11 cdweb1 kernel: [263979.364709] Node 0 DMA32 scan:34 a_anon:207629 i_anon:6754 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:11 cdweb1 kernel: [263979.364732] Node 0 Normal scan:152 a_anon:25002 i_anon:18944 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:11 cdweb1 kernel: [263979.364757] Total 258329 anon:258329 file:0 a_anon:232631 i_anon:25698 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:11 cdweb1 kernel: [263979.364783] UB: 102 RAM: 262133 / 262144 [2] SWAP: 2902 / 0 [2] KMEM: 14434304 / 9223372036854775807 [0] Dirty 65
Oct 12 11:00:11 cdweb1 kernel: [263979.364812] Out of memory in UB: kill process 2767 (mysqld_safe) or a child
Oct 12 11:00:11 cdweb1 kernel: [263979.364854] Killed process 388297 (mysqld) vsz:654016kB, anon-rss:142404kB, file-rss:96kB
Oct 12 11:00:11 cdweb1 kernel: [263979.366053] OOM killed process mysqld (pid=426543, ve=102) exited, free=382138.
Oct 12 11:00:11 cdweb1 kernel: [263979.366078] UB-102-Mem-Info:
Oct 12 11:00:11 cdweb1 kernel: [263979.366088] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:11 cdweb1 kernel: [263979.366110] Node 0 DMA32 scan:34 a_anon:173102 i_anon:5685 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:11 cdweb1 kernel: [263979.366133] Node 0 Normal scan:152 a_anon:25001 i_anon:18899 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:11 cdweb1 kernel: [263979.366157] Total 222687 anon:222687 file:0 a_anon:198103 i_anon:24584 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:11 cdweb1 kernel: [263979.366184] UB: 102 RAM: 226737 / 262144 [2] SWAP: 2726 / 0 [2] KMEM: 13860864 / 9223372036854775807 [0] Dirty 65
Oct 12 11:00:18 cdweb1 kernel: [263986.782064] httpd invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
Oct 12 11:00:18 cdweb1 kernel: [263986.782090] httpd cpuset=102 mems_allowed=0
Oct 12 11:00:18 cdweb1 kernel: [263986.782104] UB-102-Mem-Info:
Oct 12 11:00:18 cdweb1 kernel: [263986.782114] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:18 cdweb1 kernel: [263986.782137] Node 0 DMA32 scan:0 a_anon:204537 i_anon:8088 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:18 cdweb1 kernel: [263986.782163] Node 0 Normal scan:0 a_anon:25947 i_anon:19960 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:18 cdweb1 kernel: [263986.783029] Total 258532 anon:258532 file:0 a_anon:230484 i_anon:28048 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:18 cdweb1 kernel: [263986.783900] UB: 102 RAM: 262126 / 262144 [3] SWAP: 2689 / 0 [3] KMEM: 13524992 / 9223372036854775807 [0] Dirty 65
Oct 12 11:00:18 cdweb1 kernel: [263986.784782] Out of memory in UB: kill process 521090 (mysqld) or a child
Oct 12 11:00:18 cdweb1 kernel: [263986.785677] Killed process 521090 (mysqld) vsz:562148kB, anon-rss:88916kB, file-rss:80kB
Oct 12 11:00:18 cdweb1 kernel: [263986.822162] OOM killed process mysqld (pid=521100, ve=102) exited, free=387945.
Oct 12 11:00:18 cdweb1 kernel: [263986.823045] UB-102-Mem-Info:
Oct 12 11:00:18 cdweb1 kernel: [263986.823891] Node 0 DMA scan:0 a_anon:0 i_anon:0 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:18 cdweb1 kernel: [263986.824735] Node 0 DMA32 scan:0 a_anon:186103 i_anon:8088 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:18 cdweb1 kernel: [263986.825583] Node 0 Normal scan:0 a_anon:22156 i_anon:19956 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:18 cdweb1 kernel: [263986.826432] Total 236303 anon:236303 file:0 a_anon:208259 i_anon:28044 a_file:0 i_file:0 unevictable:0
Oct 12 11:00:18 cdweb1 kernel: [263986.827296] UB: 102 RAM: 240209 / 262144 [4] SWAP: 2689 / 0 [4] KMEM: 13074432 / 9223372036854775807 [0] Dirty 65
Oct 12 11:00:19 cdweb1 kernel: [263987.212242] httpd invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
Oct 12 11:00:19 cdweb1 kernel: [263987.213181] httpd cpuset=102 mems_allowed=0
Oct 12 11:00:19 cdweb1 kernel: [263987.214054] Out of memory in UB: kill process 519542 (httpd) or a child
Oct 12 11:00:19 cdweb1 kernel: [263987.214930] Killed process 519542 (httpd) vsz:433924kB, anon-rss:325472kB, file-rss:568kB
Oct 12 11:00:19 cdweb1 kernel: [263987.289643] OOM killed process httpd (pid=519542, ve=102) exited, free=449995.
Oct 12 11:01:48 cdweb1 kernel: [264076.924977] httpd invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=0
Oct 12 11:01:48 cdweb1 kernel: [264076.925850] httpd cpuset=102 mems_allowed=0
Oct 12 11:01:48 cdweb1 kernel: [264076.926699] Out of memory in UB: kill process 521119 (mysqld) or a child
Oct 12 11:01:48 cdweb1 kernel: [264076.927564] Killed process 521119 (mysqld) vsz:566316kB, anon-rss:90784kB, file-rss:100kB
Oct 12 11:01:48 cdweb1 kernel: [264076.929631] OOM killed process mysqld (pid=521139, ve=102) exited, free=383884.
Oct 12 11:09:57 cdweb1 kernel: imklog 4.6.2, log source = /proc/kmsg started.
Oct 12 11:09:57 cdweb1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1625" x-info="ht tp :// www. rsyslog. com"] (re)start
Oct 12 11:09:57 cdweb1 kernel: [ 0.000000] Initializing cgroup subsys cpuset
Oct 12 11:09:57 cdweb1 kernel: [
...
https://www.nethub.com.hk (VPS Hosting Provider in Hong Kong)
|
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
