OpenVZ Forum: Support » Main Server Hangs

Home » General » Support » Main Server Hangs (Server randomly hangs and becomes unavailable )

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Main Server Hangs [message #43847]

Mon, 24 October 2011 13:57

YaoDzi
Messages: 2
Registered: October 2011

Junior Member

Hello,

The main server hangs due to the overload, CPU load gets CRITICAL - load average: 121.90 with more than 1200 processes.

We have checked the statistics of each virtual server and didn't find any troublemaker there. What other things should we check or how to find the cause of this overload ?

Recent log when it froze:

Quote:

Oct 24 11:46:50 v1 kernel: INFO: task events/0:18 blocked for more than 300 seconds.
Oct 24 11:46:50 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:50 v1 kernel: events/0 D ffff81032ebff360 0 18 1 19 17 (L-TLB)
Oct 24 11:46:50 v1 kernel: ffff81032ee01dc0 0000000000000046 0000000000000000 0000000100000002
Oct 24 11:46:50 v1 kernel: ffff81032ebff360 ffff81032f30c160 00018a72fa95cb4c 000276225002dd32
Oct 24 11:46:50 v1 kernel: ffff81032ebff568 0000000480499d00 0000000000000000 ffff81032f308000
Oct 24 11:46:50 v1 kernel: Call Trace:
Oct 24 11:46:50 v1 kernel: [<ffffffff8023a575>] linkwatch_event+0x0/0x30
Oct 24 11:46:50 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:50 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:50 v1 kernel: [<ffffffff8023a57e>] linkwatch_event+0x9/0x30
Oct 24 11:46:50 v1 kernel: [<ffffffff8004fb4c>] run_workqueue+0x9e/0xfc
Oct 24 11:46:50 v1 kernel: [<ffffffff8004c2b8>] worker_thread+0x0/0x122
Oct 24 11:46:50 v1 kernel: [<ffffffff8004c3a8>] worker_thread+0xf0/0x122
Oct 24 11:46:50 v1 kernel: [<ffffffff8008b636>] default_wake_function+0x0/0xe
Oct 24 11:46:50 v1 kernel: [<ffffffff80033a47>] kthread+0xfe/0x132
Oct 24 11:46:50 v1 kernel: [<ffffffff80061001>] child_rip+0xa/0x11
Oct 24 11:46:50 v1 kernel: [<ffffffff80033949>] kthread+0x0/0x132
Oct 24 11:46:50 v1 kernel: [<ffffffff80060ff7>] child_rip+0x0/0x11
Oct 24 11:46:50 v1 kernel:
Oct 24 11:46:50 v1 xinetd[10793]: EXIT: nrpe status=0 pid=121739 duration=10(sec)
Oct 24 11:46:50 v1 kernel: INFO: task irqbalance:10395 blocked for more than 300 seconds.
Oct 24 11:46:50 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:50 v1 kernel: irqbalance D ffff81032cc90260 0 10395 1 10426 9215 (NOTLB)
Oct 24 11:46:50 v1 kernel: ffff81032106dd78 0000000000000086 3738313633303839 000000000000006c
Oct 24 11:46:50 v1 kernel: ffff81032cc90260 ffff8100418d3660 00018a790d47804d 0002762c03a4248c
Oct 24 11:46:50 v1 kernel: ffff81032cc90468 000000002711fa28 ffff8102ea36f680 ffff81031187e000
Oct 24 11:46:50 v1 kernel: Call Trace:
Oct 24 11:46:50 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:50 v1 kernel: [<ffffffff80231e8f>] dev_name_hash+0x1e/0x64
Oct 24 11:46:50 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:50 v1 kernel: [<ffffffff802322c9>] dev_load+0x18/0x46
Oct 24 11:46:50 v1 kernel: [<ffffffff802329e1>] dev_ioctl+0x317/0x497
Oct 24 11:46:50 v1 kernel: [<ffffffff80060e39>] error_exit+0x0/0x84
Oct 24 11:46:50 v1 kernel: [<ffffffff80227730>] sock_ioctl+0x1d4/0x1e5
Oct 24 11:46:50 v1 kernel: [<ffffffff80043f27>] do_ioctl+0x21/0x6b
Oct 24 11:46:50 v1 kernel: [<ffffffff80031550>] vfs_ioctl+0x457/0x4b9
Oct 24 11:46:50 v1 kernel: [<ffffffff800c3842>] audit_syscall_entry+0x1a8/0x1d3
Oct 24 11:46:50 v1 kernel: [<ffffffff8004ec0e>] sys_ioctl+0x3c/0x5c
Oct 24 11:46:50 v1 kernel: [<ffffffff800602dd>] tracesys+0xd5/0xe0
Oct 24 11:46:50 v1 kernel:
Oct 24 11:46:50 v1 kernel: INFO: task ntpd:17253 blocked for more than 300 seconds.
Oct 24 11:46:50 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:50 v1 kernel: ntpd D ffff810316328960 0 17253 16562 18452 17104 (NOTLB)
Oct 24 11:46:50 v1 kernel: ffff8102f26b1cb8 0000000000200086 0000000000000000 0000000000000000
Oct 24 11:46:51 v1 kernel: ffff810316328960 ffff81032f3f2220 00018a7b5d8c2b7c 0002762fb5cac0a4
Oct 24 11:46:51 v1 kernel: ffff810316328b68 00000007173f2e40 0000000000000000 ffff81032f3ee000
Oct 24 11:46:51 v1 kernel: Call Trace:
Oct 24 11:46:51 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:51 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:51 v1 kernel: [<ffffffff8001abaf>] vsnprintf+0x5df/0x627
Oct 24 11:46:51 v1 kernel: [<ffffffff802326ee>] dev_ioctl+0x24/0x497
Oct 24 11:46:51 v1 kernel: [<ffffffff80227730>] sock_ioctl+0x1d4/0x1e5
Oct 24 11:46:51 v1 kernel: [<ffffffff80043f27>] do_ioctl+0x21/0x6b
Oct 24 11:46:51 v1 kernel: [<ffffffff80031550>] vfs_ioctl+0x457/0x4b9
Oct 24 11:46:51 v1 kernel: [<ffffffff80109026>] inotify_d_instantiate+0x12/0x3c
Oct 24 11:46:51 v1 kernel: [<ffffffff8004ec0e>] sys_ioctl+0x3c/0x5c
Oct 24 11:46:51 v1 kernel: [<ffffffff8010f267>] dev_ifconf+0xe5/0x1ab
Oct 24 11:46:51 v1 kernel: [<ffffffff8010d20a>] compat_sys_ioctl+0x24e/0x292
Oct 24 11:46:51 v1 kernel: [<ffffffff80062766>] ia32_sysret+0x0/0x5
Oct 24 11:46:51 v1 kernel:
Oct 24 11:46:51 v1 kernel: INFO: task nmbd:18732 blocked for more than 300 seconds.
Oct 24 11:46:51 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:51 v1 kernel: nmbd D ffff810315fccae0 0 18732 18550 938512 18727 (NOTLB)
Oct 24 11:46:51 v1 kernel: ffff8102eb0edb58 0000000000000082 ffff8102eb0eddd8 ffff8102eb0edde0
Oct 24 11:46:51 v1 kernel: ffff810315fccae0 ffff81032f39a1e0 00018a797e82e2b6 0002762cb8879978
Oct 24 11:46:51 v1 kernel: ffff810315fccce8 0000000600000000 0000000000000000 ffff81032f396000
Oct 24 11:46:51 v1 kernel: Call Trace:
Oct 24 11:46:51 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:51 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:51 v1 kernel: [<ffffffff800ab824>] ub_sock_getwres_other+0x94/0xc3
Oct 24 11:46:51 v1 kernel: [<ffffffff80239afd>] rtnetlink_rcv+0x15/0x3b
Oct 24 11:46:51 v1 kernel: [<ffffffff80248109>] netlink_data_ready+0x12/0x50
Oct 24 11:46:51 v1 kernel: [<ffffffff80248c18>] netlink_sendmsg+0x4a4/0x4d8
Oct 24 11:46:51 v1 kernel: [<ffffffff8008b636>] default_wake_function+0x0/0xe
Oct 24 11:46:51 v1 kernel: [<ffffffff80057ce7>] sock_sendmsg+0xe5/0x137
Oct 24 11:46:51 v1 kernel: [<ffffffff800a3453>] autoremove_wake_function+0x0/0x2e
Oct 24 11:46:51 v1 kernel: [<ffffffff8001abaf>] vsnprintf+0x5df/0x627
Oct 24 11:46:51 v1 kernel: [<ffffffff800f890d>] core_sys_select+0x21d/0x27f
Oct 24 11:46:51 v1 kernel: [<ffffffff8002f0c8>] __wake_up+0x38/0x4f
Oct 24 11:46:51 v1 kernel: [<ffffffff80228a4f>] sys_sendto+0x131/0x164
Oct 24 11:46:51 v1 kernel: [<ffffffff802281a9>] move_addr_to_user+0x5d/0x78
Oct 24 11:46:51 v1 kernel: [<ffffffff802286b4>] sys_getsockname+0x72/0xa2
Oct 24 11:46:51 v1 kernel: [<ffffffff800602dd>] tracesys+0xd5/0xe0
Oct 24 11:46:51 v1 kernel:
Oct 24 11:46:51 v1 kernel: INFO: task nmbd:26759 blocked for more than 300 seconds.
Oct 24 11:46:51 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:51 v1 kernel: nmbd D ffff8102b38d65e0 0 26759 25068 26762 26750 (NOTLB)
Oct 24 11:46:51 v1 kernel: ffff810296d7db58 0000000000000082 ffff8102b38d67e8 0000000680030d42
Oct 24 11:46:51 v1 kernel: ffff8102b38d65e0 ffff81032f30c160 00018a73ded436d0 00027623bca1af74
Oct 24 11:46:51 v1 kernel: ffff8102b38d67e8 000000048003f7bc 0000000000000000 ffff81032f308000
Oct 24 11:46:51 v1 kernel: Call Trace:
Oct 24 11:46:51 v1 kernel: [<ffffffff8004d560>] try_to_del_timer_sync+0x7f/0x88
Oct 24 11:46:51 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:51 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:51 v1 kernel: [<ffffffff800ab824>] ub_sock_getwres_other+0x94/0xc3
Oct 24 11:46:51 v1 kernel: [<ffffffff80239afd>] rtnetlink_rcv+0x15/0x3b
Oct 24 11:46:51 v1 kernel: [<ffffffff80248109>] netlink_data_ready+0x12/0x50
Oct 24 11:46:51 v1 kernel: [<ffffffff80248c18>] netlink_sendmsg+0x4a4/0x4d8
Oct 24 11:46:51 v1 kernel: [<ffffffff8008b636>] default_wake_function+0x0/0xe
Oct 24 11:46:51 v1 kernel: [<ffffffff80057ce7>] sock_sendmsg+0xe5/0x137
Oct 24 11:46:51 v1 kernel: [<ffffffff8005ab1a>] __ip_route_output_key+0x17a/0x97f
Oct 24 11:46:51 v1 kernel: [<ffffffff800a3453>] autoremove_wake_function+0x0/0x2e
Oct 24 11:46:51 v1 kernel: [<ffffffff8001abaf>] vsnprintf+0x5df/0x627
Oct 24 11:46:51 v1 kernel: [<ffffffff8002f0c8>] __wake_up+0x38/0x4f
Oct 24 11:46:51 v1 kernel: [<ffffffff80228a4f>] sys_sendto+0x131/0x164
Oct 24 11:46:51 v1 kernel: [<ffffffff802281a9>] move_addr_to_user+0x5d/0x78
Oct 24 11:46:51 v1 kernel: [<ffffffff802286b4>] sys_getsockname+0x72/0xa2
Oct 24 11:46:51 v1 kernel: [<ffffffff800602dd>] tracesys+0xd5/0xe0
Oct 24 11:46:51 v1 kernel:
Oct 24 11:46:52 v1 kernel: INFO: task dsm_om_connsvcd:43862 blocked for more than 300 seconds.
Oct 24 11:46:52 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:52 v1 kernel: dsm_om_connsv D ffff81022509cb60 0 43862 43703 43918 43861 (NOTLB)
Oct 24 11:46:52 v1 kernel: ffff810220ba3b58 0000000000000082 ffff810324cad440 ffff810058bc08c8
Oct 24 11:46:52 v1 kernel: ffff81022509cb60 ffff81032f2460a0 00018a82fbe5f51c 0002763be17c22b6
Oct 24 11:46:52 v1 kernel: ffff81022509cd68 0000000180089baa 0000000000000000 ffff81032f242000
Oct 24 11:46:52 v1 kernel: Call Trace:
Oct 24 11:46:52 v1 kernel: [<ffffffff8008ae62>] find_busiest_group+0x177/0x638
Oct 24 11:46:52 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:52 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:52 v1 kernel: [<ffffffff800ab824>] ub_sock_getwres_other+0x94/0xc3
Oct 24 11:46:52 v1 kernel: [<ffffffff80239afd>] rtnetlink_rcv+0x15/0x3b
Oct 24 11:46:52 v1 kernel: [<ffffffff80248109>] netlink_data_ready+0x12/0x50
Oct 24 11:46:52 v1 kernel: [<ffffffff80248c18>] netlink_sendmsg+0x4a4/0x4d8
Oct 24 11:46:52 v1 kernel: [<ffffffff8008b636>] default_wake_function+0x0/0xe
Oct 24 11:46:52 v1 kernel: [<ffffffff80057ce7>] sock_sendmsg+0xe5/0x137
Oct 24 11:46:52 v1 kernel: [<ffffffff800a3453>] autoremove_wake_function+0x0/0x2e
Oct 24 11:46:52 v1 kernel: [<ffffffff8001abaf>] vsnprintf+0x5df/0x627
Oct 24 11:46:52 v1 kernel: [<ffffffff8002f0c8>] __wake_up+0x38/0x4f
Oct 24 11:46:52 v1 kernel: [<ffffffff80228a4f>] sys_sendto+0x131/0x164
Oct 24 11:46:52 v1 kernel: [<ffffffff802281a9>] move_addr_to_user+0x5d/0x78
Oct 24 11:46:52 v1 kernel: [<ffffffff802286ce>] sys_getsockname+0x8c/0xa2
Oct 24 11:46:52 v1 kernel: [<ffffffff800602dd>] tracesys+0xd5/0xe0
Oct 24 11:46:52 v1 kernel:
Oct 24 11:46:52 v1 kernel: INFO: task php:66525 blocked for more than 300 seconds.
Oct 24 11:46:52 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:52 v1 kernel: php D ffff81030c5c2d20 0 66525 66410 (NOTLB)
Oct 24 11:46:52 v1 kernel: ffff810082d47cb8 0000000000000082 ffffffff88707150 ffffffff80329a80
Oct 24 11:46:52 v1 kernel: ffff81030c5c2d20 ffff8100462112e0 00018a72fe0007b2 000276225577756e
Oct 24 11:46:52 v1 kernel: ffff81030c5c2f28 8000000000666025 ffff810037abd5a0 ffff81026888e000
Oct 24 11:46:52 v1 kernel: Call Trace:
Oct 24 11:46:52 v1 kernel: [<ffffffff80008c04>] __handle_mm_fault+0xb79/0xd9d
Oct 24 11:46:52 v1 kernel: [<ffffffff800d4be9>] __rmqueue+0x44/0xca
Oct 24 11:46:52 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:52 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:52 v1 kernel: [<ffffffff802326ee>] dev_ioctl+0x24/0x497
Oct 24 11:46:52 v1 kernel: [<ffffffff800ac143>] __pb_add_ref+0x12b/0x1bc
Oct 24 11:46:52 v1 kernel: [<ffffffff800a8ebd>] __ub_unused_privvm_dec+0x47/0x63
Oct 24 11:46:52 v1 kernel: [<ffffffff80008c04>] __handle_mm_fault+0xb79/0xd9d
Oct 24 11:46:52 v1 kernel: [<ffffffff80227730>] sock_ioctl+0x1d4/0x1e5
Oct 24 11:46:53 v1 kernel: [<ffffffff80043f27>] do_ioctl+0x21/0x6b
Oct 24 11:46:53 v1 kernel: [<ffffffff80031550>] vfs_ioctl+0x457/0x4b9
Oct 24 11:46:53 v1 kernel: [<ffffffff8004ec0e>] sys_ioctl+0x3c/0x5c
Oct 24 11:46:53 v1 kernel: [<ffffffff8010f267>] dev_ifconf+0xe5/0x1ab
Oct 24 11:46:53 v1 kernel: [<ffffffff8010d20a>] compat_sys_ioctl+0x24e/0x292
Oct 24 11:46:53 v1 kernel: [<ffffffff80062766>] ia32_sysret+0x0/0x5
Oct 24 11:46:53 v1 kernel:
Oct 24 11:46:53 v1 kernel: INFO: task openvpn:67926 blocked for more than 300 seconds.
Oct 24 11:46:53 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:53 v1 kernel: openvpn D ffff810040962ba0 0 67926 954839 (L-TLB)
Oct 24 11:46:53 v1 kernel: ffff81028839bd68 0000000000200046 0000000100000000 ffff8102400ab208
Oct 24 11:46:53 v1 kernel: ffff810040962ba0 ffffffff8031aba0 00018a72fa95a934 000276225002a6b4
Oct 24 11:46:53 v1 kernel: ffff810040962da8 0000000000000000 0000000000000000 ffffffff804a4280
Oct 24 11:46:53 v1 kernel: Call Trace:
Oct 24 11:46:53 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:53 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:53 v1 kernel: [<ffffffff88932a20>] :tun:tun_chr_close+0x32/0x78
Oct 24 11:46:53 v1 kernel: [<ffffffff800128ef>] __fput+0xd3/0x1c2
Oct 24 11:46:53 v1 kernel: [<ffffffff800248f8>] filp_close+0x5c/0x64
Oct 24 11:46:53 v1 kernel: [<ffffffff8003a949>] put_files_struct+0x63/0xae
Oct 24 11:46:53 v1 kernel: [<ffffffff80015ba0>] do_exit+0x74c/0xe2d
Oct 24 11:46:53 v1 kernel: [<ffffffff800644f6>] thread_return+0x6a/0x177
Oct 24 11:46:53 v1 kernel: [<ffffffff800c3842>] audit_syscall_entry+0x1a8/0x1d3
Oct 24 11:46:53 v1 kernel: [<ffffffff8004b5f5>] cpuset_exit+0x0/0x88
Oct 24 11:46:53 v1 kernel: [<ffffffff80062766>] ia32_sysret+0x0/0x5
Oct 24 11:46:53 v1 kernel:
Oct 24 11:46:53 v1 kernel: INFO: task php:67927 blocked for more than 300 seconds.
Oct 24 11:46:53 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:53 v1 kernel: php D ffff810194296ce0 0 67927 31492 (NOTLB)
Oct 24 11:46:53 v1 kernel: ffff8100778abcb8 0000000000000082 ffffffff88707150 ffffffff80329a80
Oct 24 11:46:53 v1 kernel: ffff810194296ce0 ffff8100462112e0 00018a7303c3de2e 000276225ead1da2
Oct 24 11:46:53 v1 kernel: ffff810194296ee8 8000000000666025 ffff81008446c448 ffff81026888e000
Oct 24 11:46:53 v1 kernel: Call Trace:
Oct 24 11:46:53 v1 kernel: [<ffffffff80008c04>] __handle_mm_fault+0xb79/0xd9d
Oct 24 11:46:53 v1 kernel: [<ffffffff800d4be9>] __rmqueue+0x44/0xca
Oct 24 11:46:53 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:53 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:53 v1 kernel: [<ffffffff802326ee>] dev_ioctl+0x24/0x497
Oct 24 11:46:53 v1 kernel: [<ffffffff800ac143>] __pb_add_ref+0x12b/0x1bc
Oct 24 11:46:53 v1 kernel: [<ffffffff800a8ebd>] __ub_unused_privvm_dec+0x47/0x63
Oct 24 11:46:53 v1 kernel: [<ffffffff80008c04>] __handle_mm_fault+0xb79/0xd9d
Oct 24 11:46:53 v1 kernel: [<ffffffff80227730>] sock_ioctl+0x1d4/0x1e5
Oct 24 11:46:53 v1 kernel: [<ffffffff80043f27>] do_ioctl+0x21/0x6b
Oct 24 11:46:53 v1 kernel: [<ffffffff80031550>] vfs_ioctl+0x457/0x4b9
Oct 24 11:46:53 v1 kernel: [<ffffffff8004ec0e>] sys_ioctl+0x3c/0x5c
Oct 24 11:46:53 v1 kernel: [<ffffffff8010f267>] dev_ifconf+0xe5/0x1ab
Oct 24 11:46:53 v1 kernel: [<ffffffff8010d20a>] compat_sys_ioctl+0x24e/0x292
Oct 24 11:46:53 v1 kernel: [<ffffffff80062766>] ia32_sysret+0x0/0x5
Oct 24 11:46:53 v1 kernel:
Oct 24 11:46:53 v1 kernel: INFO: task php:67932 blocked for more than 300 seconds.
Oct 24 11:46:53 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:53 v1 kernel: php D ffff81011f7920a0 0 67932 66409 (NOTLB)
Oct 24 11:46:53 v1 kernel: ffff81006db45cb8 0000000000000086 ffffffff88707150 ffffffff80329a80
Oct 24 11:46:53 v1 kernel: ffff81011f7920a0 ffff8100a40c2da0 00018a73118589ca 0002762274a6fe90
Oct 24 11:46:53 v1 kernel: ffff81011f7922a8 8000000000666025 ffff81000ab02a80 ffff81026888e000
Oct 24 11:46:53 v1 kernel: Call Trace:
Oct 24 11:46:53 v1 kernel: [<ffffffff80008c04>] __handle_mm_fault+0xb79/0xd9d
Oct 24 11:46:53 v1 kernel: [<ffffffff800d4be9>] __rmqueue+0x44/0xca
Oct 24 11:46:53 v1 kernel: [<ffffffff8006520d>] __mutex_lock_slowpath+0x60/0x9b
Oct 24 11:46:53 v1 kernel: [<ffffffff80065257>] .text.lock.mutex+0xf/0x14
Oct 24 11:46:53 v1 kernel: [<ffffffff802326ee>] dev_ioctl+0x24/0x497
Oct 24 11:46:53 v1 kernel: [<ffffffff800ac143>] __pb_add_ref+0x12b/0x1bc
Oct 24 11:46:53 v1 kernel: [<ffffffff800a8ebd>] __ub_unused_privvm_dec+0x47/0x63
Oct 24 11:46:53 v1 kernel: [<ffffffff80008c04>] __handle_mm_fault+0xb79/0xd9d
Oct 24 11:46:53 v1 kernel: [<ffffffff80227730>] sock_ioctl+0x1d4/0x1e5
Oct 24 11:46:53 v1 kernel: [<ffffffff80043f27>] do_ioctl+0x21/0x6b
Oct 24 11:46:53 v1 kernel: [<ffffffff80031550>] vfs_ioctl+0x457/0x4b9
Oct 24 11:46:54 v1 kernel: [<ffffffff8004ec0e>] sys_ioctl+0x3c/0x5c
Oct 24 11:46:54 v1 kernel: [<ffffffff8010f267>] dev_ifconf+0xe5/0x1ab
Oct 24 11:46:54 v1 kernel: [<ffffffff8010d20a>] compat_sys_ioctl+0x24e/0x292
Oct 24 11:46:54 v1 kernel: [<ffffffff80062766>] ia32_sysret+0x0/0x5
Oct 24 11:46:54 v1 kernel:
Oct 24 11:47:11 v1 xinetd[10793]: START: nrpe pid=123632 from=195.222.15.79
Oct 24 11:47:11 v1 xinetd[10793]: EXIT: nrpe status=0 pid=123632 duration=0(sec)
Oct 24 11:47:47 v1 xinetd[10793]: EXIT: nrpe status=0 pid=121785 duration=60(sec)
Oct 24 11:47:53 v1 xinetd[10793]: START: nrpe pid=126992 from=195.222.15.79
Oct 24 11:47:53 v1 xinetd[10793]: EXIT: nrpe status=0 pid=126992 duration=0(sec)
Oct 24 11:48:36 v1 xinetd[10793]: START: nrpe pid=131254 from=195.222.15.79
Oct 24 11:48:36 v1 xinetd[10793]: EXIT: nrpe status=0 pid=131254 duration=0(sec)
Oct 24 11:48:47 v1 xinetd[10793]: START: nrpe pid=133142 from=195.222.15.79
Oct 24 11:48:57 v1 xinetd[10793]: START: nrpe pid=133189 from=195.222.15.79
Oct 24 11:48:57 v1 xinetd[10793]: EXIT: nrpe status=0 pid=133189 duration=0(sec)
Oct 24 11:49:11 v1 xinetd[10793]: START: nrpe pid=134031 from=195.222.15.79
Oct 24 11:49:11 v1 xinetd[10793]: EXIT: nrpe status=0 pid=134031 duration=0(sec)
Oct 24 11:49:47 v1 xinetd[10793]: EXIT: nrpe status=0 pid=133142 duration=60(sec)
Oct 24 11:50:36 v1 xinetd[10793]: START: nrpe pid=142170 from=195.222.15.79
Oct 24 11:50:36 v1 xinetd[10793]: EXIT: nrpe status=0 pid=142170 duration=0(sec)
Oct 24 11:50:47 v1 xinetd[10793]: START: nrpe pid=142316 from=195.222.15.79
Oct 24 11:51:11 v1 xinetd[10793]: START: nrpe pid=145578 from=195.222.15.79
Oct 24 11:51:11 v1 xinetd[10793]: EXIT: nrpe status=0 pid=145578 duration=0(sec)
Oct 24 11:51:25 v1 xinetd[10793]: START: nrpe pid=146051 from=195.222.15.79
Oct 24 11:51:25 v1 xinetd[10793]: START: nrpe pid=146052 from=195.222.15.79
Oct 24 11:51:25 v1 xinetd[10793]: EXIT: nrpe status=0 pid=146051 duration=0(sec)
Oct 24 11:51:25 v1 xinetd[10793]: EXIT: nrpe status=0 pid=146052 duration=0(sec)
Oct 24 11:51:40 v1 xinetd[10793]: START: nrpe pid=149896 from=195.222.15.79
Oct 24 11:51:47 v1 xinetd[10793]: EXIT: nrpe status=0 pid=142316 duration=60(sec)
Oct 24 11:51:47 v1 xinetd[10793]: START: nrpe pid=149925 from=195.222.15.79
Oct 24 11:51:47 v1 xinetd[10793]: EXIT: nrpe status=0 pid=149925 duration=0(sec)
Oct 24 11:51:50 v1 xinetd[10793]: EXIT: nrpe status=0 pid=149896 duration=10(sec)
Oct 24 11:52:53 v1 xinetd[10793]: START: nrpe pid=154095 from=195.222.15.79
Oct 24 11:52:53 v1 xinetd[10793]: EXIT: nrpe status=0 pid=154095 duration=0(sec)
Oct 24 11:53:57 v1 xinetd[10793]: START: nrpe pid=158603 from=195.222.15.79
Oct 24 11:53:57 v1 xinetd[10793]: EXIT: nrpe status=0 pid=158603 duration=0(sec)
Oct 24 11:58:21 v1 syslogd 1.4.1: restart.
Oct 24 11:58:21 v1 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 24 11:58:21 v1 kernel: Linux version 2.6.18-274.el5.028stab093.2 (root@rhel5-build-x64) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Aug 23 16:46:17 MSD 2011
Oct 24 11:58:21 v1 kernel: Command line: ro root=LABEL=/1 selinux=0
Oct 24 11:58:21 v1 kernel: BIOS-provided physical RAM map:
Oct 24 11:58:21 v1 kernel: BIOS-e820: 0000000000010000 - 00000000000a0000 (usable)
Oct 24 11:58:21 v1 kernel: BIOS-e820: 0000000000100000 - 00000000cfb50000 (usable)
Oct 24 11:58:21 v1 kernel: BIOS-e820: 00000000cfb50000 - 00000000cfb66000 (reserved)
Oct 24 11:58:21 v1 kernel: BIOS-e820: 00000000cfb66000 - 00000000cfb85c00 (ACPI data)
Oct 24 11:58:21 v1 kernel: BIOS-e820: 00000000cfb85c00 - 00000000d0000000 (reserved)
Oct 24 11:58:21 v1 kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
Oct 24 11:58:21 v1 kernel: BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved)
Oct 24 11:58:21 v1 kernel: BIOS-e820: 0000000100000000 - 0000000330000000 (usable)
Oct 24 11:58:21 v1 kernel: DMI 2.5 present.
Oct 24 11:58:21 v1 kernel: No NUMA configuration found
Oct 24 11:58:21 v1 kernel: Faking a node at 0000000000000000-0000000330000000
Oct 24 11:58:21 v1 kernel: Bootmem setup node 0 0000000000000000-0000000330000000
Oct 24 11:58:21 v1 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Oct 24 11:58:21 v1 kernel: disabling kdump
Oct 24 11:58:21 v1 kernel: ACPI: PM-Timer IO Port: 0x808
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Oct 24 11:58:21 v1 kernel: Processor #0 6:15 APIC version 20
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Oct 24 11:58:21 v1 kernel: Processor #4 6:15 APIC version 20
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
Oct 24 11:58:21 v1 kernel: Processor #2 6:15 APIC version 20
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
Oct 24 11:58:21 v1 kernel: Processor #6 6:15 APIC version 20
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
Oct 24 11:58:21 v1 kernel: Processor #1 6:15 APIC version 20
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled)
Oct 24 11:58:21 v1 kernel: Processor #5 6:15 APIC version 20
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x03] enabled)
Oct 24 11:58:21 v1 kernel: Processor #3 6:15 APIC version 20
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
Oct 24 11:58:21 v1 kernel: Processor #7 6:15 APIC version 20
Oct 24 11:58:21 v1 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
Oct 24 11:58:21 v1 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
Oct 24 11:58:21 v1 kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
Oct 24 11:58:21 v1 kernel: ACPI: IOAPIC (id[0x09] address[0xfec81000] gsi_base[64])
Oct 24 11:58:21 v1 kernel: IOAPIC[1]: apic_id 9, version 32, address 0xfec81000, GSI 64-87
Oct 24 11:58:21 v1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Oct 24 11:58:22 v1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Oct 24 11:58:22 v1 kernel: Setting APIC routing to flat
Oct 24 11:58:22 v1 kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000
Oct 24 11:58:22 v1 kernel: Using ACPI (MADT) for SMP configuration information
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000000a0000 - 0000000000100000
Oct 24 11:58:22 v1 rpc.statd[10512]: Version 1.0.9 Starting
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000cfb50000 - 00000000cfb66000
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000cfb66000 - 00000000cfb85000
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000cfb85000 - 00000000cfb86000
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000cfb86000 - 00000000d0000000
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000d0000000 - 00000000e0000000
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000e0000000 - 00000000f0000000
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000f0000000 - 00000000fe000000
Oct 24 11:58:22 v1 kernel: Nosave address range: 00000000fe000000 - 0000000100000000
Oct 24 11:58:22 v1 kernel: Allocating PCI resources starting at d1000000 (gap: d0000000:10000000)
Oct 24 11:58:22 v1 kernel: Built 1 zonelists. Total pages: 3090744
Oct 24 11:58:22 v1 kernel: Kernel command line: ro root=LABEL=/1 selinux=0

Report message to a moderator

Re: Main Server Hangs [message #43858 is a reply to message #43847]

Tue, 25 October 2011 10:31

YaoDzi
Messages: 2
Registered: October 2011

Junior Member

One more error:

kernel: Fatal resource shortage: privvmpages, UB 137.

Quote:

Oct 24 11:45:02 v1 kernel: Fatal resource shortage: privvmpages, UB 137.
Oct 24 11:45:47 v1 xinetd[10793]: EXIT: nrpe status=0 pid=109139 duration=60(sec)
Oct 24 11:45:53 v1 xinetd[10793]: START: nrpe pid=118811 from=x.x.x.x
Oct 24 11:45:53 v1 xinetd[10793]: EXIT: nrpe status=0 pid=118811 duration=0(sec)
Oct 24 11:46:25 v1 xinetd[10793]: START: nrpe pid=120903 from=x.x.x.x
Oct 24 11:46:25 v1 xinetd[10793]: START: nrpe pid=120904 from=x.x.x.x
Oct 24 11:46:25 v1 xinetd[10793]: EXIT: nrpe status=0 pid=120903 duration=0(sec)
Oct 24 11:46:25 v1 xinetd[10793]: EXIT: nrpe status=0 pid=120904 duration=0(sec)
Oct 24 11:46:36 v1 xinetd[10793]: START: nrpe pid=121335 from=x.x.x.x
Oct 24 11:46:36 v1 xinetd[10793]: EXIT: nrpe status=0 pid=121335 duration=0(sec)
Oct 24 11:46:40 v1 xinetd[10793]: START: nrpe pid=121739 from=x.x.x.x
Oct 24 11:46:47 v1 xinetd[10793]: START: nrpe pid=121784 from=x.x.x.x
Oct 24 11:46:47 v1 xinetd[10793]: START: nrpe pid=121785 from=x.x.x.x
Oct 24 11:46:47 v1 xinetd[10793]: EXIT: nrpe status=0 pid=121784 duration=0(sec)
Oct 24 11:46:50 v1 kernel: INFO: task events/0:18 blocked for more than 300 seconds.
Oct 24 11:46:50 v1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 24 11:46:50 v1 kernel: events/0 D ffff81032ebff360 0 18 1 19 17 (L-TLB)
Oct 24 11:46:50 v1 kernel: ffff81032ee01dc0 0000000000000046 0000000000000000 0000000100000002
Oct 24 11:46:50 v1 kernel: ffff81032ebff360 ffff81032f30c160 00018a72fa95cb4c 000276225002dd32
Oct 24 11:46:50 v1 kernel: ffff81032ebff568 0000000480499d00 0000000000000000 ffff81032f308000
Oct 24 11:46:50 v1 kernel: Call Trace: