OpenVZ Forum: Support » zabbix-agent inside OpenVZ container

Home » General » Support » zabbix-agent inside OpenVZ container

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

zabbix-agent inside OpenVZ container [message #44849]

Sun, 08 January 2012 11:51

reaper
Messages: 3
Registered: January 2012
Location: Russia

Junior Member

Today I stumbled upon a nasty system behaviour. I was trying to migrate (not online) OpenVZ container with zabbix-agent inside from one node to another. In the last part of migration where container is being stopped on source node I've got kernel oops and zabbix process became unresponsive. That behaviour made entire node unresponsive as all new vzctl processed are ending up in D state immediately.

After some attempts I was able to reproduce this on another node. Nodes are all Dell R710/R810 running Debian Squeeze with latest stable kernel. OpenVZ containers are also Debian Squeeze with zabbix-agent 1.8.2 from stable repository. I've also tried zabbix-agent 1.8.9 from testing with exactly the same result.

Anyone can confirm this behaviour? Here's Oops messages from syslog:

Jan  8 04:43:21 wz-us13 kernel: [1636769.658596] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
Jan  8 04:43:21 wz-us13 kernel: [1636769.658651] IP: [<ffffffff812eb2e0>] _spin_lock+0x5/0x1b
Jan  8 04:43:21 wz-us13 kernel: [1636769.658689] PGD 16f89be067 PUD 16f89bf067 PMD 0
Jan  8 04:43:21 wz-us13 kernel: [1636769.658722] Oops: 0002 [#1] SMP
Jan  8 04:43:21 wz-us13 kernel: [1636769.658750] last sysfs file: /sys/module/inet_diag/initstate
Jan  8 04:43:21 wz-us13 kernel: [1636769.658779] CPU 6
Jan  8 04:43:21 wz-us13 kernel: [1636769.658802] Modules linked in: tcp_diag inet_diag iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 vzethdev vznetdev simfs vzrst vzcpt vzdquota vzmon vzdev xt_tcpudp xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables x_tables vzevent xfs exportfs dm_snapshot loop snd_pcm snd_timer joydev psmouse snd soundcore snd_page_alloc usbhid hid serio_raw pcspkr evdev dcdbas processor power_meter button ext3 jbd mbcache ses sd_mod crc_t10dif enclosure dm_mod uhci_hcd thermal ehci_hcd usbcore megaraid_sas bnx2 nls_base scsi_mod thermal_sys [last unloaded: scsi_wait_scan]
Jan  8 04:43:21 wz-us13 kernel: [1636769.659202] Pid: 3478, comm: zabbix_agentd Not tainted 2.6.32-5-openvz-amd64 #1 feoktistov PowerEdge R510
Jan  8 04:43:21 wz-us13 kernel: [1636769.659251] RIP: 0010:[<ffffffff812eb2e0>]  [<ffffffff812eb2e0>] _spin_lock+0x5/0x1b
Jan  8 04:43:21 wz-us13 kernel: [1636769.659302] RSP: 0018:ffff8808ad0cfea0  EFLAGS: 00010202
Jan  8 04:43:21 wz-us13 kernel: [1636769.659330] RAX: 0000000000010000 RBX: ffff8820342932a0 RCX: ffff8816d21e2010
Jan  8 04:43:21 wz-us13 kernel: [1636769.659375] RDX: ffff8816d21e2070 RSI: ffff8816d21e2010 RDI: 0000000000000004
Jan  8 04:43:21 wz-us13 kernel: [1636769.659419] RBP: ffff8816d21e2010 R08: 0000000082ec0f5c R09: 0000000000000069
Jan  8 04:43:21 wz-us13 kernel: [1636769.659464] R10: ffffffff812eab18 R11: 0000000000000008 R12: ffff8816d21e2048
Jan  8 04:43:21 wz-us13 kernel: [1636769.659509] R13: ffff88171b172c00 R14: ffff8816d21e2070 R15: 0000000000000000
Jan  8 04:43:21 wz-us13 kernel: [1636769.659555] FS:  00007fc730f78700(0000) GS:ffff881080860000(0000) knlGS:0000000000000000
Jan  8 04:43:21 wz-us13 kernel: [1636769.659601] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  8 04:43:21 wz-us13 kernel: [1636769.659630] CR2: 0000000000000004 CR3: 00000008ad0c2000 CR4: 00000000000006e0
Jan  8 04:43:21 wz-us13 kernel: [1636769.659675] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan  8 04:43:21 wz-us13 kernel: [1636769.659720] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan  8 04:43:21 wz-us13 kernel: [1636769.659766] Process zabbix_agentd (pid: 3478, veid=109, threadinfo ffff8808ad0ce000, task ffff88103387d800)
Jan  8 04:43:21 wz-us13 kernel: [1636769.659815] Stack:
Jan  8 04:43:21 wz-us13 kernel: [1636769.659836]  ffffffff8114cdd4 ffff8816d21e2010 0000000000000000 0000000000000000
Jan  8 04:43:21 wz-us13 kernel: [1636769.659874] <0> ffff88171b172c00 ffff8808ad0cfee8 ffffffff8114d146 0000000000000000
Jan  8 04:43:21 wz-us13 kernel: [1636769.659929] <0> 0000000000000000 00000000000007dc ffffffff810f4a11 0000000082ec0f5c
Jan  8 04:43:21 wz-us13 kernel: [1636769.660002] Call Trace:
Jan  8 04:43:21 wz-us13 kernel: [1636769.660029]  [<ffffffff8114cdd4>] ? freeary+0x6c/0x144
Jan  8 04:43:21 wz-us13 kernel: [1636769.660058]  [<ffffffff8114d146>] ? sys_semctl+0x29a/0x2d7
Jan  8 04:43:21 wz-us13 kernel: [1636769.660089]  [<ffffffff810f4a11>] ? sys_newstat+0x23/0x30
Jan  8 04:43:21 wz-us13 kernel: [1636769.660119]  [<ffffffff81010c12>] ? system_call_fastpath+0x16/0x1b
Jan  8 04:43:21 wz-us13 kernel: [1636769.660148] Code: 00 00 00 01 74 05 e8 60 6d e9 ff 48 89 d0 5e c3 fa 66 0f 1f 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 46 6d e9 ff c3 b8 00 00 01 00 <f0> 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 eb f5
Jan  8 04:43:21 wz-us13 kernel: [1636769.660370] RIP  [<ffffffff812eb2e0>] _spin_lock+0x5/0x1b
Jan  8 04:43:21 wz-us13 kernel: [1636769.660401]  RSP <ffff8808ad0cfea0>
Jan  8 04:43:21 wz-us13 kernel: [1636769.660425] CR2: 0000000000000004
Jan  8 04:43:21 wz-us13 kernel: [1636769.660779] ---[ end trace 0dafff67eb788883 ]---
Jan  8 04:44:26 wz-us13 kernel: [1636834.585745] BUG: soft lockup - CPU#6 stuck for 61s! [zabbix_agentd:3478]
Jan  8 04:44:26 wz-us13 kernel: [1636834.585819] Modules linked in: tcp_diag inet_diag iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 vzethdev vznetdev simfs vzrst vzcpt vzdquota vzmon vzdev xt_tcpudp xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables x_tables vzevent xfs exportfs dm_snapshot loop snd_pcm snd_timer joydev psmouse snd soundcore snd_page_alloc usbhid hid serio_raw pcspkr evdev dcdbas processor power_meter button ext3 jbd mbcache ses sd_mod crc_t10dif enclosure dm_mod uhci_hcd thermal ehci_hcd usbcore megaraid_sas bnx2 nls_base scsi_mod thermal_sys [last unloaded: scsi_wait_scan]
Jan  8 04:44:26 wz-us13 kernel: [1636834.588977] CPU 6:
Jan  8 04:44:26 wz-us13 kernel: [1636834.589080] Modules linked in: tcp_diag inet_diag iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 vzethdev vznetdev simfs vzrst vzcpt vzdquota vzmon vzdev xt_tcpudp xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables x_tables vzevent xfs exportfs dm_snapshot loop snd_pcm snd_timer joydev psmouse snd soundcore snd_page_alloc usbhid hid serio_raw pcspkr evdev dcdbas processor power_meter button ext3 jbd mbcache ses sd_mod crc_t10dif enclosure dm_mod uhci_hcd thermal ehci_hcd usbcore megaraid_sas bnx2 nls_base scsi_mod thermal_sys [last unloaded: scsi_wait_scan]
Jan  8 04:44:26 wz-us13 kernel: [1636834.592231] Pid: 3478, comm: zabbix_agentd Tainted: G      D    2.6.32-5-openvz-amd64 #1 feoktistov PowerEdge R510
Jan  8 04:44:26 wz-us13 kernel: [1636834.592321] RIP: 0010:[<ffffffff812eb2f0>]  [<ffffffff812eb2f0>] _spin_lock+0x15/0x1b
Jan  8 04:44:26 wz-us13 kernel: [1636834.592458] RSP: 0018:ffff8808ad0cfbc0  EFLAGS: 00000297
Jan  8 04:44:26 wz-us13 kernel: [1636834.592525] RAX: 0000000000000004 RBX: ffff88103387d800 RCX: 0000000000000000
Jan  8 04:44:26 wz-us13 kernel: [1636834.592609] RDX: 0000000000000003 RSI: 0000000000000001 RDI: ffff8816d21e2010
Jan  8 04:44:26 wz-us13 kernel: [1636834.592693] RBP: ffffffff8101172e R08: ffff881080872990 R09: 0000000002d70a40
Jan  8 04:44:26 wz-us13 kernel: [1636834.592777] R10: ffff881034383a00 R11: 0000000000000000 R12: ffff881080873050
Jan  8 04:44:26 wz-us13 kernel: [1636834.592861] R13: ffff88203385f148 R14: 0000000039e84170 R15: ffffffff8146a600
Jan  8 04:44:26 wz-us13 kernel: [1636834.592945] FS:  0000000000000000(0000) GS:ffff881080860000(0000) knlGS:0000000000000000
Jan  8 04:44:26 wz-us13 kernel: [1636834.593032] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan  8 04:44:26 wz-us13 kernel: [1636834.593099] CR2: 0000000000000004 CR3: 0000000001001000 CR4: 00000000000006e0
Jan  8 04:44:26 wz-us13 kernel: [1636834.593183] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan  8 04:44:26 wz-us13 kernel: [1636834.593267] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan  8 04:44:26 wz-us13 kernel: [1636834.593351] Call Trace:
Jan  8 04:44:26 wz-us13 kernel: [1636834.593417]  [<ffffffff8114a40a>] ? ipc_lock+0x2a/0x42
Jan  8 04:44:26 wz-us13 kernel: [1636834.593485]  [<ffffffff8114a42e>] ? ipc_lock_check+0xc/0x3c
Jan  8 04:44:26 wz-us13 kernel: [1636834.593553]  [<ffffffff8114c1c1>] ? exit_sem+0x8a/0x1d5
Jan  8 04:44:26 wz-us13 kernel: [1636834.593623]  [<ffffffff81051c72>] ? do_exit+0x23c/0x758
Jan  8 04:44:26 wz-us13 kernel: [1636834.593691]  [<ffffffff812ec1cc>] ? oops_end+0xaf/0xb4
Jan  8 04:44:26 wz-us13 kernel: [1636834.593760]  [<ffffffff81032357>] ? no_context+0x1e9/0x1f8
Jan  8 04:44:26 wz-us13 kernel: [1636834.593830]  [<ffffffff81101885>] ? dput+0xf4/0x1cc
Jan  8 04:44:26 wz-us13 kernel: [1636834.593897]  [<ffffffff8103250a>] ? __bad_area_nosemaphore+0x1a4/0x1c8
Jan  8 04:44:26 wz-us13 kernel: [1636834.593969]  [<ffffffffa019226b>] ? sim_systemcall+0x92/0x263 [simfs]
Jan  8 04:44:26 wz-us13 kernel: [1636834.594042]  [<ffffffff81106842>] ? mntput_no_expire+0x23/0xed
Jan  8 04:44:26 wz-us13 kernel: [1636834.594112]  [<ffffffff810e7745>] ? virt_to_head_page+0x9/0x2a
Jan  8 04:44:26 wz-us13 kernel: [1636834.594181]  [<ffffffff812ed6e4>] ? do_page_fault+0x1bf/0x2fc
Jan  8 04:44:26 wz-us13 kernel: [1636834.594250]  [<ffffffff812eb695>] ? page_fault+0x25/0x30
Jan  8 04:44:26 wz-us13 kernel: [1636834.594319]  [<ffffffff812eab18>] ? down_write+0x9/0x27
Jan  8 04:44:26 wz-us13 kernel: [1636834.594387]  [<ffffffff812eb2e0>] ? _spin_lock+0x5/0x1b
Jan  8 04:44:26 wz-us13 kernel: [1636834.599445]  [<ffffffff8114cdd4>] ? freeary+0x6c/0x144
Jan  8 04:44:26

...

[ Show the rest of the message ]

Report message to a moderator