| Kernel panic? [message #44753] |
Thu, 29 December 2011 10:25  |
Griffith
Messages: 7
Registered: October 2007
|
Junior Member |
|
|
I've tried to migrate a few containers from a OpenVZ node running CentOS-5 -> CentOS-6.2.
While I was away for 5 minutes during a migration, I lost the connection to the server. After rebooting it from remote location I found this in the log files:
Dec 29 10:51:50 vz03 kernel: [58577.746434] ------------[ cut here ]------------
Dec 29 10:51:50 vz03 kernel: [58577.746448] WARNING: at kernel/bc/beancounter.c:430 delayed_release_beancounter+0x4a3/0x4d0() (Not tainted)
Dec 29 10:51:50 vz03 kernel: [58577.746453] Hardware name: PowerEdge R410
Dec 29 10:51:50 vz03 kernel: [58577.746456] Modules linked in: vzethdev simfs vzrst nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables vzevent vznetdev vzmon vzdev ipv6 dm_mirror dm_region_hash dm_log power_meter hwmon dcdbas microcode serio_raw ghes hed iTCO_wdt iTCO_vendor_support sg i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: speedstep_lib]
Dec 29 10:51:50 vz03 kernel: [58577.746516] Pid: 238, comm: ubcleand veid: 0 Not tainted 2.6.32-042stab044.11 #1
Dec 29 10:51:50 vz03 kernel: [58577.746520] Call Trace:
Dec 29 10:51:50 vz03 kernel: [58577.746529] [<ffffffff810697f7>] ? warn_slowpath_common+0x87/0xc0
Dec 29 10:51:50 vz03 kernel: [58577.746534] [<ffffffff8106984a>] ? warn_slowpath_null+0x1a/0x20
Dec 29 10:51:50 vz03 kernel: [58577.746540] [<ffffffff810a5263>] ? delayed_release_beancounter+0x4a3/0x4d0
Dec 29 10:51:50 vz03 kernel: [58577.746546] [<ffffffff814ca872>] ? thread_return+0x4e/0x85c
Dec 29 10:51:50 vz03 kernel: [58577.746553] [<ffffffff810935fe>] ? prepare_to_wait+0x4e/0x80
Dec 29 10:51:50 vz03 kernel: [58577.746558] [<ffffffff810a4dc0>] ? delayed_release_beancounter+0x0/0x4d0
Dec 29 10:51:50 vz03 kernel: [58577.746566] [<ffffffff8108cc30>] ? worker_thread+0x190/0x2d0
Dec 29 10:51:50 vz03 kernel: [58577.746571] [<ffffffff81093310>] ? autoremove_wake_function+0x0/0x40
Dec 29 10:51:50 vz03 kernel: [58577.746577] [<ffffffff8108caa0>] ? worker_thread+0x0/0x2d0
Dec 29 10:51:50 vz03 kernel: [58577.7Dec 29 11:06:18 vz03 kernel: imklog 4.6.2, log source = /proc/kmsg started.
I also noticed I had a few of these:
Dec 29 10:41:01 vz03 kernel: [57930.072415] Core dump to |/usr/libexec/abrt-hook-ccpp 11 0 29401 0 0 1325151661 e pipe failed
Any idea what causes this?
|
|
|
|
|
|
|
|
|
|
| Re: Kernel panic? [message #44901 is a reply to message #44753] |
Wed, 11 January 2012 18:04  |
mustardman
Messages: 91
Registered: October 2009
|
Member |
|
|
What version kernel?
uname -r
I had the same thing happen the other day running 042stab044.11 x64. Migrated a bunch of containers from a rhel5 i386 ovz node and got a kernel panic about an hour after the last container was migrated.
After reboot the node ran ok for about a day but I decided to move everything back to the rhel5 node as it's production and I cannot afford to deal with kernel panics.
[Updated on: Wed, 11 January 2012 18:15] Report message to a moderator |
|
|
|
|
|
| Re: Kernel panic? [message #44912 is a reply to message #44909] |
Thu, 12 January 2012 22:12 |
mustardman
Messages: 91
Registered: October 2009
|
Member |
|
|
There must be a problem with that kernel that comes up when you migrate VPS's. Ended up causing me to delay my upgrade plans for probably another year. That is good that you filed a bug report.
The OVZ kernel guys are pretty sharp and seem to be actively maintaining things. Kernel updates are happening fairly regularly. Only thing I noticed is they all seem to be Eastern European and some are not so fluent in English. So some people might interpret that as lack of support or understanding.
At least that's my theory... 
[Updated on: Thu, 12 January 2012 22:14] Report message to a moderator |
|
|
|
|
|
| Re: Kernel panic? [message #45436 is a reply to message #45197] |
Thu, 08 March 2012 08:04 |
iowissen
Messages: 11
Registered: February 2012
|
Junior Member |
|
|
we have the same trouble with beancouter warning and then the kernel panic, but no core dump (so different from bugzilla 2090, i think). bug info has filed into the thread of bugzilla 2124.
[ 2594.799934] WARNING: at kernel/bc/beancounter.c:430 delayed_release_beancounter+0x4a3/0x4d0() (Not tainted)
[ 2594.799957] Hardware name: PowerEdge R710
[ 2594.799967] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables ipt_REDIRECT nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_state xt_conntrack nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length ipt_LOG xt_hl xt_tcpmss xt_TCPMSS ipt_REJECT xt_DSCP xt_dscp xt_multiport xt_limit iptable_mangle iptable_filter ip_tables vzevent configfs i2c_dev i2c_core vznetdev vzmon vzdev bonding be2iscsi iscsi_boot_sysfs ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic uio ipv6 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_round_robin dm_multipath video output sbs sbshc acpi_pad parport_pc lp parport sg power_meter hwmon ses enclosure sr_mod cdrom serio_raw snd_pcsp bnx2 snd_pcm dcdbas tpm_tis tpm tpm_bios snd_timer i7core_edac snd edac_core pata_acpi iTCO_wdt soundcore iTCO_vendor_support ata_generic snd_page_alloc ghes hed dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix shpchp megaraid_sas sd_mod crc_t10dif ext3 jbd mbcache [last unloaded: freq_table]
[ 2594.801283] Pid: 238, comm: ubcleand veid: 0 Not tainted 2.6.32-042stab044.11 #1
[ 2594.801416] Call Trace:
[ 2594.802118] [<ffffffff810697f7>] ? warn_slowpath_common+0x87/0xc0
[ 2594.802195] [<ffffffff8106984a>] ? warn_slowpath_null+0x1a/0x20
[ 2594.802271] [<ffffffff810a5263>] ? delayed_release_beancounter+0x4a3/0x4d0
[ 2594.802346] [<ffffffff814ca872>] ? thread_return+0x4e/0x85c
[ 2594.802420] [<ffffffff810935fe>] ? prepare_to_wait+0x4e/0x80
[ 2594.802496] [<ffffffff810a4dc0>] ? delayed_release_beancounter+0x0/0x4d0
[ 2594.802572] [<ffffffff8108cc30>] ? worker_thread+0x190/0x2d0
[ 2594.802650] [<ffffffff81093310>] ? autoremove_wake_function+0x0/0x40
[ 2594.802729] [<ffffffff8108caa0>] ? worker_thread+0x0/0x2d0
[ 2594.802810] [<ffffffff81092d36>] ? kthread+0x96/0xa0
[ 2594.802885] [<ffffffff8100c2ca>] ? child_rip+0xa/0x20
[ 2594.802957] [<ffffffff81092ca0>] ? kthread+0x0/0xa0
[ 2594.803027] [<ffffffff8100c2c0>] ? child_rip+0x0/0x20
[ 2594.803097] ---[ end trace 8dd1d073138ba8af ]---
[ 2594.803169] UB: Bad refcount (-1) on put of 351 (ffff88041abbc180)
[ 2594.808744] Ub 351 helds -1 in refcount on put
[ 2594.808822] Ub 351 helds -1 in pincount on put
[ 2594.808891] UB: leaked beancounter 351 (ffff88041abbc180)
[ 2595.387837] CT: 353: stopped
[ 2596.849247] CT: 353: started
[ 2687.735205] CT: 295: stopped
[ 2687.814491] Ub 295 helds 18 in physpages on put
[ 2687.814579] UB: leaked beancounter 295 (ffff88081a55b200)
[ 2721.176150] CT: 354: stopped
[ 2721.781172] CT: 354: started
[ 2725.309205] CT: 352: stopped
[ 2845.744194] CT: 355: stopped
[ 2845.748494] Ub 355 helds 15 in physpages on put
[ 2845.748593] UB: leaked beancounter 355 (ffff880421fbc440)
[ 2853.750016] CT: 355: started
[ 2872.889755] CT: 313: stopped
[ 2874.738767] CT: 353: stopped
[ 2948.280057] CT: 354: stopped
[ 2968.639291] CT: 356: stopped
[ 2968.942380] CT: 356: started
[ 3028.594808] CT: 316: stopped
[ 3060.629938] CT: 357: stopped
[ 3060.631827] Ub 357 helds 12 in physpages on put
[ 3060.631900] UB: leaked beancounter 357 (ffff88081f020ec0)
[ 3061.680176] CT: 357: started
[ 3079.226591] CT: 355: stopped
[ 3128.893591] CT: 358: stopped
[ 3129.473643] CT: 358: started
[ 3206.064412] connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4297868617, last ping 4297873617, now 4297878617
[ 3206.064566] connection3:0: detected conn error (1011)
[ 3206.066375] connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4297868619, last ping 4297873619, now 4297878619
[ 3206.066526] connection1:0: detected conn error (1011)
[ 3209.472223] connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4297872031, last ping 4297877031, now 4297882031
[ 3209.472364] connection2:0: detected conn error (1011)
[ 3209.874545] connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4297872434, last ping 4297877434, now 4297882434
[ 3209.874690] connection4:0: detected conn error (1011)
[ 3270.669541] BUG: soft lockup - CPU#0 stuck for 67s! [apache2:27298]
[ 3270.669613] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables ipt_REDIRECT nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_state xt_conntrack nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length ipt_LOG xt_hl xt_tcpmss xt_TCPMSS ipt_REJECT xt_DSCP xt_dscp xt_multiport xt_limit iptable_mangle iptable_filter ip_tables vzevent configfs i2c_dev i2c_core vznetdev vzmon vzdev bonding be2iscsi iscsi_boot_sysfs ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic uio ipv6 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_round_robin dm_multipath video output sbs sbshc acpi_pad parport_pc lp parport sg power_meter hwmon ses enclosure sr_mod cdrom serio_raw snd_pcsp bnx2 snd_pcm dcdbas tpm_tis tpm tpm_bios snd_timer i7core_edac snd edac_core pata_acpi iTCO_wdt soundcore iTCO_vendor_support ata_generic snd_page_alloc ghes hed dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix shpchp megaraid_sas sd_mod crc_t10dif ext3 jbd mbcache [last unloaded: freq_table]
[ 3270.672095] CPU 0:
[ 3270.672097] Modules linked in: vzethdev simfs vzrst vzcpt nfs lockd fscache nfs_acl auth_rpcgss sunrpc vzdquota ip6t_REJECT ip6table_mangle ip6table_filter ip6_tables ipt_REDIRECT nf_nat_irc nf_nat_ftp iptable_nat nf_nat xt_helper xt_state xt_conntrack nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length ipt_LOG xt_hl xt_tcpmss xt_TCPMSS ipt_REJECT xt_DSCP xt_dscp xt_multiport xt_limit iptable_mangle iptable_filter ip_tables vzevent configfs i2c_dev i2c_core vznetdev vzmon vzdev bonding be2iscsi iscsi_boot_sysfs ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic uio ipv6 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_round_robin dm_multipath video output sbs sbshc acpi_pad parport_pc lp parport sg power_meter hwmon ses enclosure sr_mod cdrom serio_raw snd_pcsp bnx2 snd_pcm dcdbas tpm_tis tpm tpm_bios snd_timer i7core_edac snd edac_core pata_acpi iTCO_wdt soundcore iTCO_vendor_support ata_generic snd_page_alloc ghes hed dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod ata_piix shpchp megaraid_sas sd_mod crc_t10dif ext3 jbd mbcache [last unloaded: freq_table]
[ 3270.673414] Pid: 27298, comm: apache2 veid: 204 Tainted: G WC ---------------- 2.6.32-042stab044.11 #1 042stab044_11 PowerEdge R710
[ 3270.673418] RIP: 0010:[<ffffffff814cd14e>] [<ffffffff814cd14e>] _spin_lock+0x1e/0x30
[ 3270.673429] RSP: 0000:ffff8805f1601e58 EFLAGS: 00000287
[ 3270.673432] RAX: 0000000000009801 RBX: ffff8805f1601e58 RCX: 0000000000000000
[ 3270.673435] RDX: 00000000000097f1 RSI: ffffffff8179afc2 RDI: ffffffff823c6880
[ 3270.673438] RBP: ffffffff8100bd8e R08: 0000000000000000 R09: 0000000000000013
[ 3270.673442] R10: ffff8805f1600000 R11: 0000000000000000 R12: 1ee0d8ca00000000
[ 3270.673445] R13: 0000000000000000 R14: ffff88082e525380 R15: 0000000000000000
[ 3270.673448] FS: 0000000000000000(0000) GS:ffff880450400000(0063) knlGS:00000000b7707700
[ 3270.673452] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 3270.673455] CR2: 00007f4420d2b006 CR3: 00000006f2daf000 CR4: 00000000000006f0
[ 3270.673458] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3270.673462] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 3270.673465] Call Trace:
[ 3270.673531] [<ffffffff811a567b>] ? new_inode+0x3b/0xb0
[ 3270.673600] [<ffffffff81400fd9>] ? sock_alloc+0x19/0x60
[ 3270.673668] [<ffffffff8140143e>] ? __sock_create+0x9e/0x210
[ 3270.673737] [<ffffffff81401610>] ? sock_create+0x30/0x40
[ 3270.673805] [<ffffffff8140182c>] ? sys_socket+0x3c/0x80
[ 3270.673874] [<ffffffff8142ed87>] ? compat_sys_socketcall+0x67/0x200
[ 3270.673947] [<ffffffff81047d93>] ? ia32_sysret+0x0/0x5
[ 3270.674014] Kernel panic - not syncing: softlockup: hung tasks
[ 3270.674083] Pid: 27298, comm: apache2 veid: 204 Tainted: G WC ---------------- 2.6.32-042stab044.11 #1
[ 3270.674211] Call Trace:
[ 3270.674271] <IRQ> [<ffffffff814ca02c>] ? panic+0x78/0x143
[ 3270.674346] [<ffffffff810e5a60>] ? watchdog_timer_fn+0x0/0x1d0
[ 3270.674415] [<ffffffff810e5c1d>] ? watchdog_timer_fn+0x1bd/0x1d0
[ 3270.674486] [<ffffffff81097a6e>] ? __run_hrtimer+0x8e/0x1a0
[ 3270.674558] [<ffffffff81012999>] ? read_tsc+0x9/0x20
[ 3270.674626] [<ffffffff81097e16>] ? hrtimer_interrupt+0xe6/0x250
[ 3270.674697] [<ffffffff814d2c8b>] ? smp_apic_timer_interrupt+0x6b/0x9b
[ 3270.674768] [<ffffffff8100bd93>] ? apic_timer_interrupt+0x13/0x20
[ 3270.674836] <EOI> [<ffffffff814cd14e>] ? _spin_lock+0x1e/0x30
[ 3270.674909] [<ffffffff811a567b>] ? new_inode+0x3b/0xb0
[ 3270.674976] [<ffffffff81400fd9>] ? sock_alloc+0x19/0x60
[ 3270.675044] [<ffffffff8140143e>] ? __sock_create+0x9e/0x210
[ 3270.675114] [<ffffffff81401610>] ? sock_create+0x30/0x40
[ 3270.675183] [<ffffffff8140182c>] ? sys_socket+0x3c/0x80
[ 3270.675254] [<ffffffff8142ed87>] ? compat_sys_socket
...
|
|
|
|
| Re: Kernel panic? [message #45486 is a reply to message #44753] |
Tue, 13 March 2012 01:55 |
mustardman
Messages: 91
Registered: October 2009
|
Member |
|
|
I'm not saying this is any sort of fix. However, things seem to stabilize a bit after you reboot the node a couple times. So reboot once after migrating everything, make sure all migrated containers have been switched over to a vzswap template config file which uses the new default memory settings instead of the old beancounters template. Then reboot the node again. Also, don't allocate any vzswap memory. At least not at first until things seem stable then try enable it.
[Updated on: Tue, 13 March 2012 01:58] Report message to a moderator |
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
