Hey guys, Here's my problem... Basically, we have a node running on CentOS 5.3 64-bit and it has been solid forever. And then 2-3 days ago we upgraded the kernel from 028stab062.3 to 028stab064.4. About 24-48 hours after the upgrade, the node crashed. The datacenter rebooted it and we checked the logs... nothing. So exactly 24 hours after the first crash, it happened again. I asked the datacenter to give me all of the information they had and I was told: "Looks like a kernel panic related to a crond process, something about __posix_lock_file_conf". A few hours later, it crashed again. The problem is I just can't pinpoint where this is coming from. There are no oops messages, no conflicts, nothing starting right before the crash, nothing. There's simply nothing being logged when this happens. Anyway, I know this is a long-shot, but has anyone ever experienced a problem related to "posix_lock_file_conf"? I've done the obvious and reverted to the old kernel for now, which so far has remained stable. The node is a quad-core with 8GB ram and normally runs at about 0.10 to 0.40 with around 6GB of ram actually used. So yeah, has anyone had any issues with this latest kernel?
Hi. i'm afraid you faced some new issue, so we are very interested in details on it. Could you please configure a serial console on the node (or at least - network console) http://wiki.openvz.org/Remote_console_setup Unfortunately without logs we can do nothing so we have to wait next crash - please, collect the logs, take picture of the oops on the local console and file a bug in bugzilla for this issue. Good luck! -- Konstantin
You need to gather console logs (full kernel oops message etc.), see http://wiki.openvz.org/Remote_console_setup. This is very important to get this info. When done, file a bug report directly to bugzilla.openvz.org. And no, I haven't seen any complaints like this so far.
Thanks for the suggestions everyone... Unfortunately I don't have the ability to setup something like serial console. The only thing I can really do is ask the datacenter to take a picture next time if they can. However, I'm in no hurry to submit a bug, because since rolling back to the old kernel everything has been stable (knock on wood). I'll try to recreate this on a test server if I can.
Hello, We saw what looks like this exact same crash this morning. We're on RHEL 5.3, kernel 2.6.18-128.2.1.el5.028stab064.4. Unfortunately I do not have a picture or the complete text of the panic, however I wrote down the stack trace on a notepad. the process was sshd. We had some rdiff-backups running at the time over ssh on the physical machine, however it is also possible that the sshd process in question was running in a VE. The stack trace was: __posix_lock_file_conf fnctl_setlk do_sigaction sys_fnctl system_call and the actual function that crashed it: locks_alloc_lock I'll try to setup remote console so that I can give you the complete stack trace if this happens again.
Hello, We are using same kernel and it crashed on the same hour three consecutive days. Since we were not sure if it was hardware related we moved all VE's to another hardware (with a different kernel). I believe we have the same issue here. Couldn't get kernel message because it is a colocated box. Is this issue fixed on newer kernel? Now I have netconsole enabled on this server which had problems but could not reproduce the crash (after we moved all VE's). Even running stress tests for 10+ hours everything seems fine (hardware seems to be 100%).
Again, without kernel console logs we can't say or do much. Please reproduce the crash, get the logs and file a bug to http://bugzilla.openvz.org/ Thanks!
kernel 028stab064.7 fixes a crash bug: http://wiki.openvz.org/Download/kernel/rhel5/028stab064.7 I'm hoping that it resolves the issue that I had. I haven't had any problems since upgrading to it.