OpenVZ Forum


Home » General » Support » Random crashes / lockups - No errors, no video
Random crashes / lockups - No errors, no video [message #12424] Fri, 27 April 2007 17:15 Go to next message
tutt is currently offline  tutt
Messages: 19
Registered: April 2007
Junior Member
I've got a customer with OpenVZ and HyperVM. Since they have been hosting with us, they have been riddled with problems. We went through 2 VERY expensive machines and then replaced them with 4 new VERY expensive machines to rule out hardware problems. The machines are 4 each of:

Dual 5130 Woodcrest Dual CPU
X7DVL-E SuperMicro Motherboard
1U Rackmount Chassis
4x 146GB Seagate 10Krpm HDDs
6GB FBDIMM 667MHz RAM
Hardware RAID 10

They are running CentOS 4.4 64bit with the latest OpenVZ kernel:

2.6.9-023stab043.2-smp #1 SMP Fri Mar 9 11:50:20 MSK 2007 x86_64 x86_64 x86_64 GNU/Linux

One of the 4 machines has a different SuperMicro motherboard and a SuperMicro RAID controller while the other 3 have the configurations listed above.

3 out of 4 of these machines (including the one that has slightly different hardware including the main board and RAID controller) have locked up on a completely random basis. As far as we know, there are no related errors in /var/log/messages. When one of our techs brings up a console to the machine, there is no video whatsoever. Some of these machines will run for a week or two and then have this happen. Others will go through spurts of locking up every few days. Every once in a while, massive filesystem corruption causes additional headaches.

Before you say that it must be hardware, we have burn tested each of these machines. Memory, disk, CPU all check out OK. These 4 already replace 2 even more powerful machines. These are brand new SuperMicro machines. I would think the chance of 5 SuperMicro machines all with sporadic hardware issues has to be equivelent to the chance of 10 meteors falling on my head as I type this.

I am thinking that someone out there has a similar hardware and software config as my customer and that maybe someone else is experiencing the same issues? Does ANYONE have any ideas as this has gotten to the point of frustrating me and my customer to no end!

[Updated on: Fri, 27 April 2007 17:16]

Report message to a moderator

Re: Random crashes / lockups - No errors, no video [message #12426 is a reply to message #12424] Fri, 27 April 2007 19:39 Go to previous messageGo to next message
tutt is currently offline  tutt
Messages: 19
Registered: April 2007
Junior Member
On some of the machines locking up, the last errors seen in /var/log/messages are:

kernel: Fatal resource shortage: privvmpages, UB 2820.

Sometimes the VPS identified in the error will restart a few times too.

What is this privvmpages resource shortage error all about?
Re: Random crashes / lockups - No errors, no video [message #12437 is a reply to message #12424] Sat, 28 April 2007 07:00 Go to previous messageGo to next message
Vasily Tarasov is currently offline  Vasily Tarasov
Messages: 1345
Registered: January 2006
Senior Member
Hello,

do you have a serial console? If yes, when machine is locked, please, press magic keys (http://wiki.openvz.org/Magic_SysRq_Key) and post console output to us.

As concerns resource shortage, you can read about it here: http://wiki.openvz.org/Resource_shortage

Thank you,
Vasily.
Re: Random crashes / lockups - No errors, no video [message #12790 is a reply to message #12424] Wed, 09 May 2007 15:16 Go to previous messageGo to next message
tutt is currently offline  tutt
Messages: 19
Registered: April 2007
Junior Member
Are these magic keys supposed to work with a keyboard hooked up to the machine physically? Or does it need to be a serial console?
Re: Random crashes / lockups - No errors, no video [message #12792 is a reply to message #12424] Wed, 09 May 2007 16:10 Go to previous messageGo to next message
sspt is currently offline  sspt
Messages: 100
Registered: August 2006
Location: Portugal
Senior Member
I had random crashes a few months ago.
No errors in serial console so I've replaced board, hdd's, CPU, PSU but it crashed again and again so i decided to reloaded it with CentOS 32bit and it's working fine now.


http://static.openvz.org/userbars/openvz-user.png
Re: Random crashes / lockups - No errors, no video [message #12797 is a reply to message #12792] Wed, 09 May 2007 20:36 Go to previous messageGo to next message
tutt is currently offline  tutt
Messages: 19
Registered: April 2007
Junior Member
Can you tell me what hardware you are running and which kernel versions where running when you had the crashes? Did they start after a particular kernel version?
Re: Random crashes / lockups - No errors, no video [message #12823 is a reply to message #12790] Fri, 11 May 2007 07:34 Go to previous messageGo to next message
Vasily Tarasov is currently offline  Vasily Tarasov
Messages: 1345
Registered: January 2006
Senior Member
You can use your usual physical keyboard connected to the server. Serial console is required in order all messages to go not only to the screen of your display, but also to the serial console, where you can save all these messages and send to us.

HTH,
Vasily.
Re: Random crashes / lockups - No errors, no video [message #12840 is a reply to message #12797] Fri, 11 May 2007 09:00 Go to previous messageGo to next message
sspt is currently offline  sspt
Messages: 100
Registered: August 2006
Location: Portugal
Senior Member
After the second crash i've tried the last two 2.6.9 and 2.6.18 kernels but it happen again and again with cpu usage around 10-15% (0%wa).

This happened with both AMD Athlon 64 x2 4200+ and Pentium IV EMT64, the second one without any vps running on it.


http://static.openvz.org/userbars/openvz-user.png
Re: Random crashes / lockups - No errors, no video [message #12852 is a reply to message #12840] Fri, 11 May 2007 16:11 Go to previous messageGo to next message
tutt is currently offline  tutt
Messages: 19
Registered: April 2007
Junior Member
I wonder if it is the same problem my customer is having. Do you happen to know any of the following:

Was the machine pingable?

Was there video on the screen if you or your remote hands could physically check it?

Did the machine seem powered on still despite being locked up?

Please let me know as I am starting to hope you have the answer to our problems.
Re: Random crashes / lockups - No errors, no video [message #12865 is a reply to message #12852] Sat, 12 May 2007 20:10 Go to previous messageGo to next message
sspt is currently offline  sspt
Messages: 100
Registered: August 2006
Location: Portugal
Senior Member
tutt wrote on Fri, 11 May 2007 17:11


Was the machine pingable?


No, the box wasn't pingable

tutt wrote on Fri, 11 May 2007 17:11


Was there video on the screen if you or your remote hands could physically check it?


Not even the cursor blinking or kernel panic

tutt wrote on Fri, 11 May 2007 17:11


Did the machine seem powered on still despite being locked up?


PSU and fans are working


http://static.openvz.org/userbars/openvz-user.png
Re: Random crashes / lockups - No errors, no video [message #12989 is a reply to message #12865] Thu, 17 May 2007 02:54 Go to previous message
tutt is currently offline  tutt
Messages: 19
Registered: April 2007
Junior Member
We tried re-installing 32bit CentOS on one of the machines with no luck. What kernel version are you running now that these random lockups have disappeared? Other than switching to 32-bit CentOS, was there ANYTHING else you did that may have played a role in fixing the problem?
Previous Topic: *SOLVED* VE monitoring
Next Topic: *SOLVED* How to secure /tmp?
Goto Forum:
  


Current Time: Sun Aug 11 00:58:41 GMT 2024

Total time taken to generate the page: 0.02845 seconds