OpenVZ Forum: Support » Problematic Hard Disk

Home » General » Support » Problematic Hard Disk - Read Only Mode

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Problematic Hard Disk - Read Only Mode [message #15865]

Wed, 15 August 2007 08:53

Ashley
Messages: 40
Registered: December 2006

Member

Hello,

I recently brought a new Dedicated Server from LayeredTech, installed OpenVZ and installed two VPS's and all works fine.

However at random times, can be up to 3 days between them or just a few hours the VPS will stop responding. Apache starts to throw an 500 error, and SSH an error about caught create something.

However at one point the problem happened during my already been logged into SSH, however when ever i went to do something it said, permission error (hard disk is loaded in ReadOnly mode).

This would explain for the Apache 500 error due to me running Suexec and it needing certain persmisons, and the SSH error as it can not create any files on the Hard Disk.

However the problem is i am not sure what is causing this, it happens both inside the VPS and the Host Node so i can not login to any SSH.

The datacentre has run many checks on the hard disk and changed over cabels just incase, however i now think its something to do with the setup.

I followed the guide word for word and this is the 3rd server that I have run OpenVZ on.

Any ideas?

Thanks Alot,
Ashley

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15866 is a reply to message #15865]

Wed, 15 August 2007 08:56

kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia

Senior Member

Running dmesg on the host system will probably tell you what happened to your hard drive -- some errors etc.

In some cases it might be a bad filesystem -- so it makes sense to run fsck on it.

Kir Kolyshkin

http://static.openvz.org/userbars/openvz-developer.png

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15867 is a reply to message #15865]

Wed, 15 August 2007 08:56

rickb
Messages: 368
Registered: October 2006

Senior Member

If the filesystem is new and hasn't been abused too badly (improper shutdowns whilst heavy io), usually it means a failed disk. post 'dmesg' when the fs goes into read only mode and we can tell you for sure.

Rick

-------------
Common Terms I post with: http://wiki.openvz.org/Category:Definitions

UBC. Learn it, love it, live it: http://wiki.openvz.org/Proc/user_beancounters

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15869 is a reply to message #15866]

Wed, 15 August 2007 09:04

Ashley
Messages: 40
Registered: December 2006

Member

They already run a fsck and it fixed a few Inodes, and i can not login via SSH when the server is readonly mode to grab a "dmesg"

The company did say i could have a OS Reload and Swap over the HardDisk however this will cost, and i don't want to do this if this is not the cause of the problem.

Thanks,
Ashley

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15870 is a reply to message #15869]

Wed, 15 August 2007 09:12

rickb
Messages: 368
Registered: October 2006

Senior Member

I would fire up bonnie++, and run that disk for a few hours. This should trigger the io errors if its a serious problem. Keep checking dmesg and save anything about seek errors, aborted journals, etc. Basically anything before it going into ro mode. Send that to your datacenter along with the fact that the fs was fsck'd clean just prior, and they will swap out the disk free of charge. Also, you should be using at least raid1 on this server. this will give you so many reliability and per. benefits, and will save you from reinstalling the OS when a disk goes south, like in this case.

Rick

-------------
Common Terms I post with: http://wiki.openvz.org/Category:Definitions

UBC. Learn it, love it, live it: http://wiki.openvz.org/Proc/user_beancounters

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15871 is a reply to message #15865]

Wed, 15 August 2007 09:16

Ashley
Messages: 40
Registered: December 2006

Member

Okie thanks

So if i set it up to run and then have a cronjob run every 2 minutes or so send me the output of dmesg to my email account?

As of course it could take hours for the error to appear???

Thanks,
Ashley

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15872 is a reply to message #15871]

Wed, 15 August 2007 09:29

rickb
Messages: 368
Registered: October 2006

Senior Member

run bonnie++ from the HN, there is no way to know how long to wait, I would say a few hours means your disk is ok.

Every 10 mins, check dmesg or rig a script to do this and mail to you I suppose.

Rick

-------------
Common Terms I post with: http://wiki.openvz.org/Category:Definitions

UBC. Learn it, love it, live it: http://wiki.openvz.org/Proc/user_beancounters

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15943 is a reply to message #15872]

Thu, 16 August 2007 15:16

Ashley
Messages: 40
Registered: December 2006

Member

Ok i run bonnie++ for a few hours, nothing appeared in the dmesg, i have attacked the whole output of dmesg bellow.

Any other ideas about what could cause the server to do this??

Thanks,
Ashley

Attachment: dmesg.txt
(Size: 18.20KB, Downloaded 342 times)

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15961 is a reply to message #15943]

Fri, 17 August 2007 05:01

vaverin
Messages: 708
Registered: September 2005

Senior Member

Hi Ashley,

disk remounts to ReadOnly in case of disk/filesystem errors and we need to see first error to understand the reasons of this issue.

Unfortunately in case disk/filesystem troubles the error messages cannot saved in local log files, therefore I would like to ask you to attach remote console:
http://wiki.openvz.org/Remote_console_setup

thank you,
Vasily Averin

Report message to a moderator

Re: Problematic Hard Disk - Read Only Mode [message #15968 is a reply to message #15961]

Fri, 17 August 2007 09:33

Ashley
Messages: 40
Registered: December 2006

Member

Thanks Vasily Averin

So there is no way this could be a software/configuration problem, as the datacentre has done many tests on the hard disk.

I have setup a netconsole, however the problem has not happened in the last few days.

So im hoping the problem has gone, however could it have been a config error, or would this not be able to do a remount in read only mode?

Thanks

Report message to a moderator

Previous Topic:	solved OpenVZ on CentOS 5 network issues
Next Topic:	Beancounters

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Sep 15 23:22:19 GMT 2024

Total time taken to generate the page: 0.03801 seconds