OpenVZ Forum


Home » General » Support » Weird Disk Spac Problem
Weird Disk Spac Problem [message #32795] Fri, 29 August 2008 14:22 Go to next message
mperkel is currently offline  mperkel
Messages: 253
Registered: December 2006
Senior Member
I have one server (Than I'm using as my main web server) where HTTP crashes and won't start saying that it has run out of disk space. However - there is plenty of disk space. Over 300 gigs free.

Fri Aug 29 07:00:27 2008] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.12239) (5)

The server has a single VE. But I did have an issue earlier where the disk free (df) didn't show simfs in the list. So I did a hard link /etc/mtab -> /proc/mounts and that put simfs back into the list. I had the problem before I did the hard link so I'm thinking there is something else wrong and my simlink masked it.

Why would simfs not show up in df? And why would df show plenty of space free at the time when apache thinks it is out of space?

Thanks in advance.


Re: Weird Disk Spac Problem [message #32796 is a reply to message #32795] Fri, 29 August 2008 15:31 Go to previous messageGo to next message
maratrus is currently offline  maratrus
Messages: 1495
Registered: August 2007
Location: Moscow
Senior Member
Hi,

I cannot understand if you talk about HN or about VE?
Where is httpd running?

Quote:


I did a hard link /etc/mtab -> /proc/mounts


hard link?

Quote:


Why would simfs not show up in df?



Because df gets its information from /etc/mtab?

Quote:


And why would df show plenty of space free at the time when apache thinks it is out of space?



Did you check user_beancounters?
May be disk inodes were expired? (df -i)

[Updated on: Fri, 29 August 2008 15:33]

Report message to a moderator

Re: Weird Disk Spac Problem [message #32797 is a reply to message #32795] Fri, 29 August 2008 16:43 Go to previous messageGo to next message
mperkel is currently offline  mperkel
Messages: 253
Registered: December 2006
Senior Member
Sorry - I meant simlink.

Inodes are 3% in use.

No overages on the bea counters. I'm not using quotas.


[Updated on: Fri, 29 August 2008 16:44]

Report message to a moderator

Re: Weird Disk Space Problem [message #32798 is a reply to message #32795] Fri, 29 August 2008 17:19 Go to previous messageGo to next message
mperkel is currently offline  mperkel
Messages: 253
Registered: December 2006
Senior Member
Also - restarting the VZ service clears the error.

Re: Weird Disk Space Problem [message #32799 is a reply to message #32795] Fri, 29 August 2008 19:11 Go to previous messageGo to next message
mperkel is currently offline  mperkel
Messages: 253
Registered: December 2006
Senior Member
Still investigating. Might have been a bean counter issue. One change I'm making to all my servers. Added this to my /etc/vz/vz.conf file.

# Primary parameters
AVNUMPROC="unlimited:unlimited"
NUMPROC="unlimited:unlimited"
NUMTCPSOCK="unlimited:unlimited"
NUMOTHERSOCK="unlimited:unlimited"
VMGUARPAGES="unlimited:unlimited"

# Secondary parameters
KMEMSIZE="2G:2G"
TCPSNDBUF="unlimited:unlimited"
TCPRCVBUF="unlimited:unlimited"
OTHERSOCKBUF="unlimited:unlimited"
DGRAMRCVBUF="unlimited:unlimited"
OOMGUARPAGES="unlimited:unlimited"

# Auxiliary parameters
LOCKEDPAGES="unlimited:unlimited"
SHMPAGES="unlimited:unlimited"
PRIVVMPAGES="unlimited:unlimited"
NUMFILE="unlimited:unlimited"
NUMFLOCK="unlimited:unlimited"
NUMPTY="unlimited:unlimited"
NUMSIGINFO="unlimited:unlimited"
DCACHESIZE="unlimited:unlimited"

PHYSPAGES="unlimited:unlimited"
NUMIPTENT="unlimited:unlimited"




Re: Weird Disk Spac Problem [message #32832 is a reply to message #32795] Mon, 01 September 2008 15:12 Go to previous messageGo to next message
lj2012 is currently offline  lj2012
Messages: 7
Registered: February 2008
Location: Antwerp - Belgrium
Junior Member

weird diskspace errors sometimes come up when copying to a ve straight from the main machine. SimFS is not aware, so you need to reset quota.

I don't know if that has anything to do with your problem, nevertheless... Think there is a wiki page too...

... yup:
http://wiki.openvz.org/Disk_quota%2C_df_and_stat_weird_behav iour#Other_reasons_of_strange_numbers

(only bottom section)

Good luck,
L.
Re: Weird Disk Spac Problem [message #32843 is a reply to message #32795] Tue, 02 September 2008 15:46 Go to previous messageGo to next message
kenjy is currently offline  kenjy
Messages: 50
Registered: July 2008
Location: Mexico
Member
What web server are you ussing? when you get the error please do a lsof |grep deleted the last day I had a nginx issue whit logrotate and the HD filling my hard disk.

http://kenjy.net |Live net !!!
Re: Weird Disk Spac Problem [message #32845 is a reply to message #32795] Tue, 02 September 2008 15:54 Go to previous messageGo to next message
mperkel is currently offline  mperkel
Messages: 253
Registered: December 2006
Senior Member
I'll look into that as it runs. Seeing a few of these:

httpd 9712 apache 710w REG 0,20 0 13369352 (deleted) /var/log/httpd/ssl_mutex
httpd 9713 apache 710w REG 0,20 0 13369352 (deleted) /var/log/httpd/ssl_mutex

Logrotate runs once a week. But HTTPD is restarted lately fairly often. But the server was locked up this morning with the same disk space complaint.

Here's what the logs were showing today.

[Tue Sep 02 05:08:02 2008] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed


Re: Weird Disk Spac Problem [message #32847 is a reply to message #32795] Tue, 02 September 2008 16:03 Go to previous messageGo to next message
lj2012 is currently offline  lj2012
Messages: 7
Registered: February 2008
Location: Antwerp - Belgrium
Junior Member

you didn't leave selinux on, did you?
Re: Weird Disk Spac Problem [message #32848 is a reply to message #32795] Tue, 02 September 2008 16:14 Go to previous messageGo to next message
mperkel is currently offline  mperkel
Messages: 253
Registered: December 2006
Senior Member
Nope - I get rid of that right away.

Re: Weird Disk Spac Problem [message #32863 is a reply to message #32795] Thu, 04 September 2008 14:51 Go to previous message
kenjy is currently offline  kenjy
Messages: 50
Registered: July 2008
Location: Mexico
Member
You need to correctly log rotate your ssl_mutex log file, well I think that this is the problem:


httpd 9712 apache 710w REG 0,20 0 13369352 (deleted) /var/log/httpd/ssl_mutex
httpd 9713 apache 710w REG 0,20 0 13369352 (deleted) /var/log/httpd/ssl_mutex

You have a deleted file who is growing in some way. I fix it adding this to log rotate but this is in nginx web server with this script, obviously it works rotating logs each night because we want a log each night (my logs are about 2gb so I need to make a slide at the middle of the day):

www:/opt/mantenimiento# cat nginx_logs.sh 
#Obtenemos la hora actual, dependiendo de si son las 00 o las 12 se rotara el log
hora=$(date +%H)
ayer=$(date -d yesterday +%Y-%m-%d)
hoy=$(date  +%Y-%m-%d)
ruta=/var/log/nginx

if [ "$hora" = "00" ]; then
	mv $ruta/access.log $ruta/jornada_access_$ayer-12.log
        if [ -f /var/run/nginx.pid ]; then
		kill -USR1 `cat /var/run/nginx.pid`
	fi
	
	cat $ruta/jornada_access_$ayer-12.log >> /$ruta/jornada_access_$ayer-00.log
	mv $ruta/jornada_access_$ayer-00.log $ruta/jornada_access_$ayer.log
	rm $ruta/jornada_access_$ayer-12.log
	gzip -9 $ruta/jornada_access_$ayer.log


elif [ "$hora" = "12" ]; then
	mv $ruta/access.log $ruta/jornada_access_$hoy-00.log
	if [ -f /var/run/nginx.pid ]; then
        	kill -USR1 `cat /var/run/nginx.pid`
        fi

else
	echo "No se procesa, fuera de hora."
	

fi


www:/opt/mantenimiento# 


http://kenjy.net |Live net !!!
Previous Topic: No internet on the running vps
Next Topic: Dedicated File Cache
Goto Forum:
  


Current Time: Sun Jul 27 06:54:35 GMT 2025

Total time taken to generate the page: 0.30166 seconds