Weird Disk Spac Problem [message #32795] |
Fri, 29 August 2008 14:22  |
mperkel
Messages: 253 Registered: December 2006
|
Senior Member |
|
|
I have one server (Than I'm using as my main web server) where HTTP crashes and won't start saying that it has run out of disk space. However - there is plenty of disk space. Over 300 gigs free.
Fri Aug 29 07:00:27 2008] [emerg] (28)No space left on device: Couldn't create accept lock (/etc/httpd/logs/accept.lock.12239) (5)
The server has a single VE. But I did have an issue earlier where the disk free (df) didn't show simfs in the list. So I did a hard link /etc/mtab -> /proc/mounts and that put simfs back into the list. I had the problem before I did the hard link so I'm thinking there is something else wrong and my simlink masked it.
Why would simfs not show up in df? And why would df show plenty of space free at the time when apache thinks it is out of space?
Thanks in advance.
Junk Email Filter
http://www.junkemailfilter.com
|
|
|
Re: Weird Disk Spac Problem [message #32796 is a reply to message #32795] |
Fri, 29 August 2008 15:31   |
maratrus
Messages: 1495 Registered: August 2007 Location: Moscow
|
Senior Member |
|
|
Hi,
I cannot understand if you talk about HN or about VE?
Where is httpd running?
Quote: |
I did a hard link /etc/mtab -> /proc/mounts
|
hard link?
Quote: |
Why would simfs not show up in df?
|
Because df gets its information from /etc/mtab?
Quote: |
And why would df show plenty of space free at the time when apache thinks it is out of space?
|
Did you check user_beancounters?
May be disk inodes were expired? (df -i)
[Updated on: Fri, 29 August 2008 15:33] Report message to a moderator
|
|
|
|
|
Re: Weird Disk Space Problem [message #32799 is a reply to message #32795] |
Fri, 29 August 2008 19:11   |
mperkel
Messages: 253 Registered: December 2006
|
Senior Member |
|
|
Still investigating. Might have been a bean counter issue. One change I'm making to all my servers. Added this to my /etc/vz/vz.conf file.
# Primary parameters
AVNUMPROC="unlimited:unlimited"
NUMPROC="unlimited:unlimited"
NUMTCPSOCK="unlimited:unlimited"
NUMOTHERSOCK="unlimited:unlimited"
VMGUARPAGES="unlimited:unlimited"
# Secondary parameters
KMEMSIZE="2G:2G"
TCPSNDBUF="unlimited:unlimited"
TCPRCVBUF="unlimited:unlimited"
OTHERSOCKBUF="unlimited:unlimited"
DGRAMRCVBUF="unlimited:unlimited"
OOMGUARPAGES="unlimited:unlimited"
# Auxiliary parameters
LOCKEDPAGES="unlimited:unlimited"
SHMPAGES="unlimited:unlimited"
PRIVVMPAGES="unlimited:unlimited"
NUMFILE="unlimited:unlimited"
NUMFLOCK="unlimited:unlimited"
NUMPTY="unlimited:unlimited"
NUMSIGINFO="unlimited:unlimited"
DCACHESIZE="unlimited:unlimited"
PHYSPAGES="unlimited:unlimited"
NUMIPTENT="unlimited:unlimited"
Junk Email Filter
http://www.junkemailfilter.com
|
|
|
|
|
Re: Weird Disk Spac Problem [message #32845 is a reply to message #32795] |
Tue, 02 September 2008 15:54   |
mperkel
Messages: 253 Registered: December 2006
|
Senior Member |
|
|
I'll look into that as it runs. Seeing a few of these:
httpd 9712 apache 710w REG 0,20 0 13369352 (deleted) /var/log/httpd/ssl_mutex
httpd 9713 apache 710w REG 0,20 0 13369352 (deleted) /var/log/httpd/ssl_mutex
Logrotate runs once a week. But HTTPD is restarted lately fairly often. But the server was locked up this morning with the same disk space complaint.
Here's what the logs were showing today.
[Tue Sep 02 05:08:02 2008] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed
Junk Email Filter
http://www.junkemailfilter.com
|
|
|
|
|
Re: Weird Disk Spac Problem [message #32863 is a reply to message #32795] |
Thu, 04 September 2008 14:51  |
kenjy
Messages: 50 Registered: July 2008 Location: Mexico
|
Member |
|
|
You need to correctly log rotate your ssl_mutex log file, well I think that this is the problem:
httpd 9712 apache 710w REG 0,20 0 13369352 (deleted) /var/log/httpd/ssl_mutex
httpd 9713 apache 710w REG 0,20 0 13369352 (deleted) /var/log/httpd/ssl_mutex
You have a deleted file who is growing in some way. I fix it adding this to log rotate but this is in nginx web server with this script, obviously it works rotating logs each night because we want a log each night (my logs are about 2gb so I need to make a slide at the middle of the day):
www:/opt/mantenimiento# cat nginx_logs.sh
#Obtenemos la hora actual, dependiendo de si son las 00 o las 12 se rotara el log
hora=$(date +%H)
ayer=$(date -d yesterday +%Y-%m-%d)
hoy=$(date +%Y-%m-%d)
ruta=/var/log/nginx
if [ "$hora" = "00" ]; then
mv $ruta/access.log $ruta/jornada_access_$ayer-12.log
if [ -f /var/run/nginx.pid ]; then
kill -USR1 `cat /var/run/nginx.pid`
fi
cat $ruta/jornada_access_$ayer-12.log >> /$ruta/jornada_access_$ayer-00.log
mv $ruta/jornada_access_$ayer-00.log $ruta/jornada_access_$ayer.log
rm $ruta/jornada_access_$ayer-12.log
gzip -9 $ruta/jornada_access_$ayer.log
elif [ "$hora" = "12" ]; then
mv $ruta/access.log $ruta/jornada_access_$hoy-00.log
if [ -f /var/run/nginx.pid ]; then
kill -USR1 `cat /var/run/nginx.pid`
fi
else
echo "No se procesa, fuera de hora."
fi
www:/opt/mantenimiento#
http://kenjy.net |Live net !!!
|
|
|