OpenVZ Forum - RDF feed
type: channel
items: Diskspace filled with nothing in a VE
items: Re: Diskspace filled with nothing in a VE

Diskspace filled with nothing in a VE
type: item
description: I have a VE with debian and 6gb of disk space allocated, it runs a web server who dispatches static content (pictures only), the VE took 700mb more or less, but when it start to work the next day took 3 or 4 GB, after restart it back to 700mb, my HN has 182gb of free hard disk space. This only happens on the VE where I use second level quotas. This my HN df -h output: [I][I][email]root@gribus[/email]:~# df -h S.ficheros Tamaño Usado Disp Uso% Montado en /dev/sda1 14G 1.8G 12G 14% / tmpfs 2.0G 0 2.0G 0% /lib/init/rw udev 10M 64K 10M 1% /dev tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/md0 209G 16G 182G 9% /vz [email]root@gribus[/email]:~# This is my VE df -h output after 24 hours of work: [email]root@anuncios[/email]:/# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 5.8G 2.9G 2.9G 51% / tmpfs 2.0G 0 2.0G 0% /lib/init/rw tmpfs 2.0G 0 2.0G 0% /dev/shm [email]root@anuncios[/email]:/# This is my du -sh output in / [email]root@anuncios[/email]:/# du -sh 841M . this is my VE / fyle system output: [email]root@anuncios[/email]:/# du -sh * 0 aquota.group 0 aquota.user 3.6M bin 4.0K boot 4.0K dev 1.9M etc 6.2M home 4.0K initrd 7.6M lib 0 lib64 4.0K media 4.0K mnt 32K opt 259M proc 14M root 3.7M sbin 4.0K srv 4.0K sys 8.0K tmp 195M usr 351M var [email]root@anuncios[/email]:/# Now Im going to reboot the VE: [email]root@gribus[/email]:~# vzctl restart 104 Restarting VPS Stopping VPS ... VPS was stopped VPS is not mounted Starting VPS ... vzquota : (warning) Quota is running for id 104 already VPS is mounted Deleting IP address(es): 132.248.10.53 ifdown: interface venet0:0 not configured Adding IP address(es): 132.248.10.53 Setting CPU limit: 10 Setting CPU units: 1000 Set hostname: anuncios.jornada.com.mx File resolv.conf was modified Setting quota ugidlimit: 100 VPS start in progress... [email]root@gribus[/email]:~# vzctl enter 104 entered into VPS 104 [email]root@anuncios[/email]:/# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 5.8G 584M 5.2G 10% / tmpfs 2.0G 0…
creator: gaper
date: 2007-10-24T15:47:36-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi, 1. As I can see from your post, command df has shown various consequence from command du. It can be happened when any process used some files, which you had removed from their directory; but they didn't actually delete from the file system, because somebody used it. May be command "lsof | grep deleted" will help you to find out such files. 2. If it is possible, can you conduct this experiment, when the second level quota is off, because it's hardly that quota is the reason of a problem. Thank You!
creator: maratrus
date: 2007-10-26T08:18:29-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi maratrus, tnks for your replay, this is my lsof output, I will tnks your help because I m not shure whats happening :S [email]root@anuncios[/email]:/# lsof |grep deleted [email]root@anuncios[/email]:/#
creator: gaper
date: 2007-10-26T18:55:29-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi, 1. Quote: root@anuncios:/# lsof |grep deleted root@anuncios:/# And what show du and df at that moment? I want to see "lsof | grep deleted" (and also "lsof") when df shows a lot of disk usage. 2. The difference between du and df may consists also of file system metadata. 3. Please pay attention at warning from vzquota during restart VPS. What version of OpenVZ do you use (uname -a) and what version of vzctl do you use? Can you give me this information? 4. By the way, what is in /proc/user_beancounters Thank You!
creator: maratrus
date: 2007-10-29T10:54:39-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi maratrus and good morning xd this is my df output (today 29/10/07 at 10:33am after 3 days of work without reboot the VE): root@gribus:~# vzctl enter 104 entered into VPS 104 root@anuncios:/# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 5.8G 4.7G 1.1G 82% / tmpfs 2.0G 0 2.0G 0% /lib/init/rw tmpfs 2.0G 0 2.0G 0% /dev/shm root@anuncios:/# This is my du output: root@anuncios:/# du -sh * 0 aquota.group 0 aquota.user 3.6M bin 4.0K boot 4.0K dev 1.9M etc 6.3M home 4.0K initrd 7.6M lib 0 lib64 4.0K media 4.0K mnt 32K opt 259M proc 14M root 3.7M sbin 4.0K srv 4.0K sys 8.0K tmp 196M usr 260M var root@anuncios:/# du -sh 750M . root@anuncios:/# This is my output of user_beancounter root@gribus:~# cat /proc/user_beancounters Version: 2.5 uid resource held maxheld barrier limit failcnt 102: kmemsize 3251922 4769063 14790793 15569256 0 lockedpages 0 0 512 512 0 privvmpages 5308 29209 59441 59441 0 shmpages 1 9506 25600 25600 0 dummy 0 0 0 0 0 numproc 19 34 320 320 0 physpages 3115 13483 0 2147483647 0 vmguarpages 0 0 59441 2147483647 0 oomguarpages 3115 13483 52428 2147483647 0 numtcpsock 8 31 512 512 0 numflock 1 7 512 512 0 numpty 0 5 32 32 0 numsiginfo 0 10 512 512 0 tcpsndbuf 154712 1147472 1342177 2684354 0 tcprcvbuf 126712 731424 1342177 2684354 0 othersockbuf 4624 322328 1342177 2684354 0 dgramrcvbuf 0 8456 671088 1342177 0 numothersock 4 20 512 512 0 dcachesize 0 0 4026531 4026531 0 numfile 713 1492 8192 8192 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 numiptent 91 91 768 768 0 101: kmemsize 7149044 9943432 118326348 124554051 0 lockedpages 0 0 4096 4096 0 privvmpages 355026 432736 475529 475529 0 shmpages 0 0 204800 204800 0 dummy 0 0 0 0 0 numproc 36 51 2560 2560 0 physpages 314862 396008 0 9223372036854775807 0 vmguarpages 0 0 475529 9223372036854775807 0 oomguarpages 314862 396008 419430 9223372036854775807 0 numtcpsock 16 306 4096 4096 0 numflock 5 9 4096 4096 0 numpty 0 3 256 256 0 numsiginfo 0 34 4096 4096 0 tcpsndbuf 1280848 2555024 10737418…
creator: gaper
date: 2007-10-29T16:50:26-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi, 1. You have forgotten to show the output of "lsof | grep delete" at THAT moment (after 3 days of work without reboot). Please attach the output of "lsof" command at THAT time too. 2. Also can you show the output of "uname -a" and version vzctl you use? 3. Try to do following: a)stop VE b)vzquota drop <veid> c) vzctl start <veid> Thank You!
creator: maratrus
date: 2007-10-31T09:21:52-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi maratrus, this is my lsof output (extrange but real :S) root@anuncios:/# lsof |grep delete root@anuncios:/# If I just do lsof it show a lot of files but If I do a grep delete it don't show nothing. this is my uname -a output: root@anuncios:/# uname -a Linux anuncios.jornada.com.mx 2.6.18+openvz #1 SMP Thu Aug 9 14:38:45 CDT 2007 x86_64 GNU/Linux root@anuncios:/# My version of vzctl its 3.0.11-13 server virtualization solution - control tools I tried the vzquota drop 104 but it do this: root@gribus:~# vzctl stop 104 Stopping VPS ... VPS was stopped VPS is not mounted root@gribus:~# vzquota drop 104 vzquota : (warning) Quota is running for id 104 root@gribus:~# vzctl start 104 Starting VPS ... vzquota : (warning) Quota is running for id 104 already VPS is mounted Adding IP address(es): 132.248.10.53 Setting CPU limit: 10 Setting CPU units: 1000 Set hostname: anuncios.jornada.com.mx File resolv.conf was modified Setting quota ugidlimit: 100 VPS start in progress... root@gribus:~# any way if you want it I wait 3 days to do the lsof and paste it here. Really man, tnks for your time and wish of help to others.
creator: gaper
date: 2007-10-31T17:09:48-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi, Sorry for delay. It seems that some file from ve private area is in use. May be some process uses the file or somebody has current dir /vz/private/<veid> or /vz/root/<veid>. Check that during the VE stoping nobody uses any file from ve private area. I hope "lsof" (on hardware node) helps us (may be somebody has a ssh session with current dir /vz/private/<veid>). Please show the output: a) vzctl stop 104 b) vzquota off 104 c) vzquota drop 104 d) vzctl start 104 Thank You!
creator: maratrus
date: 2007-11-02T14:05:39-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: If it is possible can you please attach "lsof" from ve (when du and df show different result) and lsof from hardware node just before you try to stop vps. Thank You!
creator: maratrus
date: 2007-11-02T14:25:20-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi maratrus, and sorry for the delay too, this is my df and du taked from the VE, they are diferents right now: droot@anuncios:/# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 5.8G 1.2G 4.7G 20% / tmpfs 2.0G 0 2.0G 0% /lib/init/rw tmpfs 2.0G 0 2.0G 0% /dev/shm root@anuncios:/# du -sh 724M . root@anuncios:/# du -sh * 0 aquota.group 0 aquota.user 3.6M bin 4.0K boot 4.0K dev 1.9M etc 6.6M home 4.0K initrd 7.6M lib 0 lib64 4.0K media 4.0K mnt 32K opt 259M proc 14M root 3.7M sbin 4.0K srv 4.0K sys 8.0K tmp 196M usr 234M var root@anuncios:/# The lsof output at this time in the VE its: COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME init 1 root cwd DIR 0,30 4096 10291083 / init 1 root rtd DIR 0,30 4096 10291083 / init 1 root txt REG 0,30 37720 10485851 /sbin/init init 1 root mem REG 9,0 10485851 /sbin/init (path dev=0,30) init 1 root mem REG 0,0 0 [heap] (stat: No such file or directory) init 1 root mem REG 9,0 10457850 /lib/ld-2.3.6.so (path dev=0,30) init 1 root mem REG 9,0 10456554 /lib/libsepol.so.1 (path dev=0,30) init 1 root mem REG 9,0 10456658 /lib/libselinux.so.1 (path dev=0,30) init 1 root mem REG 9,0 10457854 /lib/libc-2.3.6.so (path dev=0,30) init 1 root mem REG 9,0 10457856 /lib/libdl-2.3.6.so (path dev=0,30) init 1 root 10u FIFO 0,30 10456901 /dev/initctl syslogd 15742 root cwd DIR 0,30 4096 10291083 / syslogd 15742 root rtd DIR 0,30 4096 10291083 / syslogd 15742 root txt REG 0,30 33368 10485814 /sbin/syslogd syslogd 15742 root mem REG 9,0 10485814 /sbin/syslogd (path dev=0,30) syslogd 15742 root mem REG 0,0 0 [heap] (stat: No such file or directory) syslogd 15742 root mem REG 9,0 10457850 /lib/ld-2.3.6.so (path dev=0,30) syslogd 15742 root mem REG 9,0 10457854 /lib/libc-2.3.6.so (path dev=0,30) syslogd 15742 root mem REG 9,0 10457839 /lib/libnss_files-2.3.6.so (path dev=0,30) syslogd 15742 root 0u unix 0xffff81002873e6c0 93623677 /dev/log syslogd 15742 root 2w REG 0,30 1579 10485791 /var/log/auth.log syslogd 15742 root 3w REG 0,30 247188 1048589…
creator: gaper
date: 2007-11-06T15:57:29-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi, 1. I suppose that in this case deleted files are the solution of the problem. 2. That files are used by nginx that is why they actually are not removed from file system (df takes into account their) but they are removed from their directories and du doesn't take into account that files. Thank You!
creator: maratrus
date: 2007-11-07T07:36:39-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi maratrus nice week , I did a test and I just restarted nginx, what was the surprise? The ghost space was recovered: root@anuncios:/# /etc/init.d/nginx stop Stopping nginx: nginx. root@anuncios:/# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 5.8G 1.8G 4.0G 31% / tmpfs 2.0G 0 2.0G 0% /lib/init/rw tmpfs 2.0G 0 2.0G 0% /dev/shm root@anuncios:/# /etc/init.d/nginx start Starting nginx: nginx. root@anuncios:/# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 5.8G 480M 5.3G 9% / tmpfs 2.0G 0 2.0G 0% /lib/init/rw tmpfs 2.0G 0 2.0G 0% /dev/shm root@anuncios:/# How can I fix this without restarting the VE or nginx each night?
creator: gaper
date: 2007-11-07T19:01:41-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi, 1. I'm afraid that it is not OpenVz specific problem. 2. Is it possible for you to read more about nginx? May be we can manage the log files to prevent such behavior? It would be fine! Thank You!
creator: maratrus
date: 2007-11-09T08:21:47-00:00

Re: Diskspace filled with nothing in a VE
type: item
description: Hi maratrus, you are right, the problem was in nginx, when you use log-rotate, the solution where adding this options to the /etc/logrotate.d/nginx file that I create to rotate the nginx logs: postrotate if [ -f /var/run/nginx.pid ]; then kill -USR1 `cat /var/run/nginx.pid` fi It kill one instance of nginx and the zombies files. Nginx open another instance when it got another request. Maratrus, really tnks for your time, the problem its solved =)
creator: gaper
date: 2007-11-09T15:55:57-00:00