occasional high loadavg without any noticeable cpu/memory/io load
Hello, I occasionally get this extreme load on one of our VPS servers. It is quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + parked/addon/subdomains. The hardware node has 12 active VPS servers and most of the time things are chugging along just fine, something like this. 1401: 0.00 0.00 0.00 1/23 4561 1402: 0.02 0.05 0.05 1/57 16991 1404: 0.01 0.02 0.00 1/73 18863 1406: 0.07 0.13 0.06 1/39 31189 1407: 0.86 1.03 1.14 1/113 31460 1408: 0.17 0.17 0.18 1/79 32579 1409: 0.00 0.00 0.02 1/77 21784 1410: 0.01 0.02 0.00 1/60 7454 1413: 0.00 0.00 0.00 1/46 18579 1414: 0.00 0.00 0.00 1/41 23812 1415: 0.00 0.00 0.00 1/45 9831 1416: 0.05 0.02 0.00 1/59 11332 12 active The problem VPS is 1407. As you can see below it only uses a bit of the cpu and memory. top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, 1.09 Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st Mem: 4194304k total, 2550572k used, 1643732k free, 0k buffers Swap: 8388608k total, 105344k used, 8283264k free, 1793828k cached Also iostat and vmstat shows no particular io or swap activity. Now for the problem. Every once in a while the loadavg of this particular VPS shoots up to like crazy values, 30 or more and it becomes completely sluggish. The odd thing is load goes up for the VPS server, and starts spilling into other VPS serers on the same hardware node - but there are still no particular cpu/memory/io usage going on that I can se. No particular network activity. In this example load has fallen back to around 10 but it was much higher earlier. 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, 19.11, 18.87 1401: 0.01 0.03 0.00 1/23 2876 1402: 0.00 0.11 0.13 1/57 15334 1404: 0.02 0.20 0.16 1/77 14918 1406: 0.01 0.13 0.10 1/39 29595 1407: 10.95 15.71 15.05 1/128 13950 1408: 0.36 0.52 0.57 1/81 27167 1409: 0.09 0.26 0.43 1/78 17851 1410: 0.09 0.17 0.18 1/61 4344 1413: 0.00 0.03 0.00 1/46 16539 1414: 0.01 0.01 0.00 1/41 22372 1415: 0.00 0.01 0.00 1/45 8404 1416: 0.05 0.10 0.11 1/58 9292 12 active top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, 14.97, 14.82 Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st Mem: 4194304k total, 1173844k used, 3020460k free, 0k buffers Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache Notice how cpu is plenty idle, and only 1/4 of the available memory is being used. http://wiki.openvz.org/Ploop/Why explains "One such property that deserves a special item in this list is file system journal. While journal is a good thing to have, because it helps to maintain file system integrity and improve reboot times (by eliminating fsck in many cases), it is also a bottleneck for containers. If one container will fill up in-memory journal (with lots of small operations leading to file metadata updates, e.g. file truncates), all the other containers I/O will block waiting for the journal to be written to disk. In some extreme cases we saw up to 15 seconds of such blockage.". The problem I noticed last much longer than 15 seconds though - typically 15-30 minutes, then load goes back where it should be. Any suggestions where I could look for the cause of this? It's not like it happens everyday, maybe once or twice per month, but it's enough to cause customers to complain. Regards, Rene
Re: occasional high loadavg without any noticeable cpu/memory/io load
Hi Rene, Since CPU and MEM are fine it's most likely to be Disk-IO. I have similar Problems with a Cluster Setup based on OpenVZ. The problem is that our Storage is way to slow. We have been accessing the storage via NFS and put all our CTs private areas on it. I noticed many times that one CT was doing a lot of disk IO and all other were suffering from that... that even lead to total system failures. This has been solved by converting everything to ploop. Since then our system is at least in a stable state. IO Performance is still an issue but does not bring our system down. You should give ploop a try :-) I am very happy with it. best regards, Sirk 2012/5/21 Rene Dokbua <openvz@dokbua.com>: > Hello, > > I occasionally get this extreme load on one of our VPS servers. It is quite > large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + > parked/addon/subdomains. > > The hardware node has 12 active VPS servers and most of the time things are > chugging along just fine, something like this. > > 1401: 0.00 0.00 0.00 1/23 4561 > 1402: 0.02 0.05 0.05 1/57 16991 > 1404: 0.01 0.02 0.00 1/73 18863 > 1406: 0.07 0.13 0.06 1/39 31189 > 1407: 0.86 1.03 1.14 1/113 31460 > 1408: 0.17 0.17 0.18 1/79 32579 > 1409: 0.00 0.00 0.02 1/77 21784 > 1410: 0.01 0.02 0.00 1/60 7454 > 1413: 0.00 0.00 0.00 1/46 18579 > 1414: 0.00 0.00 0.00 1/41 23812 > 1415: 0.00 0.00 0.00 1/45 9831 > 1416: 0.05 0.02 0.00 1/59 11332 > 12 active > > The problem VPS is 1407. As you can see below it only uses a bit of the cpu > and memory. > > top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, 1.09 > Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, > 0.1%st > Mem: 4194304k total, 2550572k used, 1643732k free, 0k buffers > Swap: 8388608k total, 105344k used, 8283264k free, 1793828k cached > > Also iostat and vmstat shows no particular io or swap activity. > > Now for the problem. Every once in a while the loadavg of this particular > VPS shoots up to like crazy values, 30 or more and it becomes completely > sluggish. The odd thing is load goes up for the VPS server, and starts > spilling into other VPS serers on the same hardware node - but there are > still no particular cpu/memory/io usage going on that I can se. No > particular network activity. In this example load has fallen back to > around 10 but it was much higher earlier. > > 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, 19.11, 18.87 > > 1401: 0.01 0.03 0.00 1/23 2876 > 1402: 0.00 0.11 0.13 1/57 15334 > 1404: 0.02 0.20 0.16 1/77 14918 > 1406: 0.01 0.13 0.10 1/39 29595 > 1407: 10.95 15.71 15.05 1/128 13950 > 1408: 0.36 0.52 0.57 1/81 27167 > 1409: 0.09 0.26 0.43 1/78 17851 > 1410: 0.09 0.17 0.18 1/61 4344 > 1413: 0.00 0.03 0.00 1/46 16539 > 1414: 0.01 0.01 0.00 1/41 22372 > 1415: 0.00 0.01 0.00 1/45 8404 > 1416: 0.05 0.10 0.11 1/58 9292 > 12 active > > top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, 14.97, > 14.82 > Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, > 0.1%st > Mem: 4194304k total, 1173844k used, 3020460k free, 0k buffers > Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache > > Notice how cpu is plenty idle, and only 1/4 of the available memory is being > used. > > http://wiki.openvz.org/Ploop/Why explains "One such property that deserves a > special item in this list is file system journal. While journal is a good > thing to have, because it helps to maintain file system integrity and > improve reboot times (by eliminating fsck in many cases), it is also a > bottleneck for containers. If one container will fill up in-memory journal > (with lots of small operations leading to file metadata updates, e.g. file > truncates), all the other containers I/O will block waiting for the journal > to be written to disk. In some extreme cases we saw up to 15 seconds of such > blockage.". The problem I noticed last much longer than 15 seconds though > - typically 15-30 minutes, then load goes back where it should be. > > Any suggestions where I could look for the cause of this? It's not like it > happens everyday, maybe once or twice per month, but it's enough to cause > customers to complain. > > Regards, > Rene > > -- Satzmedia GmbH Altonaer Poststraße 9 22767 Hamburg Tel: +49 (0) 40 - 1 888 969 - 140 Fax: +49 (0) 40 - 1 888 969 - 200 E-Mail: s.johannsen@satzmedia.de E-Business-Lösungen: http://www.satzmedia.de Amtsgericht Hamburg, HRB 71729 Ust-IDNr. DE201979921 Geschäftsführer: Dipl.-Kfm. Christian Satz Dipl.-Inform. Markus Meyer-Westphal --
RE: occasional high loadavg without any noticeable cpu/memory/io load
Sorry dont have the answer for you But can you tell me what command you used to see all loads on your node ? Thanxs Steffan Van: users-bounces@openvz.org [mailto:users-bounces@openvz.org] Namens Rene Dokbua Verzonden: maandag 21 mei 2012 20:07 Aan: users@openvz.org Onderwerp: [Users] occasional high loadavg without any noticeable cpu/memory/io load Hello, I occasionally get this extreme load on one of our VPS servers. It is quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + parked/addon/subdomains. The hardware node has 12 active VPS servers and most of the time things are chugging along just fine, something like this. 1401: 0.00 0.00 0.00 1/23 4561 1402: 0.02 0.05 0.05 1/57 16991 1404: 0.01 0.02 0.00 1/73 18863 1406: 0.07 0.13 0.06 1/39 31189 1407: 0.86 1.03 1.14 1/113 31460 1408: 0.17 0.17 0.18 1/79 32579 1409: 0.00 0.00 0.02 1/77 21784 1410: 0.01 0.02 0.00 1/60 7454 1413: 0.00 0.00 0.00 1/46 18579 1414: 0.00 0.00 0.00 1/41 23812 1415: 0.00 0.00 0.00 1/45 9831 1416: 0.05 0.02 0.00 1/59 11332 12 active The problem VPS is 1407. As you can see below it only uses a bit of the cpu and memory. top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, 1.09 Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st Mem: 4194304k total, 2550572k used, 1643732k free, 0k buffers Swap: 8388608k total, 105344k used, 8283264k free, 1793828k cached Also iostat and vmstat shows no particular io or swap activity. Now for the problem. Every once in a while the loadavg of this particular VPS shoots up to like crazy values, 30 or more and it becomes completely sluggish. The odd thing is load goes up for the VPS server, and starts spilling into other VPS serers on the same hardware node - but there are still no particular cpu/memory/io usage going on that I can se. No particular network activity. In this example load has fallen back to around 10 but it was much higher earlier. 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, 19.11, 18.87 1401: 0.01 0.03 0.00 1/23 2876 1402: 0.00 0.11 0.13 1/57 15334 1404: 0.02 0.20 0.16 1/77 14918 1406: 0.01 0.13 0.10 1/39 29595 1407: 10.95 15.71 15.05 1/128 13950 1408: 0.36 0.52 0.57 1/81 27167 1409: 0.09 0.26 0.43 1/78 17851 1410: 0.09 0.17 0.18 1/61 4344 1413: 0.00 0.03 0.00 1/46 16539 1414: 0.01 0.01 0.00 1/41 22372 1415: 0.00 0.01 0.00 1/45 8404 1416: 0.05 0.10 0.11 1/58 9292 12 active top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, 14.97, 14.82 Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st Mem: 4194304k total, 1173844k used, 3020460k free, 0k buffers Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache Notice how cpu is plenty idle, and only 1/4 of the available memory is being used. http://wiki.openvz.org/Ploop/Why explains "One such property that deserves a special item in this list is file system journal. While journal is a good thing to have, because it helps to maintain file system integrity and improve reboot times (by eliminating fsck in many cases), it is also a bottleneck for containers. If one container will fill up in-memory journal (with lots of small operations leading to file metadata updates, e.g. file truncates), all the other containers I/O will block waiting for the journal to be written to disk. In some extreme cases we saw up to 15 seconds of such blockage.". The problem I noticed last much longer than 15 seconds though - typically 15-30 minutes, then load goes back where it should be. Any suggestions where I could look for the cause of this? It's not like it happens everyday, maybe once or twice per month, but it's enough to cause customers to complain. Regards, Rene
Actually I made a small shell script that loops through the list of active containers and outputs the content of each containers /proc/loadavg. It started out as a bit more elaborate script that was intended to provide some of the functionality of a script vzstat, that I used to use with Virtuozzo. You can download both scripts from https://www.ourhelpdesk.net/downloads/z.tgz On Tue, May 22, 2012 at 3:15 PM, Steffan <general@ziggo.nl> wrote: > Sorry dont have the answer for you**** > > But can you tell me what command you used to see all loads on your node ?* > *** > > ** ** > > Thanxs Steffan**** > > ** ** > > *Van:* users-bounces@openvz.org [mailto:users-bounces@openvz.org] *Namens > *Rene Dokbua > *Verzonden:* maandag 21 mei 2012 20:07 > *Aan:* users@openvz.org > *Onderwerp:* [Users] occasional high loadavg without any noticeable > cpu/memory/io load**** > > ** ** > > Hello,**** > > ** ** > > I occasionally get this extreme load on one of our VPS servers. It is > quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + > parked/addon/subdomains.**** > > ** ** > > The hardware node has 12 active VPS servers and most of the time things > are chugging along just fine, something like this.**** > > ** ** > > 1401: 0.00 0.00 0.00 1/23 4561**** > > 1402: 0.02 0.05 0.05 1/57 16991**** > > 1404: 0.01 0.02 0.00 1/73 18863**** > > 1406: 0.07 0.13 0.06 1/39 31189**** > > 1407: 0.86 1.03 1.14 1/113 31460**** > > 1408: 0.17 0.17 0.18 1/79 32579**** > > 1409: 0.00 0.00 0.02 1/77 21784**** > > 1410: 0.01 0.02 0.00 1/60 7454**** > > 1413: 0.00 0.00 0.00 1/46 18579**** > > 1414: 0.00 0.00 0.00 1/41 23812**** > > 1415: 0.00 0.00 0.00 1/45 9831**** > > 1416: 0.05 0.02 0.00 1/59 11332**** > > 12 active**** > > ** ** > > The problem VPS is 1407. As you can see below it only uses a bit of the > cpu and memory. **** > > ** ** > > top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, 1.09 > **** > > Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie**** > > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, > 0.1%st**** > > Mem: 4194304k total, 2550572k used, 1643732k free, 0k buffers** > ** > > Swap: 8388608k total, 105344k used, 8283264k free, 1793828k cached*** > * > > ** ** > > Also iostat and vmstat shows no particular io or swap activity.**** > > ** ** > > Now for the problem. Every once in a while the loadavg of this particular > VPS shoots up to like crazy values, 30 or more and it becomes completely > sluggish. The odd thing is load goes up for the VPS server, and starts > spilling into other VPS serers on the same hardware node - but there are > still no particular cpu/memory/io usage going on that I can se. No > particular network activity. In this example load has fallen back to > around 10 but it was much higher earlier.**** > > ** ** > > 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, 19.11, 18.87* > *** > > ** ** > > 1401: 0.01 0.03 0.00 1/23 2876**** > > 1402: 0.00 0.11 0.13 1/57 15334**** > > 1404: 0.02 0.20 0.16 1/77 14918**** > > 1406: 0.01 0.13 0.10 1/39 29595**** > > 1407: 10.95 15.71 15.05 1/128 13950**** > > 1408: 0.36 0.52 0.57 1/81 27167**** > > 1409: 0.09 0.26 0.43 1/78 17851**** > > 1410: 0.09 0.17 0.18 1/61 4344**** > > 1413: 0.00 0.03 0.00 1/46 16539**** > > 1414: 0.01 0.01 0.00 1/41 22372**** > > 1415: 0.00 0.01 0.00 1/45 8404**** > > 1416: 0.05 0.10 0.11 1/58 9292**** > > 12 active**** > > ** ** > > top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, 14.97, > 14.82**** > > Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie**** > > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, > 0.1%st**** > > Mem: 4194304k total, 1173844k used, 3020460k free, 0k buffers** > ** > > Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache**** > > ** ** > > Notice how cpu is plenty idle, and only 1/4 of the available memory is > being used.**** > > ** ** > > http://wiki.openvz.org/Ploop/Why explains "One such property that > deserves a special item in this list is file system journal. While journal > is a good thing to have, because it helps to maintain file system integrity > and improve reboot times (by eliminating fsck in many cases), it is also a > bottleneck for containers. If one container will fill up in-memory journal > (with lots of small operations leading to file metadata updates, e.g. file > truncates), all the other containers I/O will block waiting for the journal > to be written to disk. In some extreme cases we saw up to 15 seconds of > such blockage.". The problem I noticed last much longer than 15 seconds > though - typically 15-30 minutes, then load goes back where it should be.* > *** > > ** ** > > Any suggestions where I could look for the cause of this? It's not like > it happens everyday, maybe once or twice per month, but it's enough to > cause customers to complain.**** > > ** ** > > Regards, > Rene**** > > ** ** >
Hi Sirk, Thanks for your reply. I'm so pleased having found this mailing list after having tried the forum, which seem to have very little activity! Ploop is a great idea technically, but I'm a little concerned about the " Warning: This is a new feature, not yet ready for production systems. Use with caution." on the OpenVZ Wiki page, so I'm kinda waiting for the green-light that it's ready for production environments. It did occur to me that disk-IO could be the cause of the problem, but iostat on the hardware node did not suggest any particular IO problems. I still haven't found a way to see the IO activity within a container - iostat just comes up blank when it's run within a container. Is there a way? We're not using any network storage with this server so that is not the reason. The server has 4 SATA-3 drives, with the root partition being on one drive, the problem container alone on a second drive, and the remaining containers on a third. Best, Rene On Tue, May 22, 2012 at 3:06 PM, Sirk Johannsen <s.johannsen@satzmedia.de>wrote: > Hi Rene, > > Since CPU and MEM are fine it's most likely to be Disk-IO. > I have similar Problems with a Cluster Setup based on OpenVZ. > The problem is that our Storage is way to slow. > We have been accessing the storage via NFS and put all our CTs private > areas on it. > I noticed many times that one CT was doing a lot of disk IO and all > other were suffering from that... that even lead to total system > failures. > This has been solved by converting everything to ploop. Since then our > system is at least in a stable state. > IO Performance is still an issue but does not bring our system down. > > You should give ploop a try :-) I am very happy with it. > > best regards, > > Sirk > > 2012/5/21 Rene Dokbua <openvz@dokbua.com>: > > Hello, > > > > I occasionally get this extreme load on one of our VPS servers. It is > quite > > large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + > > parked/addon/subdomains. > > > > The hardware node has 12 active VPS servers and most of the time things > are > > chugging along just fine, something like this. > > > > 1401: 0.00 0.00 0.00 1/23 4561 > > 1402: 0.02 0.05 0.05 1/57 16991 > > 1404: 0.01 0.02 0.00 1/73 18863 > > 1406: 0.07 0.13 0.06 1/39 31189 > > 1407: 0.86 1.03 1.14 1/113 31460 > > 1408: 0.17 0.17 0.18 1/79 32579 > > 1409: 0.00 0.00 0.02 1/77 21784 > > 1410: 0.01 0.02 0.00 1/60 7454 > > 1413: 0.00 0.00 0.00 1/46 18579 > > 1414: 0.00 0.00 0.00 1/41 23812 > > 1415: 0.00 0.00 0.00 1/45 9831 > > 1416: 0.05 0.02 0.00 1/59 11332 > > 12 active > > > > The problem VPS is 1407. As you can see below it only uses a bit of the > cpu > > and memory. > > > > top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, > 1.09 > > Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie > > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, > > 0.1%st > > Mem: 4194304k total, 2550572k used, 1643732k free, 0k buffers > > Swap: 8388608k total, 105344k used, 8283264k free, 1793828k cached > > > > Also iostat and vmstat shows no particular io or swap activity. > > > > Now for the problem. Every once in a while the loadavg of this particular > > VPS shoots up to like crazy values, 30 or more and it becomes completely > > sluggish. The odd thing is load goes up for the VPS server, and starts > > spilling into other VPS serers on the same hardware node - but there are > > still no particular cpu/memory/io usage going on that I can se. No > > particular network activity. In this example load has fallen back to > > around 10 but it was much higher earlier. > > > > 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, 19.11, 18.87 > > > > 1401: 0.01 0.03 0.00 1/23 2876 > > 1402: 0.00 0.11 0.13 1/57 15334 > > 1404: 0.02 0.20 0.16 1/77 14918 > > 1406: 0.01 0.13 0.10 1/39 29595 > > 1407: 10.95 15.71 15.05 1/128 13950 > > 1408: 0.36 0.52 0.57 1/81 27167 > > 1409: 0.09 0.26 0.43 1/78 17851 > > 1410: 0.09 0.17 0.18 1/61 4344 > > 1413: 0.00 0.03 0.00 1/46 16539 > > 1414: 0.01 0.01 0.00 1/41 22372 > > 1415: 0.00 0.01 0.00 1/45 8404 > > 1416: 0.05 0.10 0.11 1/58 9292 > > 12 active > > > > top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, 14.97, > > 14.82 > > Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie > > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, > > 0.1%st > > Mem: 4194304k total, 1173844k used, 3020460k free, 0k buffers > > Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache > > > > Notice how cpu is plenty idle, and only 1/4 of the available memory is > being > > used. > > > > http://wiki.openvz.org/Ploop/Why explains "One such property that > deserves a > > special item in this list is file system journal. While journal is a good > > thing to have, because it helps to maintain file system integrity and > > improve reboot times (by eliminating fsck in many cases), it is also a > > bottleneck for containers. If one container will fill up in-memory > journal > > (with lots of small operations leading to file metadata updates, e.g. > file > > truncates), all the other containers I/O will block waiting for the > journal > > to be written to disk. In some extreme cases we saw up to 15 seconds of > such > > blockage.". The problem I noticed last much longer than 15 seconds > though > > - typically 15-30 minutes, then load goes back where it should be. > > > > Any suggestions where I could look for the cause of this? It's not like > it > > happens everyday, maybe once or twice per month, but it's enough to cause > > customers to complain. > > > > Regards, > > Rene > > > > > -- > Satzmedia GmbH > > Altonaer Poststraße 9 > 22767 Hamburg > Tel: +49 (0) 40 - 1 888 969 - 140 > Fax: +49 (0) 40 - 1 888 969 - 200 > E-Mail: s.johannsen@satzmedia.de > E-Business-Lösungen: http://www.satzmedia.de > Amtsgericht Hamburg, HRB 71729 > Ust-IDNr. DE201979921 > Geschäftsführer: > Dipl.-Kfm. Christian Satz > Dipl.-Inform. Markus Meyer-Westphal > > -- > >
2012/5/22 Rene C. <openvz@dokbua.com>: > Hi Sirk, > Hi Rene, > Thanks for your reply. I'm so pleased having found this mailing list after > having tried the forum, which seem to have very little activity! > True, but this list has helped me a lot as well :-) > Ploop is a great idea technically, but I'm a little concerned about the " > Warning: This is a new feature, not yet ready for production systems. Use > with caution." on the OpenVZ Wiki page, so I'm kinda waiting for the > green-light that it's ready for production environments. > If you want some practical information on ploop: We are using it in a highly productive environment. It was either, try ploop and hope it works, or have the systems fail every 2nd day. So we decided to use ploop and are more than happy. It even solves a lot of issues we had with the private areas directly on the nfs share. But of course, thats totally up to you. I started with only a few "unimportant" CTs and then merged everything after a while (42 CTs). > It did occur to me that disk-IO could be the cause of the problem, but > iostat on the hardware node did not suggest any particular IO problems. I > still haven't found a way to see the IO activity within a container - iostat > just comes up blank when it's run within a container. Is there a way? > To be honest, I don't know. iostat ist not working because you do not really have a device. This ist handled the way with ploop sadly but could be modified I guess. For ploop you have the ploop-stat command but that dosen't work as expected for me :-) > We're not using any network storage with this server so that is not the > reason. > > The server has 4 SATA-3 drives, with the root partition being on one drive, > the problem container alone on a second drive, and the remaining containers > on a third. So you have a different FileSystem for the "problem"-Container that is even on a different disk ? If that is the case, this CT should not affect the others at all in terms of IO. best regards, Sirk > > Best, > Rene > > On Tue, May 22, 2012 at 3:06 PM, Sirk Johannsen <s.johannsen@satzmedia.de> > wrote: >> >> Hi Rene, >> >> Since CPU and MEM are fine it's most likely to be Disk-IO. >> I have similar Problems with a Cluster Setup based on OpenVZ. >> The problem is that our Storage is way to slow. >> We have been accessing the storage via NFS and put all our CTs private >> areas on it. >> I noticed many times that one CT was doing a lot of disk IO and all >> other were suffering from that... that even lead to total system >> failures. >> This has been solved by converting everything to ploop. Since then our >> system is at least in a stable state. >> IO Performance is still an issue but does not bring our system down. >> >> You should give ploop a try :-) I am very happy with it. >> >> best regards, >> >> Sirk >> >> 2012/5/21 Rene Dokbua <openvz@dokbua.com>: >> > Hello, >> > >> > I occasionally get this extreme load on one of our VPS servers. It is >> > quite >> > large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + >> > parked/addon/subdomains. >> > >> > The hardware node has 12 active VPS servers and most of the time things >> > are >> > chugging along just fine, something like this. >> > >> > 1401: 0.00 0.00 0.00 1/23 4561 >> > 1402: 0.02 0.05 0.05 1/57 16991 >> > 1404: 0.01 0.02 0.00 1/73 18863 >> > 1406: 0.07 0.13 0.06 1/39 31189 >> > 1407: 0.86 1.03 1.14 1/113 31460 >> > 1408: 0.17 0.17 0.18 1/79 32579 >> > 1409: 0.00 0.00 0.02 1/77 21784 >> > 1410: 0.01 0.02 0.00 1/60 7454 >> > 1413: 0.00 0.00 0.00 1/46 18579 >> > 1414: 0.00 0.00 0.00 1/41 23812 >> > 1415: 0.00 0.00 0.00 1/45 9831 >> > 1416: 0.05 0.02 0.00 1/59 11332 >> > 12 active >> > >> > The problem VPS is 1407. As you can see below it only uses a bit of the >> > cpu >> > and memory. >> > >> > top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, >> > 1.09 >> > Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie >> > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, >> > 0.1%st >> > Mem: 4194304k total, 2550572k used, 1643732k free, 0k buffers >> > Swap: 8388608k total, 105344k used, 8283264k free, 1793828k cached >> > >> > Also iostat and vmstat shows no particular io or swap activity. >> > >> > Now for the problem. Every once in a while the loadavg of this >> > particular >> > VPS shoots up to like crazy values, 30 or more and it becomes completely >> > sluggish. The odd thing is load goes up for the VPS server, and starts >> > spilling into other VPS serers on the same hardware node - but there are >> > still no particular cpu/memory/io usage going on that I can se. No >> > particular network activity. In this example load has fallen back to >> > around 10 but it was much higher earlier. >> > >> > 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, 19.11, >> > 18.87 >> > >> > 1401: 0.01 0.03 0.00 1/23 2876 >> > 1402: 0.00 0.11 0.13 1/57 15334 >> > 1404: 0.02 0.20 0.16 1/77 14918 >> > 1406: 0.01 0.13 0.10 1/39 29595 >> > 1407: 10.95 15.71 15.05 1/128 13950 >> > 1408: 0.36 0.52 0.57 1/81 27167 >> > 1409: 0.09 0.26 0.43 1/78 17851 >> > 1410: 0.09 0.17 0.18 1/61 4344 >> > 1413: 0.00 0.03 0.00 1/46 16539 >> > 1414: 0.01 0.01 0.00 1/41 22372 >> > 1415: 0.00 0.01 0.00 1/45 8404 >> > 1416: 0.05 0.10 0.11 1/58 9292 >> > 12 active >> > >> > top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, 14.97, >> > 14.82 >> > Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie >> > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, >> > 0.1%st >> > Mem: 4194304k total, 1173844k used, 3020460k free, 0k buffers >> > Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache >> > >> > Notice how cpu is plenty idle, and only 1/4 of the available memory is >> > being >> > used. >> > >> > http://wiki.openvz.org/Ploop/Why explains "One such property that >> > deserves a >> > special item in this list is file system journal. While journal is a >> > good >> > thing to have, because it helps to maintain file system integrity and >> > improve reboot times (by eliminating fsck in many cases), it is also a >> > bottleneck for containers. If one container will fill up in-memory >> > journal >> > (with lots of small operations leading to file metadata updates, e.g. >> > file >> > truncates), all the other containers I/O will block waiting for the >> > journal >> > to be written to disk. In some extreme cases we saw up to 15 seconds of >> > such >> > blockage.". The problem I noticed last much longer than 15 seconds >> > though >> > - typically 15-30 minutes, then load goes back where it should be. >> > >> > Any suggestions where I could look for the cause of this? It's not like >> > it >> > happens everyday, maybe once or twice per month, but it's enough to >> > cause >> > customers to complain. >> > >> > Regards, >> > Rene >> > >> > >> -- >> Satzmedia GmbH >> >> Altonaer Poststraße 9 >> 22767 Hamburg >> Tel: +49 (0) 40 - 1 888 969 - 140 >> Fax: +49 (0) 40 - 1 888 969 - 200 >> E-Mail: s.johannsen@satzmedia.de >> E-Business-Lösungen: http://www.satzmedia.de >> Amtsgericht Hamburg, HRB 71729 >> Ust-IDNr. DE201979921 >> Geschäftsführer: >> Dipl.-Kfm. Christian Satz >> Dipl.-Inform. Markus Meyer-Westphal >> >> -- >> >> >> -- Satzmedia GmbH Altonaer Poststraße 9 22767 Hamburg Tel: +49 (0) 40 - 1 888 969 - 140 Fax: +49 (0) 40 - 1 888 969 - 200 E-Mail: s.johannsen@satzmedia.de E-Business-Lösungen: http://www.satzmedia.de Amtsgericht Hamburg, HRB 71729 Ust-IDNr. DE201979921 Geschäftsführer: Dipl.-Kfm. Christian Satz Dipl.-Inform. Markus Meyer-Westphal --
Hi Rene, Did you check the /proc/user_beancounters of that VPS? Sometimes a high load could be caused by buffers that are full. Hope it helpes you, Kind Regards, Esme de Wolf Van: users-bounces@openvz.org [mailto:users-bounces@openvz.org] Namens Rene Dokbua Verzonden: maandag 21 mei 2012 20:07 Aan: users@openvz.org Onderwerp: [Users] occasional high loadavg without any noticeable cpu/memory/io load Hello, I occasionally get this extreme load on one of our VPS servers. It is quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 websites + parked/addon/subdomains. The hardware node has 12 active VPS servers and most of the time things are chugging along just fine, something like this. 1401: 0.00 0.00 0.00 1/23 4561 1402: 0.02 0.05 0.05 1/57 16991 1404: 0.01 0.02 0.00 1/73 18863 1406: 0.07 0.13 0.06 1/39 31189 1407: 0.86 1.03 1.14 1/113 31460 1408: 0.17 0.17 0.18 1/79 32579 1409: 0.00 0.00 0.02 1/77 21784 1410: 0.01 0.02 0.00 1/60 7454 1413: 0.00 0.00 0.00 1/46 18579 1414: 0.00 0.00 0.00 1/41 23812 1415: 0.00 0.00 0.00 1/45 9831 1416: 0.05 0.02 0.00 1/59 11332 12 active The problem VPS is 1407. As you can see below it only uses a bit of the cpu and memory. top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, 0.95, 1.09 Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st Mem: 4194304k total, 2550572k used, 1643732k free, 0k buffers Swap: 8388608k total, 105344k used, 8283264k free, 1793828k cached Also iostat and vmstat shows no particular io or swap activity. Now for the problem. Every once in a while the loadavg of this particular VPS shoots up to like crazy values, 30 or more and it becomes completely sluggish. The odd thing is load goes up for the VPS server, and starts spilling into other VPS serers on the same hardware node - but there are still no particular cpu/memory/io usage going on that I can se. No particular network activity. In this example load has fallen back to around 10 but it was much higher earlier. 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, 19.11, 18.87 1401: 0.01 0.03 0.00 1/23 2876 1402: 0.00 0.11 0.13 1/57 15334 1404: 0.02 0.20 0.16 1/77 14918 1406: 0.01 0.13 0.10 1/39 29595 1407: 10.95 15.71 15.05 1/128 13950 1408: 0.36 0.52 0.57 1/81 27167 1409: 0.09 0.26 0.43 1/78 17851 1410: 0.09 0.17 0.18 1/61 4344 1413: 0.00 0.03 0.00 1/46 16539 1414: 0.01 0.01 0.00 1/41 22372 1415: 0.00 0.01 0.00 1/45 8404 1416: 0.05 0.10 0.11 1/58 9292 12 active top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, 14.97, 14.82 Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, 0.0%si, 0.1%st Mem: 4194304k total, 1173844k used, 3020460k free, 0k buffers Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache Notice how cpu is plenty idle, and only 1/4 of the available memory is being used. http://wiki.openvz.org/Ploop/Why explains "One such property that deserves a special item in this list is file system journal. While journal is a good thing to have, because it helps to maintain file system integrity and improve reboot times (by eliminating fsck in many cases), it is also a bottleneck for containers. If one container will fill up in-memory journal (with lots of small operations leading to file metadata updates, e.g. file truncates), all the other containers I/O will block waiting for the journal to be written to disk. In some extreme cases we saw up to 15 seconds of such blockage.". The problem I noticed last much longer than 15 seconds though - typically 15-30 minutes, then load goes back where it should be. Any suggestions where I could look for the cause of this? It's not like it happens everyday, maybe once or twice per month, but it's enough to cause customers to complain. Regards, Rene
Hi Sirk, > If you want some practical information on ploop: We are using it in a > highly productive environment. > It was either, try ploop and hope it works, or have the systems fail > every 2nd day. > So we decided to use ploop and are more than happy. > It even solves a lot of issues we had with the private areas directly > on the nfs share. > But of course, thats totally up to you. > I started with only a few "unimportant" CTs and then merged everything > after a while (42 CTs). > Thanks for the info, much appreciated! Maybe a little off topic, but I am curious to know: At the moment I find it very convenient to go directly into a containers filesystem from the hardware node - i.e. something like /vz/private/xxx/var/log/... etc - Would I be correct in presuming that by using ploop this will no longer be possible? I know I could just setup a test system and try it out but if you know already it would save me some time ;) > So you have a different FileSystem for the "problem"-Container that is > even on a different disk ? > If that is the case, this CT should not affect the others at all in terms > of IO. > > Indeed, this is the only container on that filesystem and that physical drive. This time there were no "spill over" but previous times when load hit 50 or more the load certainly did spill into other containers. Best, Rene
Hi Esme, > Did you check the /proc/user_beancounters of that VPS? Sometime’s a high load could be caused by buffers that are full. Thanks for the suggestion, much appreciated! I didn't think of checking at the time I'm afraid. I suppose since the container has not been rebooted since, the beancounters should still show any problems encountered at the time right? Below is the user_beancounters of the problem CT. I notice physpages and dcachesize have maxheld values very close to limits (even if failcnt is zero) could that have been the cause? uid resource held maxheld barrier limit failcnt 1407: kmemsize 252703307 1124626432 1932525568 2147483648 0 lockedpages 0 15 524288 524288 0 privvmpages 893372 5683554 9223372036854775807 9223372036854775807 0 shmpages 23 7399 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 numproc 136 480 9223372036854775807 9223372036854775807 0 physpages 733468 1048591 0 1048576 0 vmguarpages 0 0 0 9223372036854775807 0 oomguarpages 137691 676209 0 9223372036854775807 0 numtcpsock 101 459 9223372036854775807 9223372036854775807 0 numflock 7 37 9223372036854775807 9223372036854775807 0 numpty 1 4 9223372036854775807 9223372036854775807 0 numsiginfo 0 66 9223372036854775807 9223372036854775807 0 tcpsndbuf 4024896 34884168 9223372036854775807 9223372036854775807 0 tcprcvbuf 1654784 7520256 9223372036854775807 9223372036854775807 0 othersockbuf 195136 3887232 9223372036854775807 9223372036854775807 0 dgramrcvbuf 0 155848 9223372036854775807 9223372036854775807 0 numothersock 130 346 9223372036854775807 9223372036854775807 0 dcachesize 222868425 1073741824 965738496 1073741824 0 numfile 3853 12765 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 numiptent 197 197 9223372036854775807 9223372036854775807 0 I'm not that familiar with the nitty-gritties of the beancounters but these are the values I have in the 1407.conf file. PHYSPAGES="0:4096M" SWAPPAGES="0:8192M" KMEMSIZE="1843M:2048M" DCACHESIZE="921M:1024M" LOCKEDPAGES="2048M" PRIVVMPAGES="unlimited" SHMPAGES="unlimited" NUMPROC="unlimited" VMGUARPAGES="0:unlimited" OOMGUARPAGES="0:unlimited" NUMTCPSOCK="unlimited" NUMFLOCK="unlimited" NUMPTY="unlimited" NUMSIGINFO="unlimited" TCPSNDBUF="unlimited" TCPRCVBUF="unlimited" OTHERSOCKBUF="unlimited" DGRAMRCVBUF="unlimited" NUMOTHERSOCK="unlimited" NUMFILE="unlimited" NUMIPTENT="unlimited" When user_beancounters physpage limit is 1048576, with PHYSPAGES set to 4GB, then the held value of 733468 should correspond to about 3GB, right? But top only shows about 1.5GB used at the same time - how is that possible? dcachesize I think is filesystem stuff? But there seems to be plenty of resources there; # df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/simfs 20000000 3046139 16953861 16% / none 524288 109 524179 1% /dev # df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 492G 156G 312G 34% / none 2.0G 4.0K 2.0G 1% /dev Best, Rene
> > > When user_beancounters physpage limit is 1048576, with PHYSPAGES set to > 4GB, then the held value of 733468 should correspond to about 3GB, right? > But top only shows about 1.5GB used at the same time - how is that > possible? > > Actually at the time I cat'ed these beans the memory used according to top was around 2.5GB so that seems right enough. Still doesn't explain how maxheld is so close to limit when top at the time of the trouble showed just around 1.5G memory used.
Looks like in your case you've hit physpages limit. In such situations VPS behaves as a standalone machine - it starts to swap out (though "virtually") and process stuck in D state (swap in / swap out), which contributes to loadavg. So either increase memory limits for your VPS or kill/tune the memory hungry workload. Note: loadavg can also increase due to CPU limits as processes are delayed when overuse their CPU. Thanks, Kirill On May 22, 2012, at 14:49 , Rene C. wrote: Hi Esme, > Did you check the /proc/user_beancounters of that VPS? Sometime’s a high load could be caused by buffers that are full. Thanks for the suggestion, much appreciated! I didn't think of checking at the time I'm afraid. I suppose since the container has not been rebooted since, the beancounters should still show any problems encountered at the time right? Below is the user_beancounters of the problem CT. I notice physpages and dcachesize have maxheld values very close to limits (even if failcnt is zero) could that have been the cause? uid resource held maxheld barrier limit failcnt 1407: kmemsize 252703307 1124626432 1932525568 2147483648 0 lockedpages 0 15 524288 524288 0 privvmpages 893372 5683554 9223372036854775807 9223372036854775807 0 shmpages 23 7399 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 numproc 136 480 9223372036854775807 9223372036854775807 0 physpages 733468 1048591 0 1048576 0 vmguarpages 0 0 0 9223372036854775807 0 oomguarpages 137691 676209 0 9223372036854775807 0 numtcpsock 101 459 9223372036854775807 9223372036854775807 0 numflock 7 37 9223372036854775807 9223372036854775807 0 numpty 1 4 9223372036854775807 9223372036854775807 0 numsiginfo 0 66 9223372036854775807 9223372036854775807 0 tcpsndbuf 4024896 34884168 9223372036854775807 9223372036854775807 0 tcprcvbuf 1654784 7520256 9223372036854775807 9223372036854775807 0 othersockbuf 195136 3887232 9223372036854775807 9223372036854775807 0 dgramrcvbuf 0 155848 9223372036854775807 9223372036854775807 0 numothersock 130 346 9223372036854775807 9223372036854775807 0 dcachesize 222868425 1073741824 965738496 1073741824 0 numfile 3853 12765 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 numiptent 197 197 9223372036854775807 9223372036854775807 0 I'm not that familiar with the nitty-gritties of the beancounters but these are the values I have in the 1407.conf file. PHYSPAGES="0:4096M" SWAPPAGES="0:8192M" KMEMSIZE="1843M:2048M" DCACHESIZE="921M:1024M" LOCKEDPAGES="2048M" PRIVVMPAGES="unlimited" SHMPAGES="unlimited" NUMPROC="unlimited" VMGUARPAGES="0:unlimited" OOMGUARPAGES="0:unlimited" NUMTCPSOCK="unlimited" NUMFLOCK="unlimited" NUMPTY="unlimited" NUMSIGINFO="unlimited" TCPSNDBUF="unlimited" TCPRCVBUF="unlimited" OTHERSOCKBUF="unlimited" DGRAMRCVBUF="unlimited" NUMOTHERSOCK="unlimited" NUMFILE="unlimited" NUMIPTENT="unlimited" When user_beancounters physpage limit is 1048576, with PHYSPAGES set to 4GB, then the held value of 733468 should correspond to about 3GB, right? But top only shows about 1.5GB used at the same time - how is that possible? dcachesize I think is filesystem stuff? But there seems to be plenty of resources there; # df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/simfs 20000000 3046139 16953861 16% / none 524288 109 524179 1% /dev # df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 492G 156G 312G 34% / none 2.0G 4.0K 2.0G 1% /dev Best, Rene <ATT00001.c>
I also think that these UBC settings are not consistent. Especially when you have all containers configured with these same UBC settings you will have soon or later problems. See: http://wiki.openvz.org/UBC_consistency_check and other pages on the WIKI. Kind Regards, Esme Van: users-bounces@openvz.org [mailto:users-bounces@openvz.org] Namens Kirill Korotaev Verzonden: dinsdag 22 mei 2012 13:05 Aan: users@openvz.org users@openvz.org; Rene C. Onderwerp: Re: [Users] occasional high loadavg without any noticeable cpu/memory/io load Looks like in your case you've hit physpages limit. In such situations VPS behaves as a standalone machine - it starts to swap out (though "virtually") and process stuck in D state (swap in / swap out), which contributes to loadavg. So either increase memory limits for your VPS or kill/tune the memory hungry workload. Note: loadavg can also increase due to CPU limits as processes are delayed when overuse their CPU. Thanks, Kirill On May 22, 2012, at 14:49 , Rene C. wrote: Hi Esme, > Did you check the /proc/user_beancounters of that VPS? Sometimes a high load could be caused by buffers that are full. Thanks for the suggestion, much appreciated! I didn't think of checking at the time I'm afraid. I suppose since the container has not been rebooted since, the beancounters should still show any problems encountered at the time right? Below is the user_beancounters of the problem CT. I notice physpages and dcachesize have maxheld values very close to limits (even if failcnt is zero) could that have been the cause? uid resource held maxheld barrier limit failcnt 1407: kmemsize 252703307 1124626432 1932525568 2147483648 0 lockedpages 0 15 524288 524288 0 privvmpages 893372 5683554 9223372036854775807 9223372036854775807 0 shmpages 23 7399 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 numproc 136 480 9223372036854775807 9223372036854775807 0 physpages 733468 1048591 0 1048576 0 vmguarpages 0 0 0 9223372036854775807 0 oomguarpages 137691 676209 0 9223372036854775807 0 numtcpsock 101 459 9223372036854775807 9223372036854775807 0 numflock 7 37 9223372036854775807 9223372036854775807 0 numpty 1 4 9223372036854775807 9223372036854775807 0 numsiginfo 0 66 9223372036854775807 9223372036854775807 0 tcpsndbuf 4024896 34884168 9223372036854775807 9223372036854775807 0 tcprcvbuf 1654784 7520256 9223372036854775807 9223372036854775807 0 othersockbuf 195136 3887232 9223372036854775807 9223372036854775807 0 dgramrcvbuf 0 155848 9223372036854775807 9223372036854775807 0 numothersock 130 346 9223372036854775807 9223372036854775807 0 dcachesize 222868425 1073741824 965738496 1073741824 0 numfile 3853 12765 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 numiptent 197 197 9223372036854775807 9223372036854775807 0 I'm not that familiar with the nitty-gritties of the beancounters but these are the values I have in the 1407.conf file. PHYSPAGES="0:4096M" SWAPPAGES="0:8192M" KMEMSIZE="1843M:2048M" DCACHESIZE="921M:1024M" LOCKEDPAGES="2048M" PRIVVMPAGES="unlimited" SHMPAGES="unlimited" NUMPROC="unlimited" VMGUARPAGES="0:unlimited" OOMGUARPAGES="0:unlimited" NUMTCPSOCK="unlimited" NUMFLOCK="unlimited" NUMPTY="unlimited" NUMSIGINFO="unlimited" TCPSNDBUF="unlimited" TCPRCVBUF="unlimited" OTHERSOCKBUF="unlimited" DGRAMRCVBUF="unlimited" NUMOTHERSOCK="unlimited" NUMFILE="unlimited" NUMIPTENT="unlimited" When user_beancounters physpage limit is 1048576, with PHYSPAGES set to 4GB, then the held value of 733468 should correspond to about 3GB, right? But top only shows about 1.5GB used at the same time - how is that possible? dcachesize I think is filesystem stuff? But there seems to be plenty of resources there; # df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/simfs 20000000 3046139 16953861 16% / none 524288 109 524179 1% /dev # df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 492G 156G 312G 34% / none 2.0G 4.0K 2.0G 1% /dev Best, Rene <ATT00001.c>
2012/5/22 Rene C. <openvz@dokbua.com>: > Hi Sirk, > >> >> If you want some practical information on ploop: We are using it in a >> highly productive environment. >> It was either, try ploop and hope it works, or have the systems fail >> every 2nd day. >> So we decided to use ploop and are more than happy. >> It even solves a lot of issues we had with the private areas directly >> on the nfs share. >> But of course, thats totally up to you. >> I started with only a few "unimportant" CTs and then merged everything >> after a while (42 CTs). > > > Thanks for the info, much appreciated! > > Maybe a little off topic, but I am curious to know: At the moment I find it > very convenient to go directly into a containers filesystem from the > hardware node - i.e. something like /vz/private/xxx/var/log/... etc - Would > I be correct in presuming that by using ploop this will no longer be > possible? I know I could just setup a test system and try it out but if you > know already it would save me some time ;) Only partially correct :-) You can enter the filesystem of a CT when it's mounted. Meaning - you can enter the root directory when the CT is running. If the CT is shut down you always have the possibility to mount the ploop file to any directory you desire. > >> >> So you have a different FileSystem for the "problem"-Container that is >> even on a different disk ? >> If that is the case, this CT should not affect the others at all in terms >> of IO. >> > > Indeed, this is the only container on that filesystem and that physical > drive. This time there were no "spill over" but previous times when load > hit 50 or more the load certainly did spill into other containers. > > Best, > Rene > > -- Satzmedia GmbH Altonaer Poststraße 9 22767 Hamburg Tel: +49 (0) 40 - 1 888 969 - 140 Fax: +49 (0) 40 - 1 888 969 - 200 E-Mail: s.johannsen@satzmedia.de E-Business-Lösungen: http://www.satzmedia.de Amtsgericht Hamburg, HRB 71729 Ust-IDNr. DE201979921 Geschäftsführer: Dipl.-Kfm. Christian Satz Dipl.-Inform. Markus Meyer-Westphal --
On Tue, May 22, 2012 at 6:59 PM, Esmé de Wolf <esme@elements.nl> wrote: > I also think that these UBC settings are not consistent. Especially when > you have all containers configured with these same UBC settings you will > have soon or later problems.**** > > ** ** > > See: http://wiki.openvz.org/UBC_consistency_check and other pages on the > WIKI.**** > > ** ** > > Kind Regards,**** > > > Esme > I read that UBC page already and used it to set these values. No, all my containers do not have the same UBC settings, they were set depending on how much resources each container should have. Please let me know where any of the values in my conf file conflicts with the UBC recommendations. I do understand that they may need to be fine tuned in each case, but that's basically what this question is about :) So basically at this time I have two questions I don't understand: 1) how is it possible to have physpages hit the limit when top never shows more than about 75-80% of the memory used? 2) how did dcachesize hit limit when both df -i and df -h shows plenty of resources - and haven't been close to limits? Could the values in the beancounter file be old? Is there a way to reset them (without restarting the CT) ? Best, Rene
On May 22, 2012, at 16:17 , Rene C. wrote: On Tue, May 22, 2012 at 6:59 PM, Esmé de Wolf <esme@elements.nl<mailto:esme@elements.nl>> wrote: I also think that these UBC settings are not consistent. Especially when you have all containers configured with these same UBC settings you will have soon or later problems. See: http://wiki.openvz.org/UBC_consistency_check and other pages on the WIKI. Kind Regards, Esme I read that UBC page already and used it to set these values. No, all my containers do not have the same UBC settings, they were set depending on how much resources each container should have. Please let me know where any of the values in my conf file conflicts with the UBC recommendations. I do understand that they may need to be fine tuned in each case, but that's basically what this question is about :) So basically at this time I have two questions I don't understand: 1) how is it possible to have physpages hit the limit when top never shows more than about 75-80% of the memory used? once again: top shows current (immedeate) values. maxheld in user_beancounters shows you *maximum* over time. There is an API for resetting it AFAIR, but no user-space tool in OpenVZ :((( 2) how did dcachesize hit limit when both df -i and df -h shows plenty of resources - and haven't been close to limits? dcachesize has nothing to do with df. it's kernel memory used for paths and pinned by opened files and CWD. You can safely increase it if needed. It's just DoS protection. Could the values in the beancounter file be old? Is there a way to reset them (without restarting the CT) ? Best, Rene <ATT00001.c>
You could check your <VEID>.conf with vzcfgvalidate. But I think there is quite a risk when giving one of your CTs unlimited resources. If you want to read-out the UBCs from the node and see when one fails I could recommend you a very good script Im using myself; http://wiki.openvz.org/UBC_failcnt_reset There is no need to reset the values inside your CT. Van: users-bounces@openvz.org [mailto:users-bounces@openvz.org] Namens Rene C. Verzonden: dinsdag 22 mei 2012 14:17 Aan: users@openvz.org Onderwerp: Re: [Users] occasional high loadavg without any noticeable cpu/memory/io load On Tue, May 22, 2012 at 6:59 PM, Esmé de Wolf <esme@elements.nl> wrote: I also think that these UBC settings are not consistent. Especially when you have all containers configured with these same UBC settings you will have soon or later problems. See: http://wiki.openvz.org/UBC_consistency_check and other pages on the WIKI. Kind Regards, Esme I read that UBC page already and used it to set these values. No, all my containers do not have the same UBC settings, they were set depending on how much resources each container should have. Please let me know where any of the values in my conf file conflicts with the UBC recommendations. I do understand that they may need to be fine tuned in each case, but that's basically what this question is about :) So basically at this time I have two questions I don't understand: 1) how is it possible to have physpages hit the limit when top never shows more than about 75-80% of the memory used? 2) how did dcachesize hit limit when both df -i and df -h shows plenty of resources - and haven't been close to limits? Could the values in the beancounter file be old? Is there a way to reset them (without restarting the CT) ? Best, Rene
On Tue, May 22, 2012 at 7:54 PM, Esmé de Wolf <esme@elements.nl> wrote: > You could check your <VEID>.conf with vzcfgvalidate. But I think there is > quite a risk when giving one of your CT’s unlimited resources. If you want > to read-out the UBC’s from the node and see when one fails I could > recommend you a very good script I’m using myself; > http://wiki.openvz.org/UBC_failcnt_reset There is no need to reset the > value’s inside your CT.**** > > > Apparently no problems with the file: # vzcfgvalidate -v yes 1407.conf Validation completed: success Thank you to everyone who provided suggestions, ideas and insight. I've added the user_beancounters to my loadmonitoring script. Next time there is a problem I'll check if any values are hitting the limit and see if increasing them may fix the problem. Best, Rene
Hi, ?? 22.5.2012 ?. 13:27 ?., Rene C. ??????: > Hi Sirk, > > > If you want some practical information on ploop: We are using it in a > highly productive environment. > It was either, try ploop and hope it works, or have the systems fail > every 2nd day. > So we decided to use ploop and are more than happy. > It even solves a lot of issues we had with the private areas directly > on the nfs share. > But of course, thats totally up to you. > I started with only a few "unimportant" CTs and then merged everything > after a while (42 CTs). > > > Thanks for the info, much appreciated! > > Maybe a little off topic, but I am curious to know: At the moment I > find it very convenient to go directly into a containers filesystem > from the hardware node - i.e. something like > /vz/private/xxx/var/log/... etc - Would I be correct in presuming > that by using ploop this will no longer be possible? I know I could > just setup a test system and try it out but if you know already it > would save me some time ;) > It's not very practical to access the containers from the VZ/private mount point, as it breaks for example the quota stats of the container. If you still want to do things there better go for the VZ/root mount point. (Advice given to me by one of the now-a-days developer of Viruozzo) And as you already mentioned ploop, as far as I know the ploop-container will be mounted to VZ/root of the CT and you'll still have access to the info in there. > > So you have a different FileSystem for the "problem"-Container that is > even on a different disk ? > If that is the case, this CT should not affect the others at all > in terms of IO. > > > Indeed, this is the only container on that filesystem and that > physical drive. This time there were no "spill over" but previous > times when load hit 50 or more the load certainly did spill into > other containers. > Best, > Rene > >
"Rene C." <openvz@dokbua.com> writes: > Thank you to everyone who provided suggestions, ideas and insight. I've > added the user_beancounters to my loadmonitoring script. Next time there > is a problem I'll check if any values are hitting the limit and see if > increasing them may fix the problem. Well, all your failcnt's were zero so there was nothing hitting the limit. I'm also interested in what you found since I'm having the same problems on one of my clusters: unexplained high load that goes away as it came: unexplained and suddenly ;)
On 05/22/2012 09:09 PM, Rene C. wrote: > > > On Tue, May 22, 2012 at 7:54 PM, Esmé de Wolf <esme@elements.nl > <mailto:esme@elements.nl>> wrote: > > You could check your <VEID>.conf with vzcfgvalidate. But I think > there is quite a risk when giving one of your CT’s unlimited > resources. If you want to read-out the UBC’s from the node and see > when one fails I could recommend you a very good script I’m using > myself; http://wiki.openvz.org/UBC_failcnt_reset There is no need > to reset the value’s inside your CT. > > > > Apparently no problems with the file: > > # vzcfgvalidate -v yes 1407.conf > Validation completed: success > > Thank you to everyone who provided suggestions, ideas and insight. > I've added the user_beancounters to my loadmonitoring script. Next > time there is a problem I'll check if any values are hitting the limit > and see if increasing them may fix the problem. > Try vzubc -q or something, it might help.
On 05/22/2012 01:06 PM, Rene C. wrote: > > Actually I made a small shell script that loops through the list of > active containers and outputs the content of each containers > /proc/loadavg. It started out as a bit more elaborate script that was > intended to provide some of the functionality of a script vzstat, that > I used to use with Virtuozzo. > > You can download both scripts from > https://www.ourhelpdesk.net/downloads/z.tgz vzlist have laverage field that might be of use. I.e. vzlist -o ctid,laverage > > > > On Tue, May 22, 2012 at 3:15 PM, Steffan <general@ziggo.nl > <mailto:general@ziggo.nl>> wrote: > > Sorry dont have the answer for you > > But can you tell me what command you used to see all loads on your > node ? > > Thanxs Steffan > > *Van:*users-bounces@openvz.org <mailto:users-bounces@openvz.org> > [mailto:users-bounces@openvz.org > <mailto:users-bounces@openvz.org>] *Namens *Rene Dokbua > *Verzonden:* maandag 21 mei 2012 20:07 > *Aan:* users@openvz.org <mailto:users@openvz.org> > *Onderwerp:* [Users] occasional high loadavg without any > noticeable cpu/memory/io load > > Hello, > > I occasionally get this extreme load on one of our VPS servers. It > is quite large, 4 full E31230 cores, 4 GB RAM and hosting ca. 400 > websites + parked/addon/subdomains. > > The hardware node has 12 active VPS servers and most of the time > things are chugging along just fine, something like this. > > 1401: 0.00 0.00 0.00 1/23 4561 > > 1402: 0.02 0.05 0.05 1/57 16991 > > 1404: 0.01 0.02 0.00 1/73 18863 > > 1406: 0.07 0.13 0.06 1/39 31189 > > 1407: 0.86 1.03 1.14 1/113 31460 > > 1408: 0.17 0.17 0.18 1/79 32579 > > 1409: 0.00 0.00 0.02 1/77 21784 > > 1410: 0.01 0.02 0.00 1/60 7454 > > 1413: 0.00 0.00 0.00 1/46 18579 > > 1414: 0.00 0.00 0.00 1/41 23812 > > 1415: 0.00 0.00 0.00 1/45 9831 > > 1416: 0.05 0.02 0.00 1/59 11332 > > 12 active > > The problem VPS is 1407. As you can see below it only uses a bit > of the cpu and memory. > > top - 17:34:12 up 32 days, 12:21, 0 users, load average: 0.78, > 0.95, 1.09 > > Tasks: 102 total, 4 running, 90 sleeping, 0 stopped, 8 zombie > > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, > 0.0%si, 0.1%st > > Mem: 4194304k total, 2550572k used, 1643732k free, 0k > buffers > > Swap: 8388608k total, 105344k used, 8283264k free, 1793828k > cached > > Also iostat and vmstat shows no particular io or swap activity. > > Now for the problem. Every once in a while the loadavg of this > particular VPS shoots up to like crazy values, 30 or more and it > becomes completely sluggish. The odd thing is load goes up for the > VPS server, and starts spilling into other VPS serers on the same > hardware node - but there are still no particular cpu/memory/io > usage going on that I can se. No particular network activity. > In this example load has fallen back to around 10 but it was much > higher earlier. > > 16:19:44 up 32 days, 11:19, 3 users, load average: 12.87, > 19.11, 18.87 > > 1401: 0.01 0.03 0.00 1/23 2876 > > 1402: 0.00 0.11 0.13 1/57 15334 > > 1404: 0.02 0.20 0.16 1/77 14918 > > 1406: 0.01 0.13 0.10 1/39 29595 > > 1407: 10.95 15.71 15.05 1/128 13950 > > 1408: 0.36 0.52 0.57 1/81 27167 > > 1409: 0.09 0.26 0.43 1/78 17851 > > 1410: 0.09 0.17 0.18 1/61 4344 > > 1413: 0.00 0.03 0.00 1/46 16539 > > 1414: 0.01 0.01 0.00 1/41 22372 > > 1415: 0.00 0.01 0.00 1/45 8404 > > 1416: 0.05 0.10 0.11 1/58 9292 > > 12 active > > top - 16:20:02 up 32 days, 11:07, 0 users, load average: 9.14, > 14.97, 14.82 > > Tasks: 135 total, 1 running, 122 sleeping, 0 stopped, 12 zombie > > Cpu(s): 16.3%us, 2.9%sy, 0.4%ni, 78.5%id, 1.8%wa, 0.0%hi, > 0.0%si, 0.1%st > > Mem: 4194304k total, 1173844k used, 3020460k free, 0k > buffers > > Swap: 8388608k total, 115576k used, 8273032k free, 725144k cache > > Notice how cpu is plenty idle, and only 1/4 of the available > memory is being used. > > http://wiki.openvz.org/Ploop/Why explains "One such property that > deserves a special item in this list is file system journal. While > journal is a good thing to have, because it helps to maintain file > system integrity and improve reboot times (by eliminating fsck in > many cases), it is also a bottleneck for containers. If one > container will fill up in-memory journal (with lots of small > operations leading to file metadata updates, e.g. file truncates), > all the other containers I/O will block waiting for the journal to > be written to disk. In some extreme cases we saw up to 15 seconds > of such blockage.". The problem I noticed last much longer than > 15 seconds though - typically 15-30 minutes, then load goes back > where it should be. > > Any suggestions where I could look for the cause of this? It's > not like it happens everyday, maybe once or twice per month, but > it's enough to cause customers to complain. > > Regards, > Rene > > > _______________________________________________ > Users mailing list > Users@openvz.org <mailto:Users@openvz.org> > https://openvz.org/mailman/listinfo/users > >
Hi Kir, Both the vzubc command and the laverage option to vzlist were new to me (the laverage options seems undocumented?) Thanks much, this is VERY useful information!! Regards, Rene