Home » General » Support » OpenVZ 7 containers crashing with ext4 errors
Re: OpenVZ 7 containers crashing with ext4 errors [message #53664 is a reply to message #53656] |
Wed, 22 July 2020 11:45 ![Go to previous message Go to previous message](/theme/ovz3/images/up.png) ![Go to next message Go to previous message](/theme/ovz3/images/down.png) |
allan.talver
Messages: 9 Registered: July 2020
|
Junior Member |
|
|
Hello
Thank you for the reply! According to your suggestion we updated all our existing OpenVZ 7 nodes to the 14 version over the past two weeks. We have not yet experienced any crashes on updated nodes (one crash did happen on 11 July on a node that had not yet been updated).
As it was pointed out, it seems that disk space could be a contributing factor to the issue. But I can assure that disks of these failing containers are definitely not full. Some have very low utilisation (around 30%). We have noticed another behaviour which we see is possibly related to the crashes we have experienced, and also points to issues with not enough disk space available. While pcompact is running, some virtual containers show extreme changes in disk utilisation. Usually the disk suddenly shows as full and goes down to normal several times during the time pcompact runs. One example of pcompact.log output while one of such vps' is being compacted:
2020-07-22T02:00:12+0200 pcompact : Inspect 7a81d5ef-9a70-4a20-bb57-cf38f45b2926
2020-07-22T02:00:12+0200 pcompact : Inspect /vz/private/4001/root.hdd/DiskDescriptor.xml
2020-07-22T02:00:12+0200 pcompact : ploop=107520MB image=39805MB data=20760MB balloon=0MB
2020-07-22T02:00:12+0200 pcompact : Rate: 17.7 (threshold=10)
2020-07-22T02:00:12+0200 pcompact : Start compacting (to free 13669MB)
2020-07-22T02:00:12+0200 : Start defrag dev=/dev/ploop12981p1 mnt=/vz/root/4001 blocksize=2048
2020-07-22T02:09:48+0200 : Trying to find free extents bigger than 0 bytes granularity=1048576
2020-07-22T02:09:49+0200 pcompact : ploop=107520MB image=29687MB data=20767MB balloon=0MB
2020-07-22T02:09:49+0200 pcompact : Stats: uuid=7a81d5ef-9a70-4a20-bb57-cf38f45b2926 ploop_size=107520MB image_size_before=39805MB image_size_after=29687MB compaction_time=577.227s type=online
2020-07-22T02:09:49+0200 pcompact : End compacting
And then we have one container where disk usage stays near 100% (but fluctuating) for 2 hours until pcompact times out (I tried attaching a screenshot, but got an error that "Attachment is too big", even if the file was quite small.). Ploop.log shows:
2020-07-22T02:00:01+0200 pcompact : Inspect 240e4613-e12c-46b1-bc06-d001b12463c8
2020-07-22T02:00:01+0200 pcompact : Inspect /vz/private/4116/root.hdd/DiskDescriptor.xml
2020-07-22T02:00:01+0200 pcompact : ploop=261120MB image=116695MB data=87568MB balloon=0MB
2020-07-22T02:00:01+0200 pcompact : Rate: 11.2 (threshold=10)
2020-07-22T02:00:01+0200 pcompact : Start compacting (to free 16070MB)
2020-07-22T02:00:01+0200 : Start defrag dev=/dev/ploop48627p1 mnt=/vz/root/4116 blocksize=2048
2020-07-22T04:00:21+0200 : Error in wait_pid (balloon.c:967): The /usr/sbin/e4defrag2 process killed by signal 15
2020-07-22T04:00:21+0200 : /usr/sbin/e4defrag2 exited with error
2020-07-22T04:00:21+0200 : Trying to find free extents bigger than 0 bytes granularity=1048576
2020-07-22T04:00:23+0200 pcompact : ploop=261120MB image=100782MB data=87487MB balloon=0MB
2020-07-22T04:00:23+0200 pcompact : Stats: uuid=240e4613-e12c-46b1-bc06-d001b12463c8 ploop_size=261120MB image_size_before=116695MB image_size_after=100782MB compaction_time=7221.741s type=online
2020-07-22T04:00:23+0200 pcompact : End compacting
This node is running a MySQL server which shows errors during these 2 hours (different errors pointing to disk being full). Eventually MySQL crashes.
We'll continue to monitor and report back how it goes. But has anyone experienced such fluctuations in disk utilisation while pcompact is running? Is it somehow expected? How to get around applications failing due to disk showing as full (even when it is actually not). Worth mentioning that all these issues described in this post happen on newly created Ubuntu 18.04 containers (as opposed to my initial post where issues were mostly related to 16.04 containers migrated from OpenVZ 6).
Thanks!
[Updated on: Wed, 22 July 2020 11:50] Report message to a moderator
|
|
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Fri, 24 July 2020 10:56
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Fri, 24 July 2020 15:27
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Wed, 29 July 2020 15:24
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Mon, 03 August 2020 10:19
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: noc.r on Thu, 20 August 2020 12:34
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Thu, 20 August 2020 12:53
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: noc.r on Thu, 20 August 2020 15:04
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Thu, 20 August 2020 12:54
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Fri, 21 August 2020 11:33
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: noc.r on Mon, 24 August 2020 10:45
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Mon, 24 August 2020 11:16
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: noc.r on Mon, 24 August 2020 14:11
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: khorenko on Wed, 26 August 2020 11:46
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: khorenko on Wed, 26 August 2020 12:02
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
![icon14.gif](/images/message_icons/icon14.gif) |
Re: OpenVZ 7 containers crashing with ext4 errors
By: noc.r on Wed, 26 August 2020 12:24
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: ktkhai on Tue, 01 September 2020 12:04
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: noc.r on Thu, 03 September 2020 12:37
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
By: noc.r on Mon, 28 September 2020 14:12
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
![Read Message Read Message](/theme/ovz3/images/read.png) |
|
Re: OpenVZ 7 containers crashing with ext4 errors
|
Goto Forum:
Current Time: Sun Aug 25 13:40:06 GMT 2024
Total time taken to generate the page: 0.03536 seconds
|