OpenVZ Forum: Support » OpenVZ 7 containers crashing with ext4 errors

Home » General » Support » OpenVZ 7 containers crashing with ext4 errors

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: OpenVZ 7 containers crashing with ext4 errors [message #53664 is a reply to message #53656]

Wed, 22 July 2020 11:45

allan.talver
Messages: 9
Registered: July 2020

Junior Member

Hello

Thank you for the reply! According to your suggestion we updated all our existing OpenVZ 7 nodes to the 14 version over the past two weeks. We have not yet experienced any crashes on updated nodes (one crash did happen on 11 July on a node that had not yet been updated).

As it was pointed out, it seems that disk space could be a contributing factor to the issue. But I can assure that disks of these failing containers are definitely not full. Some have very low utilisation (around 30%). We have noticed another behaviour which we see is possibly related to the crashes we have experienced, and also points to issues with not enough disk space available. While pcompact is running, some virtual containers show extreme changes in disk utilisation. Usually the disk suddenly shows as full and goes down to normal several times during the time pcompact runs. One example of pcompact.log output while one of such vps' is being compacted:

2020-07-22T02:00:12+0200 pcompact : Inspect 7a81d5ef-9a70-4a20-bb57-cf38f45b2926
2020-07-22T02:00:12+0200 pcompact : Inspect /vz/private/4001/root.hdd/DiskDescriptor.xml
2020-07-22T02:00:12+0200 pcompact : ploop=107520MB image=39805MB data=20760MB balloon=0MB
2020-07-22T02:00:12+0200 pcompact : Rate: 17.7 (threshold=10)
2020-07-22T02:00:12+0200 pcompact : Start compacting (to free 13669MB)
2020-07-22T02:00:12+0200 : Start defrag dev=/dev/ploop12981p1 mnt=/vz/root/4001 blocksize=2048
2020-07-22T02:09:48+0200 : Trying to find free extents bigger than 0 bytes granularity=1048576
2020-07-22T02:09:49+0200 pcompact : ploop=107520MB image=29687MB data=20767MB balloon=0MB
2020-07-22T02:09:49+0200 pcompact : Stats: uuid=7a81d5ef-9a70-4a20-bb57-cf38f45b2926 ploop_size=107520MB image_size_before=39805MB image_size_after=29687MB compaction_time=577.227s type=online
2020-07-22T02:09:49+0200 pcompact : End compacting

And then we have one container where disk usage stays near 100% (but fluctuating) for 2 hours until pcompact times out (I tried attaching a screenshot, but got an error that "Attachment is too big", even if the file was quite small.). Ploop.log shows:

2020-07-22T02:00:01+0200 pcompact : Inspect 240e4613-e12c-46b1-bc06-d001b12463c8
2020-07-22T02:00:01+0200 pcompact : Inspect /vz/private/4116/root.hdd/DiskDescriptor.xml
2020-07-22T02:00:01+0200 pcompact : ploop=261120MB image=116695MB data=87568MB balloon=0MB
2020-07-22T02:00:01+0200 pcompact : Rate: 11.2 (threshold=10)
2020-07-22T02:00:01+0200 pcompact : Start compacting (to free 16070MB)
2020-07-22T02:00:01+0200 : Start defrag dev=/dev/ploop48627p1 mnt=/vz/root/4116 blocksize=2048
2020-07-22T04:00:21+0200 : Error in wait_pid (balloon.c:967): The /usr/sbin/e4defrag2 process killed by signal 15
2020-07-22T04:00:21+0200 : /usr/sbin/e4defrag2 exited with error
2020-07-22T04:00:21+0200 : Trying to find free extents bigger than 0 bytes granularity=1048576
2020-07-22T04:00:23+0200 pcompact : ploop=261120MB image=100782MB data=87487MB balloon=0MB
2020-07-22T04:00:23+0200 pcompact : Stats: uuid=240e4613-e12c-46b1-bc06-d001b12463c8 ploop_size=261120MB image_size_before=116695MB image_size_after=100782MB compaction_time=7221.741s type=online
2020-07-22T04:00:23+0200 pcompact : End compacting

This node is running a MySQL server which shows errors during these 2 hours (different errors pointing to disk being full). Eventually MySQL crashes.

We'll continue to monitor and report back how it goes. But has anyone experienced such fluctuations in disk utilisation while pcompact is running? Is it somehow expected? How to get around applications failing due to disk showing as full (even when it is actually not). Worth mentioning that all these issues described in this post happen on newly created Ubuntu 18.04 containers (as opposed to my initial post where issues were mostly related to 16.04 containers migrated from OpenVZ 6).

Thanks! Smile

[Updated on: Wed, 22 July 2020 11:50]

Report message to a moderator

[Message index]

		OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Fri, 03 July 2020 15:06
		Re: OpenVZ 7 containers crashing with ext4 errors By: khorenko on Fri, 03 July 2020 16:07
		Re: OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Wed, 22 July 2020 11:45
		Re: OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Thu, 23 July 2020 08:06
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Fri, 24 July 2020 10:56
		Re: OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Fri, 24 July 2020 14:56
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Fri, 24 July 2020 15:27
		Re: OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Wed, 29 July 2020 14:58
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Wed, 29 July 2020 15:24
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Mon, 03 August 2020 10:19
		Re: OpenVZ 7 containers crashing with ext4 errors By: noc.r on Thu, 20 August 2020 12:34
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Thu, 20 August 2020 12:53
		Re: OpenVZ 7 containers crashing with ext4 errors By: noc.r on Thu, 20 August 2020 15:04
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Thu, 20 August 2020 12:54
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Fri, 21 August 2020 11:33
		Re: OpenVZ 7 containers crashing with ext4 errors By: noc.r on Mon, 24 August 2020 10:45
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Mon, 24 August 2020 11:16
		Re: OpenVZ 7 containers crashing with ext4 errors By: noc.r on Mon, 24 August 2020 14:11
		Re: OpenVZ 7 containers crashing with ext4 errors By: eshatokhin on Tue, 25 August 2020 17:50
		Re: OpenVZ 7 containers crashing with ext4 errors By: khorenko on Wed, 26 August 2020 11:46
		Re: OpenVZ 7 containers crashing with ext4 errors By: nathan.brownrice on Mon, 24 August 2020 16:48
		Re: OpenVZ 7 containers crashing with ext4 errors By: khorenko on Wed, 26 August 2020 12:02
		Re: OpenVZ 7 containers crashing with ext4 errors By: noc.r on Wed, 26 August 2020 12:24
		Re: OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Tue, 01 September 2020 11:00
		Re: OpenVZ 7 containers crashing with ext4 errors By: ktkhai on Tue, 01 September 2020 12:04
		Re: OpenVZ 7 containers crashing with ext4 errors By: nathan.brownrice on Sat, 03 October 2020 18:13
		Re: OpenVZ 7 containers crashing with ext4 errors By: eshatokhin on Sun, 04 October 2020 15:33
		Re: OpenVZ 7 containers crashing with ext4 errors By: noc.r on Thu, 03 September 2020 12:37
		Re: OpenVZ 7 containers crashing with ext4 errors By: eshatokhin on Thu, 03 September 2020 13:00
		Re: OpenVZ 7 containers crashing with ext4 errors By: eshatokhin on Fri, 04 September 2020 16:59
		Re: OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Fri, 11 September 2020 14:08
		Re: OpenVZ 7 containers crashing with ext4 errors By: noc.r on Mon, 28 September 2020 14:12
		Re: OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Mon, 05 October 2020 05:56
		Re: OpenVZ 7 containers crashing with ext4 errors By: eshatokhin on Mon, 05 October 2020 10:22
		Re: OpenVZ 7 containers crashing with ext4 errors By: allan.talver on Wed, 14 October 2020 08:02
		Re: OpenVZ 7 containers crashing with ext4 errors By: eshatokhin on Wed, 14 October 2020 08:27

Previous Topic:	disk softlimit exceeded during CT creation
Next Topic:	OpenVZ 7 OOM killing systemd in containers

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Aug 25 13:40:06 GMT 2024

Total time taken to generate the page: 0.03536 seconds