| Openvz7 ploop FS + snapshots = BLOAT [message #53848] |
Fri, 26 April 2024 16:27  |
nathan.brownrice
Messages: 12
Registered: August 2020
|
Junior Member |
|
|
I'll keep this one short and sweet, and this is a long-shot, but if you know of a solution to this you probably already know what I'm talking about. We've been working through this ourselves with our production machines, and here's an example of the issue:
A VPS container, df -h inside the container shows it's using ~15G:
vzctl exec 54d7ac3a-3667-44b2-a4c9-67aa394284f0 df -h
Filesystem Size Used Avail Use% Mounted on
/dev/ploop16022p1 59G 15G 42G 26% /
none 4.0M 0 4.0M 0% /sys/fs/cgroup
none 3.0G 0 3.0G 0% /dev
tmpfs 3.0G 0 3.0G 0% /dev/shm
tmpfs 1.3G 124K 1.2G 1% /run
Same container, in the /vz/root (mounted filesystem) path on the host:
du -sh /vz/root/54d7ac3a-3667-44b2-a4c9-67aa394284f0/
15G /vz/root/54d7ac3a-3667-44b2-a4c9-67aa394284f0/
Same container, in the /vz/private path:
du -sh /vz/private/54d7ac3a-3667-44b2-a4c9-67aa394284f0/
86G /vz/private/54d7ac3a-3667-44b2-a4c9-67aa394284f0/
Most of it's stuck in the /vz/private/54d7ac3a-3667-44b2-a4c9-67aa394284f0/root.hdd images:
du -sh *
4.0K DiskDescriptor.xml
0 DiskDescriptor.xml.lck
52G root.hds
29G root.hds.{4ef2408b-64ea-44d8-b8ca-d0b4b40145c9}
1.8G root.hds.{c472b739-1cc5-484c-aca9-ced7bed74699}
24K templates
We are running snapshots, and need to run snapshots. However after much experimentation it looks like any "bloat" is locked up in these snapshots and can never be cleared. Running pcompact does not free up the locked disk space and gives output saying to remove the snapshots for better compacting:
Inspect 54d7ac3a-3667-44b2-a4c9-67aa394284f0
Inspect /vz/private/54d7ac3a-3667-44b2-a4c9-67aa394284f0/root.hdd/DiskDescriptor.xml
This ploop image contains snapshots. Trying to compact the last delta file.
For best compacting results, remove snapshots.
ploop=21118MB image=511MB data=7437MB balloon=19842MB
Rate: 0.0 (threshold=10)
Start compacting
ploop=21118MB image=511MB data=7437MB balloon=19842MB
End compacting
I understand the snapshots take up extra space, but what happens here is bloat over time. Files are added, snapshotted, the base root.hdd expands, but as long as snapshots are in place it will never contract even when files are removed and the running containers disk use is shrunk. Even with the nightly pcompact cron running. We've had quite a few containers that are using an actual ~10GB, with /vz/private dirs that bloat to 100+GB.
We've experimented with ensuring pcompact runs first, before any snapshot creation, but still the bloat gets locked in. The only way to free it is to delete all snapshots, then run pcompact, which shrinks the base root.hdd file down to close to what the container is actually using. Then as months go by things just slowly bloat back out.
Deleting snapshots every few months is not really a solution, so if anyone has run into this, or come up with a solution, I'm all ears!
[Updated on: Fri, 26 April 2024 16:30] Report message to a moderator |
|
|
|
|
|
| Re: Openvz7 ploop FS + snapshots = BLOAT [message #53854 is a reply to message #53849] |
Fri, 17 May 2024 16:22  |
nathan.brownrice
Messages: 12
Registered: August 2020
|
Junior Member |
|
|
Hey, thanks for replying!
I've tried that exact command, it's no different from running pcompact as far as I can tell. I've tried running it directly on the root.hdd snapshot files as well, and it does nothing. With the snapshots in place, the disk is "locked" into the base root.hdd file. Only if I delete all of the snapshots first can I free up the wasted disk space in the root.hdd file by running pcompact.
They are live snapshots, and to be honest I haven't tested with offline snapshots. Maybe worth a shot, but we have so many live containers we can't have downtime on them even overnight it would be sloppy.
|
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
