openvz6 ploop backups and mysql innodb corruption
Hello, We are suffering from a critical issue with ploop backups. Basically, in some instances, when restoring a ploop backup, MySQL fails to start because of Innodb corruption, example: 2018-03-12 18:04:35 7fd3d4864900 InnoDB: Error: page 818 log sequence number 71622558711 InnoDB: is in the future! Current system log sequence number 71620133300. InnoDB: Your database may be corrupt or you may have copied the InnoDB InnoDB: tablespace but not the InnoDB log files. See InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recove ry.html InnoDB: for more information. We use the following method to perform the backups: https://openvz.org/Ploop/Backup#Image-based_backup What would be the proper way to perform the backups to ensure we do not run into this problem? We fear if we ever had fatal issue with the node where all containers needed to be restored we would be stuck with many containers with innodb corruption.
Re: openvz6 ploop backups and mysql innodb corruption
Follow up question on this and sorry if this is a dumb question but.. In https://openvz.org/Ploop/Backup#Image-based_backup vzctl snapshot $CTID --id $ID --skip-suspend --skip-config cp $VE_PRIVATE/root.hdd/* $BACKUPPATH/ Should we be including the snapshot in the backup itself? Wouldn't copying the snapshot be copying where any new data is taking place which is possibly why we are seeing the corruption? Maybe I'm not understand the snapshot but if we instead did.. vzctl snapshot $CTID --id $ID --skip-suspend --skip-config tar -cvf backup.tar --exclude='./dump' --exclude='./root.hdd/root.hdd.*' $VE_PRIVATE/root.hdd/ then deleted the snapshot. Wouldn't this method 'freeze' root.hdd basically ensuring no data will change within it while we backup the image to ensure the db integrity?
Yes, this is how I understand it to work as well. Presumably you would either need to 1) exclude the snapshot differentials like you've shown, OR 2) backup and restore all the files as they show, but then upon restore, also snapshot-switch it to the pre-snapshot-taken state. I'm afraid I haven't tested restoring like this yet, but will be quite soon.
This has now been tested and I can confirm that: 1. If you copy the ploop drive without first snapshotting, it *will* be marked dirty and refuse to mount 2. If you snapshot prior to copying, the backup/dupe *will* be mountable and data is fully intact and restorable 3. You can simply ignore the snapshot files (granted skipping them during backup will mean bandwidth and storage savings). You can simply mount the ploop disk file directly (ignoring the snapshots) or to recover the whole container, edit the XML file and remove the snapshot references.