OpenVZ Forum


Home » General » Support » online container migration fails with Remote exception I/O operation on closed file (Trying to lie migrate container using vzmigrate or prlctl, getting Remote exception I/O operation on closed file)
online container migration fails with Remote exception I/O operation on closed file [message #53303] Wed, 23 May 2018 14:33 Go to next message
msolovyev is currently offline  msolovyev
Messages: 29
Registered: August 2007
Location: Russia, Novosibirsk
Junior Member

From: 95.47.244*
Hello,

I'm trying to do live migration of container, and getting such an error (this happens even with empty test container):

[root@vz03 ~]# vzmigrate -vvv --online --require-realtime vz04 222
...
2018-05-23 10:30:10.732: Live migration stage started
2018-05-23 10:30:36.320: Io multiplexer aborted
2018-05-23 10:30:36.320: 2018-05-23 10:30:36.321: Phaul service failed to live migrate CT
2018-05-23 10:30:36.320: 2018-05-23 10:30:36.321: error [-73] : Phaul service failed to live migrate CT
2018-05-23 10:30:36.321: Phaul service failed to live migrate CT
2018-05-23 10:30:36.321: Phaul failed to live migrate CT (/var/log/phaul.log)
2018-05-23 10:30:36.322: 2018-05-23 10:30:36.322: cleaning : destroy CT 222
2018-05-23 10:30:36.372: 2018-05-23 10:30:36.372: cleaning : 'rm' dir : /vz/private/222
2018-05-23 10:30:36.372: 2018-05-23 10:30:36.372: can not rename : [/vz/private/222] -> [/vz/private/222.ss6sKg]
2018-05-23 10:30:36.372: 2018-05-23 10:30:36.373: cleaning : 'rmdir' dir : /vz/root/222
2018-05-23 10:30:36.372: 2018-05-23 10:30:36.373: can not find entry for delete : [/vz/root/222]
2018-05-23 10:30:37.373: 2018-05-23 10:30:37.373: unlocking 222
2018-05-23 10:30:37.375: Can't move/copy CT 222 -> CT 222, [], [] : Phaul failed to live migrate CT (/var/log/phaul.log)
2018-05-23 10:30:37.375: unlocking 222
2018-05-23 10:30:37.375: close channel


[root@vz03 ~]# tail -20 /var/log/phaul.log
10:30:33.214: 285170:           Notify (post-network-lock)
10:30:35.283: 285170: Final FS and images sync
10:30:35.522: 285170: Sending images to target
10:30:35.524: 285170:   Pack
10:30:35.561: 285170:   Add htype images
10:30:35.812: 285170: Asking target host to restore
10:30:36.271: 285170: Remote exception
10:30:36.271: 285170: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/libexec/phaul/p.haul", line 9, in <module>
    load_entry_point('phaul==0.1', 'console_scripts', 'p.haul')()
  File "/usr/lib/python2.7/site-packages/phaul/shell/phaul_client.py", line 49, in main
    worker.start_migration()
  File "/usr/lib/python2.7/site-packages/phaul/iters.py", line 161, in start_migration
    self.__start_live_migration()
  File "/usr/lib/python2.7/site-packages/phaul/iters.py", line 232, in __start_live_migration
    self.target_host.restore_from_images()
  File "/usr/lib/python2.7/site-packages/phaul/xem_rpc_client.py", line 26, in __call__
    raise Exception(resp[1])
Exception: I/O operation on closed file



Logs from destination server:

[root@vz04 ~]# tail -20 /var/log/phaul-service.log
10:30:35.562: 817892: Waiting for images to unpack
10:30:35.813: 817892: Restoring from images
10:30:35.827: 817892: Starting vzctl restore
10:30:36.269: 817892:   > Restoring the Container ...
10:30:36.269: 817892:   > Mount image: /vz/private/222/root.hdd 
10:30:36.269: 817892:   > Container is mounted
10:30:36.269: 817892:   > Setting permissions for image=/vz/private/222/root.hdd
10:30:36.269: 817892:   > (00.000283) Error (criu/util.c:694): Can't read link of fd -404: No such file or directory
10:30:36.270: 817892:   > (00.000295) Error (criu/protobuf.c:77): Unexpected EOF on (null)
10:30:36.270: 817892:   > The restore log was saved in /vz/dump/222/rst-_cQGWZ-18.05.23-10.30/criu_restore.9.log
10:30:36.270: 817892:   > criu exited with rc=17
10:30:36.270: 817892:   > Unmount image: /vz/private/222/root.hdd
10:30:36.270: 817892:   > Container is unmounted
10:30:36.270: 817892:   > Failed to restore the Container
10:30:36.321: 817892: Disconnected
10:30:36.322: 817892: Closing images
10:30:36.322: 817892: Removing images
10:30:36.373: 817892: Stop by 15
10:30:36.373: 817892: RPC Service stops
10:30:36.374: 817892: Bye!


[root@vz04 ~]# tail -20 /vz/dump/222/rst-_cQGWZ-18.05.23-10.30/criu_restore.9.log
(00.000142) Version: 3.8 (gitid 0)
(00.000188) Running on vz04.boardreader.com Linux 3.10.0-693.21.1.vz7.47.4 #1 SMP Sat Apr 28 11:48:07 MSK 2018 x86_64
(00.000237) No inventory.img image
(00.000283) Error (criu/util.c:694): Can't read link of fd -404: No such file or directory
(00.000295) Error (criu/protobuf.c:77): Unexpected EOF on (null)




Both servers run the following openvz version:

[root@vz03 ~]# uname -a
Linux vz03.boardreader.com 3.10.0-693.21.1.vz7.47.4 #1 SMP Sat Apr 28 11:48:07 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux


[root@vz03 ~]# cat /etc/*release*
OpenVZ release 7.0.8 (142)
NAME="Virtuozzo"
VERSION="7.0.8"
ID="virtuozzo"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="OpenVZ release 7.0.8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:virtuozzoproject:vz:7"
HOME_URL="http://www.virtuozzo.com"
BUG_REPORT_URL="https://bugs.openvz.org/"
OpenVZ release 7.0.8 (142)
Virtuozzo Linux release 7.4
OpenVZ release 7.0.8 (142)
cpe:/o:virtuozzoproject:vzlinux:7:ga
OpenVZ release 7.0.8 (142)
Virtuozzo Linux release 7.5.0 (549)


If I remove --online --require-realtime options, it works.
Re: online container migration fails with Remote exception I/O operation on closed file [message #53304 is a reply to message #53303] Thu, 24 May 2018 12:00 Go to previous messageGo to next message
vaverin is currently offline  vaverin
Messages: 684
Registered: September 2005
Senior Member
From: *virtuozzo.com
submitted to
https://github.com/checkpoint-restore/criu/issues/494
Re: online container migration fails with Remote exception I/O operation on closed file [message #53307 is a reply to message #53304] Fri, 25 May 2018 07:17 Go to previous messageGo to next message
vaverin is currently offline  vaverin
Messages: 684
Registered: September 2005
Senior Member
From: *virtuozzo.com
avagin on github asks to provide criu_restore.9.log file
Re: online container migration fails with Remote exception I/O operation on closed file [message #53308 is a reply to message #53303] Fri, 25 May 2018 11:38 Go to previous messageGo to next message
msolovyev is currently offline  msolovyev
Messages: 29
Registered: August 2007
Location: Russia, Novosibirsk
Junior Member

From: 95.47.244*
Actually criu_restore.9.log was already posted in my first message:

[root@vz04 ~]# cat /vz/dump/222/rst-_cQGWZ-18.05.23-10.30/criu_restore.9.log
(00.000142) Version: 3.8 (gitid 0)
(00.000188) Running on vz04.boardreader.com Linux 3.10.0-693.21.1.vz7.47.4 #1 SMP Sat Apr 28 11:48:07 MSK 2018 x86_64
(00.000237) No inventory.img image
(00.000283) Error (criu/util.c:694): Can't read link of fd -404: No such file or directory
(00.000295) Error (criu/protobuf.c:77): Unexpected EOF on (null)



I attached the whole /vz/dump/222/rst-_cQGWZ-18.05.23-10.30 dir on github


BTW, can't attach archive here, getting "File Attachment is too big (over allowed limit of 2097152 bytes)", while archive is 200KB.
Re: online container migration fails with Remote exception I/O operation on closed file [message #53309 is a reply to message #53308] Fri, 25 May 2018 12:59 Go to previous messageGo to next message
vaverin is currently offline  vaverin
Messages: 684
Registered: September 2005
Senior Member
From: *virtuozzo.com
reported as https://bugs.openvz.org/browse/OVZ-7030
Re: online container migration fails with Remote exception I/O operation on closed file [message #53493 is a reply to message #53303] Fri, 22 March 2019 18:46 Go to previous messageGo to next message
alenco is currently offline  alenco
Messages: 10
Registered: August 2014
Junior Member
From: *xs4all.nl
When will this be integrated in the stable package of Virtuozzo, or even at the factory repo? The last update on the crui package is from 2017.
Having the same issue as described here.

Or would it be OK to compile the package myself? Not sure if this will create more issues?



Re: online container migration fails with Remote exception I/O operation on closed file [message #53494 is a reply to message #53493] Fri, 22 March 2019 19:24 Go to previous message
alenco is currently offline  alenco
Messages: 10
Registered: August 2014
Junior Member
From: *xs4all.nl
Just tried to compile it to 3.10 and 3.11, but getting an error on that. Probably because p.haul is not working any more with different version than their own package.

Really much appreciated if someone can let me know if there is a way to get this working.

Live migration is simply not working anymore.
Previous Topic: factory repo, should it be used or not?
Next Topic: Bug reports should go to bugs.openvz.org
Goto Forum:
  


Current Time: Sat Mar 23 21:23:12 GMT 2019