OpenVZ Forum


Home » General » Support » online container migration fails with Remote exception I/O operation on closed file (Trying to lie migrate container using vzmigrate or prlctl, getting Remote exception I/O operation on closed file)
online container migration fails with Remote exception I/O operation on closed file [message #53303] Wed, 23 May 2018 14:33 Go to next message
msolovyev is currently offline  msolovyev
Messages: 29
Registered: August 2007
Location: Russia, Novosibirsk
Junior Member

From: 95.47.244*
Hello,

I'm trying to do live migration of container, and getting such an error (this happens even with empty test container):

[root@vz03 ~]# vzmigrate -vvv --online --require-realtime vz04 222
...
2018-05-23 10:30:10.732: Live migration stage started
2018-05-23 10:30:36.320: Io multiplexer aborted
2018-05-23 10:30:36.320: 2018-05-23 10:30:36.321: Phaul service failed to live migrate CT
2018-05-23 10:30:36.320: 2018-05-23 10:30:36.321: error [-73] : Phaul service failed to live migrate CT
2018-05-23 10:30:36.321: Phaul service failed to live migrate CT
2018-05-23 10:30:36.321: Phaul failed to live migrate CT (/var/log/phaul.log)
2018-05-23 10:30:36.322: 2018-05-23 10:30:36.322: cleaning : destroy CT 222
2018-05-23 10:30:36.372: 2018-05-23 10:30:36.372: cleaning : 'rm' dir : /vz/private/222
2018-05-23 10:30:36.372: 2018-05-23 10:30:36.372: can not rename : [/vz/private/222] -> [/vz/private/222.ss6sKg]
2018-05-23 10:30:36.372: 2018-05-23 10:30:36.373: cleaning : 'rmdir' dir : /vz/root/222
2018-05-23 10:30:36.372: 2018-05-23 10:30:36.373: can not find entry for delete : [/vz/root/222]
2018-05-23 10:30:37.373: 2018-05-23 10:30:37.373: unlocking 222
2018-05-23 10:30:37.375: Can't move/copy CT 222 -> CT 222, [], [] : Phaul failed to live migrate CT (/var/log/phaul.log)
2018-05-23 10:30:37.375: unlocking 222
2018-05-23 10:30:37.375: close channel


[root@vz03 ~]# tail -20 /var/log/phaul.log
10:30:33.214: 285170:           Notify (post-network-lock)
10:30:35.283: 285170: Final FS and images sync
10:30:35.522: 285170: Sending images to target
10:30:35.524: 285170:   Pack
10:30:35.561: 285170:   Add htype images
10:30:35.812: 285170: Asking target host to restore
10:30:36.271: 285170: Remote exception
10:30:36.271: 285170: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/libexec/phaul/p.haul", line 9, in <module>
    load_entry_point('phaul==0.1', 'console_scripts', 'p.haul')()
  File "/usr/lib/python2.7/site-packages/phaul/shell/phaul_client.py", line 49, in main
    worker.start_migration()
  File "/usr/lib/python2.7/site-packages/phaul/iters.py", line 161, in start_migration
    self.__start_live_migration()
  File "/usr/lib/python2.7/site-packages/phaul/iters.py", line 232, in __start_live_migration
    self.target_host.restore_from_images()
  File "/usr/lib/python2.7/site-packages/phaul/xem_rpc_client.py", line 26, in __call__
    raise Exception(resp[1])
Exception: I/O operation on closed file



Logs from destination server:

[root@vz04 ~]# tail -20 /var/log/phaul-service.log
10:30:35.562: 817892: Waiting for images to unpack
10:30:35.813: 817892: Restoring from images
10:30:35.827: 817892: Starting vzctl restore
10:30:36.269: 817892:   > Restoring the Container ...
10:30:36.269: 817892:   > Mount image: /vz/private/222/root.hdd 
10:30:36.269: 817892:   > Container is mounted
10:30:36.269: 817892:   > Setting permissions for image=/vz/private/222/root.hdd
10:30:36.269: 817892:   > (00.000283) Error (criu/util.c:694): Can't read link of fd -404: No such file or directory
10:30:36.270: 817892:   > (00.000295) Error (criu/protobuf.c:77): Unexpected EOF on (null)
10:30:36.270: 817892:   > The restore log was saved in /vz/dump/222/rst-_cQGWZ-18.05.23-10.30/criu_restore.9.log
10:30:36.270: 817892:   > criu exited with rc=17
10:30:36.270: 817892:   > Unmount image: /vz/private/222/root.hdd
10:30:36.270: 817892:   > Container is unmounted
10:30:36.270: 817892:   > Failed to restore the Container
10:30:36.321: 817892: Disconnected
10:30:36.322: 817892: Closing images
10:30:36.322: 817892: Removing images
10:30:36.373: 817892: Stop by 15
10:30:36.373: 817892: RPC Service stops
10:30:36.374: 817892: Bye!


[root@vz04 ~]# tail -20 /vz/dump/222/rst-_cQGWZ-18.05.23-10.30/criu_restore.9.log
(00.000142) Version: 3.8 (gitid 0)
(00.000188) Running on vz04.boardreader.com Linux 3.10.0-693.21.1.vz7.47.4 #1 SMP Sat Apr 28 11:48:07 MSK 2018 x86_64
(00.000237) No inventory.img image
(00.000283) Error (criu/util.c:694): Can't read link of fd -404: No such file or directory
(00.000295) Error (criu/protobuf.c:77): Unexpected EOF on (null)




Both servers run the following openvz version:

[root@vz03 ~]# uname -a
Linux vz03.boardreader.com 3.10.0-693.21.1.vz7.47.4 #1 SMP Sat Apr 28 11:48:07 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux


[root@vz03 ~]# cat /etc/*release*
OpenVZ release 7.0.8 (142)
NAME="Virtuozzo"
VERSION="7.0.8"
ID="virtuozzo"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="OpenVZ release 7.0.8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:virtuozzoproject:vz:7"
HOME_URL="http://www.virtuozzo.com"
BUG_REPORT_URL="https://bugs.openvz.org/"
OpenVZ release 7.0.8 (142)
Virtuozzo Linux release 7.4
OpenVZ release 7.0.8 (142)
cpe:/o:virtuozzoproject:vzlinux:7:ga
OpenVZ release 7.0.8 (142)
Virtuozzo Linux release 7.5.0 (549)


If I remove --online --require-realtime options, it works.
Re: online container migration fails with Remote exception I/O operation on closed file [message #53304 is a reply to message #53303] Thu, 24 May 2018 12:00 Go to previous messageGo to next message
vaverin is currently offline  vaverin
Messages: 682
Registered: September 2005
Senior Member
From: *virtuozzo.com
submitted to
https://github.com/checkpoint-restore/criu/issues/494
Re: online container migration fails with Remote exception I/O operation on closed file [message #53307 is a reply to message #53304] Fri, 25 May 2018 07:17 Go to previous messageGo to next message
vaverin is currently offline  vaverin
Messages: 682
Registered: September 2005
Senior Member
From: *virtuozzo.com
avagin on github asks to provide criu_restore.9.log file
Re: online container migration fails with Remote exception I/O operation on closed file [message #53308 is a reply to message #53303] Fri, 25 May 2018 11:38 Go to previous messageGo to next message
msolovyev is currently offline  msolovyev
Messages: 29
Registered: August 2007
Location: Russia, Novosibirsk
Junior Member

From: 95.47.244*
Actually criu_restore.9.log was already posted in my first message:

[root@vz04 ~]# cat /vz/dump/222/rst-_cQGWZ-18.05.23-10.30/criu_restore.9.log
(00.000142) Version: 3.8 (gitid 0)
(00.000188) Running on vz04.boardreader.com Linux 3.10.0-693.21.1.vz7.47.4 #1 SMP Sat Apr 28 11:48:07 MSK 2018 x86_64
(00.000237) No inventory.img image
(00.000283) Error (criu/util.c:694): Can't read link of fd -404: No such file or directory
(00.000295) Error (criu/protobuf.c:77): Unexpected EOF on (null)



I attached the whole /vz/dump/222/rst-_cQGWZ-18.05.23-10.30 dir on github


BTW, can't attach archive here, getting "File Attachment is too big (over allowed limit of 2097152 bytes)", while archive is 200KB.
Re: online container migration fails with Remote exception I/O operation on closed file [message #53309 is a reply to message #53308] Fri, 25 May 2018 12:59 Go to previous message
vaverin is currently offline  vaverin
Messages: 682
Registered: September 2005
Senior Member
From: *virtuozzo.com
reported as https://bugs.openvz.org/browse/OVZ-7030
Previous Topic: OpenVZ Maintenance Partnership program
Next Topic: Error after kernel update.
Goto Forum:
  


Current Time: Fri Oct 19 21:40:05 GMT 2018