OpenVZ Forum: Support » "VE already locked" after unsuccessful migration

Home » General » Support » "VE already locked" after unsuccessful migration

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

"VE already locked" after unsuccessful migration [message #11379]

Thu, 22 March 2007 10:44

ataraxis
Messages: 19
Registered: August 2006

Junior Member

Hi,

I wanted to move a VE from HN1 to HN2 because HN1 should get a kernel update.

Both HNs run 2.6.18-028test007 with a custom config.
vzctl version is 3.0.14-1dso31.2

Here is the output from the migration:
HN1:~# vzmigrate -v --online HN2 231
OPT:-v
OPT:--online
OPT:HN2
Starting online migration of VE 231 on HN2
OpenVZ is running...
Loading /etc/vz/vz.conf and /etc/vz/conf/231.conf files
Check IPs on destination node: <IP>
Preparing remote node
Copying config file
231.conf 100% 1150 1.1KB/s 00:00
Name jira assigned
Saved parameters for VE 231
Creating remote VE root dir
Creating remote VE private dir
Initializing remote quota
Quota init
Turning remote quota on
Syncing private
100%
Live migrating VE
Suspending VE
Setting up checkpoint...
suspend...
get context...
Checkpointing completed succesfully
Dumping VE
Setting up checkpoint...
join context..
dump...
Checkpointing completed succesfully
Copying dumpfile
dump.231 100% 493MB 32.9MB/s 00:15
Syncing private (2nd pass)
Syncing 2nd level quota
Dumping 2nd level quota
Copying 2nd level quota
quotadump.231 100% 21 0.0KB/s 00:00
Load 2nd level quota
Undumping VE
Restoring VE ...
Starting VE ...
vzquota : (warning) Quota is running for id 231 already
VE is mounted
undump...
Adding IP address(es): <IP>
Setting CPU units: 200033

----------------------------------------
Here i hit ctrl+c after a few minutes, because the application should not be offline that much and previous migrations have been faster although the VEs were bigger
----------------------------------------

Killed by signal 2.
Error: Failed to undump VE
Resuming...
arpsend: <IP> is detected on another computer : <MAC>
vps-net_add WARNING: arpsend -c 1 -w 1 -D -e <IP> eth0 FAILED
vzquota : (error) Quota off syscall for id 231: Device or resource busy

vzquota : (warning) Quota is running for id 231

This is from vz.log on HN1:
2007-03-22T11:16:25+0100 vzctl : VE 231 : Setting up checkpoint...
2007-03-22T11:16:25+0100 vzctl : VE 231 : suspend...
2007-03-22T11:16:25+0100 vzctl : VE 231 : get context...
2007-03-22T11:16:25+0100 vzctl : VE 231 : Checkpointing completed succesfully
2007-03-22T11:16:25+0100 vzctl : VE 231 : Setting up checkpoint...
2007-03-22T11:16:25+0100 vzctl : VE 231 : join context..
2007-03-22T11:16:25+0100 vzctl : VE 231 : dump...
2007-03-22T11:16:29+0100 vzctl : VE 231 : Checkpointing completed succesfully
2007-03-22T11:20:53+0100 vzctl : VE 231 : Resuming...

This is from vz.log on HN2:
2007-03-22T11:07:15+0100 vzctl : VE 231 : Name jira assigned
2007-03-22T11:07:15+0100 vzctl : VE 231 : Saved parameters for VE 231
2007-03-22T11:14:27+0100 vzctl : VE 231 : Restoring VE ...
2007-03-22T11:14:27+0100 vzctl : VE 231 : Starting VE ...
2007-03-22T11:14:27+0100 vzctl : VE 231 : VE is mounted
2007-03-22T11:14:28+0100 vzctl : VE 231 : Adding IP address(es): 213.214.12.231
2007-03-22T11:14:28+0100 vzctl : VE 231 : Setting CPU units: 200033
2007-03-22T11:15:01+0100 vps-bandwidth : Statsfile for VPS 231 created.
2007-03-22T11:18:55+0100 vzctl : VE 231 : VE config file does not exist
2007-03-22T11:19:08+0100 vzctl : VE 231 : VE config file does not exist
2007-03-22T11:20:06+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:20:09+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:20:14+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:20:52+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:21:30+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:21:42+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:22:03+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:25:26+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:25:30+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:28:51+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:28:54+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:30:13+0100 vzctl : VE 231 : VE already locked
2007-03-22T11:31:09+0100 vzctl : VE 231 : VE start in progress...
2007-03-22T11:31:09+0100 vzctl : VE 231 : Restoring completed succesfully

Now I can't get rid of 231 on HN2
it always says "VE already locked"

Any Ideas? Thanks in advance.

Kai

Report message to a moderator

Re: "VE already locked" after unsuccessful migration [message #11388 is a reply to message #11379]

Thu, 22 March 2007 14:22

Andrey Mirkin
Messages: 193
Registered: May 2006

Senior Member

You are using old version of 2.6.18 kernel, try to update kernel on destination node.

Also please provide us following information from destination node:

cat /proc/vz/veinfo
cat /proc/vz/veip
ls $LOCKDIR -l

where $LOCKDIR is variable from global openvz config (/etc/vz/vz.conf)

If while migration vzctl will hang again then please do the following on destination node:

echo t > /proc/sysrq-trigger

and post here kernel logs.

Andrey Mirkin

http://static.openvz.org/userbars/openvz-developer.png

Report message to a moderator

Re: "VE already locked" after unsuccessful migration [message #11397 is a reply to message #11388]

Thu, 22 March 2007 15:44

ataraxis
Messages: 19
Registered: August 2006

Junior Member

HN2:~# cat /proc/vz/veinfo
245 0 137 x.y.z.245
232 0 154 x.y.z.232
233 0 40 x.y.z.233
239 0 177 x.y.z.239
247 0 106 x.y.z.247
248 0 201 x.y.z.248
246 0 125 x.y.z.246
0 0 136

HN2:~# cat /proc/vz/veip
Version: 2.5
x.y.z.232 0
x.y.z.233 0
x.y.z.239 0
x.y.z.245 0
x.y.z.246 0
x.y.z.247 0
x.y.z.248 0

HN2:~# grep -i lock /etc/vz/vz.conf
LOCKDIR=/var/lib/vz/lock
HN2:~# ls /var/lib/vz/lock/ -l
insgesamt 0

After lunch I was suddenly able to delete the VE.
I tried to move the VEs then to a third server, it worked flawless there.

Here are infos about the third one:
HN3:~# cat /proc/vz/veinfo
241 2 168 x.y.z.241
231 2 89 x.y.z.231
240 0 178 x.y.z.240
104 0 241 x.y.z.237
0 0 125

HN3:~# cat /proc/vz/veip
Version: 2.5
x.y.z.231 0
x.y.z.237 0
x.y.z.240 0
x.y.z.241 0

I then updated HN1 to 2.6.18-028stab021 and then moving a VE to HN2 worked flawless.

Report message to a moderator

Re: "VE already locked" after unsuccessful migration [message #11401 is a reply to message #11397]

Thu, 22 March 2007 16:38

Andrey Mirkin
Messages: 193
Registered: May 2006

Senior Member

Well, 2.6.18-028test007 is old and buggy kernel and many buges were fixed since that time, but maybe you have triggered a new one.
So, please check for errors /var/log/messages on HN2, and if you find there something strange, then please post here.

Andrey Mirkin

Report message to a moderator

Re: "VE already locked" after unsuccessful migration [message #11402 is a reply to message #11401]

Thu, 22 March 2007 16:49

ataraxis
Messages: 19
Registered: August 2006

Junior Member

This is in /var/log/messages :

Mar 22 11:05:11 HN2 -- MARK --
Mar 22 11:14:28 HN2 kernel: VPS: 231: started
Mar 22 11:14:29 HN2 kernel: CPT ERR: f54eb800,231 :rst_file: -19 88760
Mar 22 11:14:29 HN2 kernel: CPT ERR: f54eb800,231 :rst_files: -19
Mar 22 11:14:29 HN2 kernel: CPT ERR: f54eb800,231 :make_baby: -19
Mar 22 11:14:29 HN2 kernel: CPT ERR: f54eb800,231 :rst_clone_children
Mar 22 11:25:13 HN2 -- MARK --
Mar 22 11:31:10 HN2 kernel: VPS: 231: stopped
Mar 22 11:45:13 HN2 -- MARK --

I later tried it with a different VE with the same results:
Mar 22 13:05:16 HN2 -- MARK --
Mar 22 13:24:03 HN2 kernel: VPS: 241: started
Mar 22 13:24:03 HN2 kernel: CPT ERR: f19cb400,241 :rst_file: -19 142096
Mar 22 13:24:03 HN2 kernel: CPT ERR: f19cb400,241 :rst_files: -19
Mar 22 13:24:03 HN2 kernel: CPT ERR: f19cb400,241 :make_baby: -19
Mar 22 13:24:03 HN2 kernel: CPT ERR: f19cb400,241 :rst_clone_children
Mar 22 13:45:17 HN2 -- MARK --
Mar 22 13:57:24 HN2 kernel: VPS: 241: stopped
Mar 22 14:25:19 HN2 -- MARK --

I have noticed that the load average went sky high during that time:
50 in the first case, 80 in the second

Report message to a moderator

Re: "VE already locked" after unsuccessful migration [message #11403 is a reply to message #11402]

Thu, 22 March 2007 17:13

ataraxis
Messages: 19
Registered: August 2006

Junior Member

now I tried to move it back to HN1 from HN3, same problem :/
(HN3 is 2.6.16-026test017, HN1 is 2.6.18-028stab021)

Quote:

HN1:~# cat /proc/vz/veinfo
231 2 4 x.y.z.231
250 0 73 x.y.z.250
230 0 22 x.y.z.230
249 0 195 x.y.z.249
229 0 76 x.y.z.229
0 0 109

Quote:

HN1:~# cat /proc/vz/veip
Version: 2.5
x.y.z.229 0
x.y.z.230 0
x.y.z.231 0
x.y.z.249 0
x.y.z.250 0

Quote:

HN1:~# grep LOCKDIR /etc/vz/vz.conf
LOCKDIR=/var/lib/vz/lock
HN1:~# ls -lh /var/lib/vz/lock/
insgesamt 4,0K
-rw------- 1 root root 5 2007-03-22 18:03 231.lck
HN1:~# ls -l /var/lib/vz/lock/
insgesamt 4
-rw------- 1 root root 5 2007-03-22 18:03 231.lck

Quote:

HN1:~# echo t > /proc/sysrq-trigger; tail -f /var/log/messages
-bash: /proc/sysrq-trigger: Datei oder Verzeichnis nicht gefunden
Mar 22 16:59:01 HN1 -- MARK --
Mar 22 17:19:02 HN1 -- MARK --
Mar 22 17:39:02 HN1 -- MARK --
Mar 22 17:59:03 HN1 -- MARK --
Mar 22 18:03:09 HN1 kernel: VE: 231: started
Mar 22 18:03:09 HN1 kernel: CPT ERR: dbe2e000,231 :rst_file: -19 127560
Mar 22 18:03:09 m01host kernel: CPT ERR: dbe2e000,231 :rst_files: -19
Mar 22 18:03:09 m01host kernel: CPT ERR: dbe2e000,231 :make_baby: -19
Mar 22 18:03:09 m01host kernel: CPT ERR: dbe2e000,231 :rst_clone_children

Quote:

HN3:~# vzmigrate -v --online m01host 231
OPT:-v
OPT:--online
OPT:HN1
Starting online migration of VE 231 on HN1
OpenVZ is running...
Loading /etc/vz/vz.conf and /etc/vz/conf/231.conf files
Check IPs on destination node: x.y.z.231
Preparing remote node
Copying config file
231.conf 100% 1150 1.1KB/s 00:00
Saved parameters for VE 231
Creating remote VE root dir
Creating remote VE private dir
Initializing remote quota
Quota init
Turning remote quota on
Syncing private
Live migrating VE
Suspending VE
Setting up checkpoint...
suspend...
get context...
Checkpointing completed succesfully
Dumping VE
Setting up checkpoint...
join context..
dump...
Checkpointing completed succesfully
Copying dumpfile
dump.231 100% 508MB 31.8MB/s 00:16
Syncing private (2nd pass)
Syncing 2nd level quota
Dumping 2nd level quota
Copying 2nd level quota
quotadump.231 100% 21 0.0KB/s 00:00
Load 2nd level quota
Undumping VE
Restoring VE ...
Starting VE ...
VE is mounted
undump...
vzquota : (warning) Quota is running for id 231 already
Adding IP address(es): x.y.z.231
Setting CPU units: 200033
------------------------------------------------------------
i hit CTRL+C after a while again
------------------------------------------------------------
Killed by signal 2.
Error: Failed to undump VE
Resuming...
arpsend: x.y.z.231 is detected on another computer : <MAC>
vps-net_add WARNING: arpsend -c 1 -w 1 -D -e x.y.z.231 eth0 FAILED
vzquota : (error) Quota off syscall for id 231: Device or resource busy
vzquota : (warning) Quota is running for id 231

how do i get /proc/sysrq-trigger?

Report message to a moderator