OpenVZ Forum - RDF feed
type: channel
items: [PATCH 0/4] - v2 - Object creation with a specified id
items: [PATCH 4/4] - v2 - PID: use the target ID specified in procfs
items: Re: [PATCH 0/4] - v2 - Object creation with a specified id
items: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
items: Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)

[PATCH 0/4] - v2 - Object creation with a specified id
type: item
description: When restarting a process that has been previously checkpointed, that process should keep on using some of its ids (such as its process id, or sysV ipc ids). This patch provides a feature that can help ensuring this saved state reuse: it makes it possible to create an object with a pre-defined id. A first implementation had been proposed 2 months ago. It consisted in changing an object's id after it had been created. Here is a second implementation based on Oren Ladaan's idea: Oren's suggestion was to force an object's id during its creation, rather than 1. create it, 2. change its id. A new file is created in procfs: /proc/self/task/<my_tid>/next_id. This makes it possible to avoid races between several threads belonging to the same process. When this file is filled with an id value, a structure pointed to by the calling task_struct is filled with that id. Then, when an object supporting this feature is created, the id present in that new structure is used, instead of the default one. The syntax is one of: . echo "LONG XX" > /proc/self/task/<my_tid>/next_id next object to be created will have an id set to XX . echo "LONG<n> X0 ... X<n-1>" > /proc/self/task/<my_tid>/next_id next object to be created will have its ids set to XX0, ... X<n-1> This is particularly useful for processes that may have several ids if they belong to nested namespaces. The objects covered here are ipc objects and processes. Today, the ids are specified as long, but having a type string specified in the next_id file makes it possible to cover more types in the future, if needed. The patches are against 2.6.25-rc3-mm2, in the following order: [PATCH 1/4] adds the procfs facility for next object to be created, this object being associated to a single id. [PATCH 2/4] enhances the procfs facility for objects associated to multiple ids (like processes). [PATCH 3/4] makes use of the specified id (if any) to allocate the new IPC object (changes the ipc_addid() path). [PATCH 4/4] uses the specified id…
creator: Nadia Derbey
date: 2008-04-18T05:44:59-00:00

[PATCH 4/4] - v2 - PID: use the target ID specified in procfs
type: item
description: [PATCH 04/04] This patch makes use of the target ids specified by a previous write to /proc/self/task/<tid>/next_id as the ids to use to allocate the next upid nrs. Upper levels upid nrs that are not specified in next_pids file are left to the kernel choice. Signed-off-by: Nadia Derbey <Nadia.Derbey@bull.net> --- include/linux/pid.h | 2 kernel/fork.c | 3 - kernel/pid.c | 141 +++++++++++++++++++++++++++++++++++++++++++++------- 3 files changed, 126 insertions(+), 20 deletions(-) Index: linux-2.6.25-rc8-mm2/include/linux/pid.h =================================================================== --- linux-2.6.25-rc8-mm2.orig/include/linux/pid.h 2008-04-17 12:50:22.000000000 +0200 +++ linux-2.6.25-rc8-mm2/include/linux/pid.h 2008-04-17 17:42:11.000000000 +0200 @@ -121,7 +121,7 @@ extern struct pid *find_get_pid(int nr); extern struct pid *find_ge_pid(int nr, struct pid_namespace *); int next_pidmap(struct pid_namespace *pid_ns, int last); -extern struct pid *alloc_pid(struct pid_namespace *ns); +extern struct pid *alloc_pid(struct pid_namespace *ns, int *retval); extern void free_pid(struct pid *pid); /* Index: linux-2.6.25-rc8-mm2/kernel/fork.c =================================================================== --- linux-2.6.25-rc8-mm2.orig/kernel/fork.c 2008-04-17 13:49:09.000000000 +0200 +++ linux-2.6.25-rc8-mm2/kernel/fork.c 2008-04-17 17:42:48.000000000 +0200 @@ -1200,8 +1200,7 @@ static struct task_struct *copy_process( goto bad_fork_cleanup_io; if (pid != &init_struct_pid) { - retval = -ENOMEM; - pid = alloc_pid(task_active_pid_ns(p)); + pid = alloc_pid(task_active_pid_ns(p), &retval); if (!pid) goto bad_fork_cleanup_io; Index: linux-2.6.25-rc8-mm2/kernel/pid.c =================================================================== --- linux-2.6.25-rc8-mm2.orig/kernel/pid.c 2008-04-17 12:50:25.000000000 +0200 +++ linux-2.6.25-rc8-mm2/kernel/pid.c 2008-04-17 17:50:00.000000000 +0200 @@ -122,6 +122,26 @@ static void free_pidmap(struct upid *upi atomic_inc(&map->nr_free)…
creator: Nadia Derbey
date: 2008-04-18T05:45:03-00:00

Re: [PATCH 0/4] - v2 - Object creation with a specified id
type: item
description: On Fri, 2008-04-18 at 07:44 +0200, Nadia.Derbey@bull.net wrote: > The syntax is one of: > . echo "LONG XX" > /proc/self/task/<my_tid>/next_id > next object to be created will have an id set to XX > . echo "LONG<n> X0 ... X<n-1>" > /proc/self/task/<my_tid>/next_id > next object to be created will have its ids set to XX0, ... X<n-1> > This is particularly useful for processes that may have several ids if > they belong to nested namespaces. I still think, in its current form, this is pretty dangerous. It might be nice to have a central place to set ids, but there's no real safety here for when we get a mishmash of 50 of these things. We have a queue and we fix the order in which things are inserted, but how do we control the order in which the kernel extracts them? Can we 100% conform to that order in the future and be consistent? Won't that also be dictated by the way userspace behaves? If we're going to go this route, I think we need to do a few things. First, these need to be more "type safe". That means that sticking an ids in here that you mean to go to an IPC should never, ever, ever be able to get interpreted as a PID or as an inode number or anything else. echo "IPC_SEM_ID NN" > /proc/self/task/<my_tid>/next_id echo "PID MM" > /proc/self/task/<my_tid>/next_id Maybe we should include the types in there as well, but those are kinda implied by the kind of id you're setting. Have you thought about 32-bit userspace with 64-bit kernels? Does that cause any issues? I also feel like we should have the kernel able to enumerate which things in a process *are* settable. What are the valid values for that first word being echoed in the file? Does anyone have a list of these ids that we're going to want to set like this? Oren? I'm a bit worried that we might want to set things other than ids with an interface like this and that it won't transition well. -- Dave _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org http…
creator: Dave Hansen
date: 2008-04-18T17:07:07-00:00

Re: [PATCH 0/4] - v2 - Object creation with a specified id
type: item
description: Dave Hansen wrote: > On Fri, 2008-04-18 at 07:44 +0200, Nadia.Derbey@bull.net wrote: > >>The syntax is one of: >> . echo "LONG XX" > /proc/self/task/<my_tid>/next_id >> next object to be created will have an id set to XX >> . echo "LONG<n> X0 ... X<n-1>" > /proc/self/task/<my_tid>/next_id >> next object to be created will have its ids set to XX0, ... X<n-1> >> This is particularly useful for processes that may have several ids if >> they belong to nested namespaces. > > > I still think, in its current form, this is pretty dangerous. > > It might be nice to have a central place to set ids, but there's no real > safety here for when we get a mishmash of 50 of these things. We have a > queue and we fix the order in which things are inserted, but how do we > control the order in which the kernel extracts them? I thought that we were the ones who are going to be controlling the restart phase? I think sysV IPCs are particular, since they could be recreated early during the restart phase: we could define a callback routine that would be called by each process being restarted and that would do: for each ipc type: 1. extract next ipc id for my uid and current ipc type 2. write(fd_next_id, id, id_sz) 3. call the XXXget() routine for current ipc_type 4. remove that id from the checkpoint file I think this could only be applied to msg queues and sems, since the creator pid is stored in the perm structure in the shm case. But in the shm case, we can replace in 1. "my uid" by "my tgid_vnr" Then the processes hierarchy restore would take place, so no confusion between the ids. > Can we 100% > conform to that order in the future and be consistent? Won't that also > be dictated by the way userspace behaves? > > If we're going to go this route, I think we need to do a few things. > First, these need to be more "type safe". That means that sticking an > ids in here that you mean to go to an IPC should never, ever, ever be > able to get interpreted as a PID or as an inode number or any…
creator: Nadia Derbey
date: 2008-04-21T11:32:41-00:00

Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Fri, Apr 18, 2008 at 07:44:59AM +0200, Nadia.Derbey@bull.net wrote: > . echo "LONG XX" > /proc/self/task/<my_tid>/next_id > next object to be created will have an id set to XX > . echo "LONG<n> X0 ... X<n-1>" > /proc/self/task/<my_tid>/next_id > next object to be created will have its ids set to XX0, ... X<n-1> > This is particularly useful for processes that may have several ids if > they belong to nested namespaces. Can we answer the following questions before merging this patch: a) should mainline kernel have checkpoint/restart feature at all b) if yes, should it be done from kernel- or userspace? Until agreement will be "yes/from userspace" such patches don't make sense in mainline. _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Alexey Dobriyan
date: 2008-04-22T18:44:29-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Tue, 2008-04-22 at 23:36 +0400, Alexey Dobriyan wrote: > On Fri, Apr 18, 2008 at 07:44:59AM +0200, Nadia.Derbey@bull.net wrote: > > . echo "LONG XX" > /proc/self/task/<my_tid>/next_id > > next object to be created will have an id set to XX > > . echo "LONG<n> X0 ... X<n-1>" > /proc/self/task/<my_tid>/next_id > > next object to be created will have its ids set to XX0, ... X<n-1> > > This is particularly useful for processes that may have several ids if > > they belong to nested namespaces. > > Can we answer the following questions before merging this patch: > > a) should mainline kernel have checkpoint/restart feature at all > b) if yes, should it be done from kernel- or userspace? > > Until agreement will be "yes/from userspace" such patches don't make > sense in mainline. What do you mean by "from kernel" or "from userspace"? Userspace has to be involved somewhere, right? It at least need to open() the checkpoint file and pass it into the kernel. -- Dave _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dave Hansen
date: 2008-04-22T18:56:20-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Quoting Dave Hansen (dave@linux.vnet.ibm.com): > On Tue, 2008-04-22 at 23:36 +0400, Alexey Dobriyan wrote: > > On Fri, Apr 18, 2008 at 07:44:59AM +0200, Nadia.Derbey@bull.net wrote: > > > . echo "LONG XX" > /proc/self/task/<my_tid>/next_id > > > next object to be created will have an id set to XX > > > . echo "LONG<n> X0 ... X<n-1>" > /proc/self/task/<my_tid>/next_id > > > next object to be created will have its ids set to XX0, ... X<n-1> > > > This is particularly useful for processes that may have several ids if > > > they belong to nested namespaces. > > > > Can we answer the following questions before merging this patch: > > > > a) should mainline kernel have checkpoint/restart feature at all > > b) if yes, should it be done from kernel- or userspace? > > > > Until agreement will be "yes/from userspace" such patches don't make > > sense in mainline. > > What do you mean by "from kernel" or "from userspace"? Userspace has to > be involved somewhere, right? It at least need to open() the checkpoint > file and pass it into the kernel. Well let's start by answering a) : I vote yes :-) As for b), good question. Except as Dave points out I think it's pretty unlikely that we'll end up with "sys_checkpoint(checkpoint_dirnam)" and "sys_restart(checkpointed_dirnam)". Anything more user-space driven will likely require userspace to set ids on resources, right? But yes, we have to completely answer b) before this patch goes in, no doubt about that. Alexey, are you advocating for a completely kernel-driven approach, or just making sure we come to a decision? thanks, -serge _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: serue
date: 2008-04-22T19:51:42-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Tue, Apr 22, 2008 at 11:56:20AM -0700, Dave Hansen wrote: > On Tue, 2008-04-22 at 23:36 +0400, Alexey Dobriyan wrote: > > On Fri, Apr 18, 2008 at 07:44:59AM +0200, Nadia.Derbey@bull.net wrote: > > > . echo "LONG XX" > /proc/self/task/<my_tid>/next_id > > > next object to be created will have an id set to XX > > > . echo "LONG<n> X0 ... X<n-1>" > /proc/self/task/<my_tid>/next_id > > > next object to be created will have its ids set to XX0, ... X<n-1> > > > This is particularly useful for processes that may have several ids if > > > they belong to nested namespaces. > > > > Can we answer the following questions before merging this patch: > > > > a) should mainline kernel have checkpoint/restart feature at all > > b) if yes, should it be done from kernel- or userspace? > > > > Until agreement will be "yes/from userspace" such patches don't make > > sense in mainline. > > What do you mean by "from kernel" or "from userspace"? By "from userspace" I mean proposed interfaces and similar: usespace by special system calls puts some state of future object and then does normal system call which creates aforementioned object. This can be used for PIDs, OK. This can be used for SystemV shmem ids. But SystemV shmem also has (let's choose) uid/gid, atimes and actual content. How would restoration look like? How to restore struct task_struct::did_exec ? Do execve(2)? A was ptracing B at checkpoint moment... Netdevices: stats, name, all sorts of flags, hw addresses, MTU iptables rules. These next ids are suitable, well, only for ids which is very, very small part of kernel state needed to restore group of processes. > Userspace has to be involved somewhere, right? It at least need to open() > the checkpoint file and pass it into the kernel. Sure, but opening dumpfile is not an interesting part. _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Alexey Dobriyan
date: 2008-04-22T20:09:21-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Wed, 2008-04-23 at 01:01 +0400, Alexey Dobriyan wrote: > On Tue, Apr 22, 2008 at 11:56:20AM -0700, Dave Hansen wrote: > > On Tue, 2008-04-22 at 23:36 +0400, Alexey Dobriyan wrote:> > > > > a) should mainline kernel have checkpoint/restart feature at all > > > b) if yes, should it be done from kernel- or userspace? > > > > > > Until agreement will be "yes/from userspace" such patches don't make > > > sense in mainline. > > > > What do you mean by "from kernel" or "from userspace"? > > By "from userspace" I mean proposed interfaces and similar: usespace by > special system calls puts some state of future object and then does > normal system call which creates aforementioned object. > > This can be used for PIDs, OK. > > This can be used for SystemV shmem ids. But SystemV shmem also has > (let's choose) uid/gid, atimes and actual content. How would restoration > look like? > > How to restore struct task_struct::did_exec ? Do execve(2)? > > A was ptracing B at checkpoint moment... > > Netdevices: stats, name, all sorts of flags, hw addresses, MTU > > iptables rules. Don't we already have interfaces to dump these out and restore them? My argument is this: If we have interfaces that exist (like setting up iptables rules) we shouldn't make a second interface *just* for checkpoint/restart. We have a hard enough time getting *one* interface right for things, I can't imagine getting two right, and *keeping* them right. If the current interface is insufficient, we should first expand it in such a way that it can be used for checkpoint. That certainly won't work in all cases. fork(), for instance, doesn't take any arguments and is going to be awfully hard to expand. :) I'd love to hear some of your insights about how things like the current iptables interfaces are insufficient for checkpoint/restart. > These next ids are suitable, well, only for ids which is very, very small > part of kernel state needed to restore group of processes. I couldn't agree more. This id setting m…
creator: Dave Hansen
date: 2008-04-22T22:56:07-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: > If the current interface is insufficient, we should first expand it in > such a way that it can be used for checkpoint. That certainly won't > work in all cases. fork(), for instance, doesn't take any arguments and > is going to be awfully hard to expand. :) > > I'd love to hear some of your insights about how things like the current > iptables interfaces are insufficient for checkpoint/restart. iptables is a bad example. Luckily for checkpointing - it always had an interface "load full state", "dump full state". But even iptables are not working in current form for checkpointing - they can't save/restore state of conntracks. Do you think netdev@/netfilter@ guys will be happy to have APIs allowing to set conntracks and have all the pain related to API stability - cause conntrack state changed a couple of times during last 2 years. Consider more intimate kernel states like: a. task statistics b. task start time c. load average d. skb state and it's data. e. mount tree. If you think over, e.g. (b) is a bad thing. It was used to be accounted in jiffies, then in timespec. (a) is another example of dataset which we can't predict. task statistics change over a time. Why bother with such intimate data in user-space at all? Why the hell user-space should know about it and be ABLE to modify it? Why do we need to export ability to set IDs for some objects which none of the operating systems do and then to have a burden to support it for application compatibility the rest of our lifes? Do you really believe none of the applications except for checkpointing will be using it? My personal vision is that: 1. user space must initialize checkpointing/restore state via some system call, supply file descriptor from where data can be read/written to. 2. must call the syscall asking kernel to restore/save different subsytems one by one. 3. finalize cpt/restore state via the syscall But user-space MUST NOT bother about data content. At least not about the data supplied by the kernel. I…
creator: Kirill Korotaev
date: 2008-04-23T06:40:21-00:00

Re: [PATCH 0/4] - v2 - Object creation with a specified id
type: item
description: Hi! > When restarting a process that has been previously checkpointed, that process > should keep on using some of its ids (such as its process id, or sysV ipc ids). > > This patch provides a feature that can help ensuring this saved state reuse: > it makes it possible to create an object with a pre-defined id. > > A first implementation had been proposed 2 months ago. It consisted in > changing an object's id after it had been created. > > Here is a second implementation based on Oren Ladaan's idea: Oren's suggestion > was to force an object's id during its creation, rather than 1. create it, > 2. change its id. > > A new file is created in procfs: /proc/self/task/<my_tid>/next_id. > This makes it possible to avoid races between several threads belonging to > the same process. Ugly... > When this file is filled with an id value, a structure pointed to by the > calling task_struct is filled with that id. > > Then, when an object supporting this feature is created, the id present in > that new structure is used, instead of the default one. > > The syntax is one of: > . echo "LONG XX" > /proc/self/task/<my_tid>/next_id > next object to be created will have an id set to XX > . echo "LONG<n> X0 ... X<n-1>" > /proc/self/task/<my_tid>/next_id > next object to be created will have its ids set to XX0, ... X<n-1> > This is particularly useful for processes that may have several ids if > they belong to nested namespaces. So ugly it is not even funny. Create fork_with_pid(int pid) and corresponding ipc primitives, if you have to... -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Pavel Machek
date: 2008-04-23T14:23:51-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Wed, 2008-04-23 at 10:40 +0400, Kirill Korotaev wrote: > Having all this functionality in a signle syscall we specifically CLAIM a black box, > and that no one can use this interfaces for something different from checkpoint/restore. > > So I think we have to know what other maintainers think before we can go. I completely agree. Have you asked any particular maintainers what they think? -- Dave _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dave Hansen
date: 2008-04-23T15:33:18-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Kirill Korotaev wrote: >> If the current interface is insufficient, we should first expand it in >> such a way that it can be used for checkpoint. That certainly won't >> work in all cases. fork(), for instance, doesn't take any arguments and >> is going to be awfully hard to expand. :) >> >> I'd love to hear some of your insights about how things like the current >> iptables interfaces are insufficient for checkpoint/restart. > > iptables is a bad example. Luckily for checkpointing - it always had an interface > "load full state", "dump full state". > > But even iptables are not working in current form for checkpointing - they can't > save/restore state of conntracks. Do you think netdev@/netfilter@ guys will be happy > to have APIs allowing to set conntracks and have all the pain related > to API stability - cause conntrack state changed a couple of times during last 2 years. > > Consider more intimate kernel states like: > a. task statistics > b. task start time > c. load average > d. skb state and it's data. > e. mount tree. > > If you think over, e.g. (b) is a bad thing. It was used to be accounted in jiffies, then in timespec. > (a) is another example of dataset which we can't predict. task statistics change over a time. > Why bother with such intimate data in user-space at all? > Why the hell user-space should know about it and be ABLE to modify it? Agreed. > Why do we need to export ability to set IDs for some objects which none of the > operating systems do and then to have a burden to support it for application compatibility > the rest of our lifes? Do you really believe none of the applications except for checkpointing > will be using it? > > My personal vision is that: > 1. user space must initialize checkpointing/restore state via some system call, > supply file descriptor from where data can be read/written to. > 2. must call the syscall asking kernel to restore/save different subsytems one by one. > 3. finalize cpt/restore state via the syscall > But …
creator: Oren Laadan
date: 2008-04-24T01:19:51-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Yeah... we talked with Andrew yesterday. He mostly agrees with a black box. He also said an interesting idea, that if you need compatibility between the kernels (like we for example, support for migration from 2.6.9 to 2.6.18 in OpenVZ) you can do image conversion in user-space... Though I think we will prefer simply to have a compatibility patch for specific kernel versions in our kernel tree (not in mainstream). Definitely, other ideas/opinions are welcome. Kirill Dave Hansen wrote: > On Wed, 2008-04-23 at 10:40 +0400, Kirill Korotaev wrote: >> Having all this functionality in a signle syscall we specifically CLAIM a black box, >> and that no one can use this interfaces for something different from checkpoint/restore. >> >> So I think we have to know what other maintainers think before we can go. > > I completely agree. > > Have you asked any particular maintainers what they think? > > -- Dave > > _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Kirill Korotaev
date: 2008-04-24T07:00:18-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Thu, 2008-04-24 at 11:00 +0400, Kirill Korotaev wrote: > Yeah... we talked with Andrew yesterday. He mostly agrees with a black > box. > He also said an interesting idea, that if you need compatibility > between the kernels (like we for example, support for migration from > 2.6.9 to 2.6.18 in OpenVZ) > you can do image conversion in user-space... That sounds compelling to me. I'm definitely willing to explore it. Any thoughts on what we should start with? What can we get merged, most easily? -- Dave _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dave Hansen
date: 2008-04-24T18:30:09-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Dave Hansen wrote: > On Thu, 2008-04-24 at 11:00 +0400, Kirill Korotaev wrote: >> Yeah... we talked with Andrew yesterday. He mostly agrees with a black >> box. >> He also said an interesting idea, that if you need compatibility >> between the kernels (like we for example, support for migration from >> 2.6.9 to 2.6.18 in OpenVZ) >> you can do image conversion in user-space... > > That sounds compelling to me. I'm definitely willing to explore it. Filtering/converting in user-space has long been part of zap: used to be able to migrate between both minor and major versions of the kernel (always upwards, of course). In fact, I already proposed it as part of the original thread that led eventually to the patch in question: https://lists.linux-foundation.org/pipermail/containers/2008-February/009849.html The need to be able to specify a desired ID (from user-space and/or from in-kernel) remains. Oren. > Any thoughts on what we should start with? What can we get merged, most > easily? > > -- Dave > > _______________________________________________ > Containers mailing list > Containers@lists.linux-foundation.org > https://lists.linux-foundation.org/mailman/listinfo/containers _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Oren Laadan
date: 2008-04-24T23:13:01-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Oren Laadan <orenl@cs.columbia.edu> writes: >> Consider more intimate kernel states like: >> a. task statistics >> b. task start time >> c. load average >> d. skb state and it's data. >> e. mount tree. >> >> If you think over, e.g. (b) is a bad thing. It was used to be accounted in > jiffies, then in timespec. >> (a) is another example of dataset which we can't predict. task statistics > change over a time. >> Why bother with such intimate data in user-space at all? >> Why the hell user-space should know about it and be ABLE to modify it? > > Agreed. Almost agreed. The reason we care is that the data is visible to user space in some form. So we need to save it, but hopefully not in it's internal kernel representation. If we don't care at all we should not save it. >> My personal vision is that: >> 1. user space must initialize checkpointing/restore state via some system > call, >> supply file descriptor from where data can be read/written to. >> 2. must call the syscall asking kernel to restore/save different subsytems one > by one. >> 3. finalize cpt/restore state via the syscall >> But user-space MUST NOT bother about data content. At least not about the data > supplied by the kernel. >> It can add additional sections if needed, e.g. about iptables state. > > I mostly agree with the vision that checkpoint/restart is probably best > implemented as a black box: I would claim an atomic unit. Roughly like a coredump is today. We know what a core dump does and we don't care how it does it. > * First, much of the work required to restore the state of a process > as well as the state of its resources, requires kernel interfaces that > are lower than the ones available to user-space. Working in user-space > will require that we design new complex interfaces for this purpose only. Yes. > * Second, much of the state that needs to be saved was not, is not, and > should probably never be exported to user-space (e.g. interval socket > buffers, t->did_exec and many more). It is…
creator: ebiederm
date: 2008-07-10T01:58:55-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Wed, 2008-07-09 at 18:58 -0700, Eric W. Biederman wrote: > In the worst case today we can restore a checkpoint by replaying all of > the user space actions that took us to get there. That is a tedious > and slow approach. Yes, tedious and slow, *and* minimally invasive in the kernel. Once we have a tedious and slow process, we'll have some really good points when we try to push the next set of patches to make it less slow and tedious. We'll be able to describe an _actual_ set of problems to our fellow kernel hackers. So, the checkpoint-as-a-corefile idea sounds good to me, but it definitely leaves a lot of questions about exactly how we'll need to do the restore. -- Dave _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dave Hansen
date: 2008-07-10T17:12:29-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Quoting Dave Hansen (dave@linux.vnet.ibm.com): > On Wed, 2008-07-09 at 18:58 -0700, Eric W. Biederman wrote: > > In the worst case today we can restore a checkpoint by replaying all of > > the user space actions that took us to get there. That is a tedious > > and slow approach. > > Yes, tedious and slow, *and* minimally invasive in the kernel. Once we > have a tedious and slow process, we'll have some really good points when > we try to push the next set of patches to make it less slow and tedious. > We'll be able to describe an _actual_ set of problems to our fellow > kernel hackers. > > So, the checkpoint-as-a-corefile idea sounds good to me, but it > definitely leaves a lot of questions about exactly how we'll need to do > the restore. Talking with Dave over irc, I kind of liked the idea of creating a new fs/binfmt_cr.c that executes a checkpoint-as-a-coredump file. One thing I do not like about the checkpoint-as-coredump is that it begs us to dump all memory out into the file. Our plan/hope was to save ourselves from writing out most memory by: 1. associating a separate swapfile with each container 2. doing a swapfile snapshot at each checkpoint 3. dumping the pte entries (/proc/self/) If we do checkpoint-as-a-coredump, then we need userspace to coordinate a kernel-generated coredump with a user-generated (?) swapfile snapshot. But I guess we figure that out later. -serge _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: serue
date: 2008-07-10T17:32:46-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: "Serge E. Hallyn" <serue@us.ibm.com> writes: >> So, the checkpoint-as-a-corefile idea sounds good to me, but it >> definitely leaves a lot of questions about exactly how we'll need to do >> the restore. > > Talking with Dave over irc, I kind of liked the idea of creating a new > fs/binfmt_cr.c that executes a checkpoint-as-a-coredump file. > > One thing I do not like about the checkpoint-as-coredump is that it begs > us to dump all memory out into the file. Our plan/hope was to save > ourselves from writing out most memory by: > > 1. associating a separate swapfile with each container > 2. doing a swapfile snapshot at each checkpoint > 3. dumping the pte entries (/proc/self/) > > If we do checkpoint-as-a-coredump, then we need userspace to coordinate > a kernel-generated coredump with a user-generated (?) swapfile snapshot. > But I guess we figure that out later. Well it is a matter of which VMAs you dump. For things that are file backed you need to dump them. I don't know that even a binfmt for per process level checkpoints is sufficient but I do know having something of that granularity looks much easier. Otherwise it takes a bazillian little syscalls to do things no one else is interested in doing. Eric _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: ebiederm
date: 2008-07-10T18:55:34-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Thu, 2008-07-10 at 11:55 -0700, Eric W. Biederman wrote: > > > If we do checkpoint-as-a-coredump, then we need userspace to coordinate > > a kernel-generated coredump with a user-generated (?) swapfile snapshot. > > But I guess we figure that out later. > > Well it is a matter of which VMAs you dump. For things that are file backed > you need to dump them. Are we talking about the VMA itself, or the memory backing the VMA? -- Dave _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dave Hansen
date: 2008-07-10T19:06:34-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Dave Hansen <dave@linux.vnet.ibm.com> writes: > On Thu, 2008-07-10 at 11:55 -0700, Eric W. Biederman wrote: >> >> > If we do checkpoint-as-a-coredump, then we need userspace to coordinate >> > a kernel-generated coredump with a user-generated (?) swapfile snapshot. >> > But I guess we figure that out later. >> >> Well it is a matter of which VMAs you dump. For things that are file backed >> you need to dump them. For things that are file backed you DO NOT need to dump them. (sorry typo). > Are we talking about the VMA itself, or the memory backing the VMA? The memory backing the VMA. We need to store the page protections that the memory was mapped with as well now that you point it out. A VMA is not user space visible, which is why we can arbitrarily split and merge VMAs. Eric _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: ebiederm
date: 2008-07-10T19:21:43-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Thu, 2008-07-10 at 12:21 -0700, Eric W. Biederman wrote: > > Are we talking about the VMA itself, or the memory backing the VMA? > > The memory backing the VMA. We need to store the page protections > that the memory was mapped with as well now that you point it out. A > VMA is not user space visible, which is why we can arbitrarily split > and merge VMAs. It is visible with /proc/$pid/{maps,smaps,numamaps?}. That's the only efficient way I know of from userspace to figure out where userspace's memory is and if it *is* file backed, and what the permissions are. We also can't truly split them arbitrarily because the memory cost of the VMAs themselves might become too heavy (the remap_file_pages() problem). It gets trickier when things are also private mappings in addition to being in a file-backed VMA. We *do* need to checkpoint those, but only the pages to which there was a write. There's also the problem of restoring things read-only, but VM_MAYWRITE. If there's a high level of (non-COW'd) sharing of these anonymous areas, we may not be able to even restore the set of processes unless we can replicate the sharing. We might run out of memory if the sharing isn't replicated. Memory is fun! :) -- Dave _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dave Hansen
date: 2008-07-10T19:47:32-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: On Thu, Jul 10, 2008 at 12:47:32PM -0700, Dave Hansen wrote: > On Thu, 2008-07-10 at 12:21 -0700, Eric W. Biederman wrote: > > > Are we talking about the VMA itself, or the memory backing the VMA? > > > > The memory backing the VMA. We need to store the page protections > > that the memory was mapped with as well now that you point it out. A > > VMA is not user space visible, which is why we can arbitrarily split > > and merge VMAs. > > It is visible with /proc/$pid/{maps,smaps,numamaps?}. That's the only > efficient way I know of from userspace to figure out where userspace's > memory is and if it *is* file backed, and what the permissions are. None of those files exist if PROC_FS is off. > We also can't truly split them arbitrarily because the memory cost of > the VMAs themselves might become too heavy (the remap_file_pages() > problem). > > It gets trickier when things are also private mappings in addition to > being in a file-backed VMA. We *do* need to checkpoint those, but only > the pages to which there was a write. > > There's also the problem of restoring things read-only, but VM_MAYWRITE. > If there's a high level of (non-COW'd) sharing of these anonymous areas, > we may not be able to even restore the set of processes unless we can > replicate the sharing. We might run out of memory if the sharing isn't > replicated. _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Alexey Dobriyan
date: 2008-07-11T00:32:58-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Eric W. Biederman wrote: > Oren Laadan <orenl@cs.columbia.edu> writes: > >>> Consider more intimate kernel states like: >>> a. task statistics >>> b. task start time >>> c. load average >>> d. skb state and it's data. >>> e. mount tree. >>> >>> If you think over, e.g. (b) is a bad thing. It was used to be accounted in >> jiffies, then in timespec. >>> (a) is another example of dataset which we can't predict. task statistics >> change over a time. >>> Why bother with such intimate data in user-space at all? >>> Why the hell user-space should know about it and be ABLE to modify it? >> Agreed. > > Almost agreed. The reason we care is that the data is visible to user > space in some form. So we need to save it, but hopefully not in it's internal > kernel representation. If we don't care at all we should not save it. yes, we probably prefer an intermediate representation to some extent, mainly because we (a) don't want garbage data that we don't use and (b) to make the task of converting from old to new kernel version easier. But only for these two reasons. I see zero value in avoiding representation used internally by the kernel for the sake of avoiding it. In fact, if it's that much easier to stick to that representation in a specific case -- so be it. There shall be userland utilities that will be able to inspect the contents (like those that can inspect the contents of internal file system data). >>> My personal vision is that: >>> 1. user space must initialize checkpointing/restore state via some system >> call, >>> supply file descriptor from where data can be read/written to. >>> 2. must call the syscall asking kernel to restore/save different subsytems one >> by one. >>> 3. finalize cpt/restore state via the syscall >>> But user-space MUST NOT bother about data content. At least not about the data >> supplied by the kernel. >>> It can add additional sections if needed, e.g. about iptables state. >> I mostly agree with the vision that checkpoint/restart is probabl…
creator: Oren Laadan
date: 2008-07-17T23:09:53-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Dave Hansen wrote: > On Wed, 2008-07-09 at 18:58 -0700, Eric W. Biederman wrote: >> In the worst case today we can restore a checkpoint by replaying all of >> the user space actions that took us to get there. That is a tedious >> and slow approach. > > Yes, tedious and slow, *and* minimally invasive in the kernel. Once we > have a tedious and slow process, we'll have some really good points when "Replaying all of the user space actions that took us to get there" - this task is not even always possible without deterministic log/replay mechanism :( Oren. > we try to push the next set of patches to make it less slow and tedious. > We'll be able to describe an _actual_ set of problems to our fellow > kernel hackers. > > So, the checkpoint-as-a-corefile idea sounds good to me, but it > definitely leaves a lot of questions about exactly how we'll need to do > the restore. > > -- Dave > _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Oren Laadan
date: 2008-07-17T23:14:25-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Serge E. Hallyn wrote: > Quoting Dave Hansen (dave@linux.vnet.ibm.com): >> On Wed, 2008-07-09 at 18:58 -0700, Eric W. Biederman wrote: >>> In the worst case today we can restore a checkpoint by replaying all of >>> the user space actions that took us to get there. That is a tedious >>> and slow approach. >> Yes, tedious and slow, *and* minimally invasive in the kernel. Once we >> have a tedious and slow process, we'll have some really good points when >> we try to push the next set of patches to make it less slow and tedious. >> We'll be able to describe an _actual_ set of problems to our fellow >> kernel hackers. >> >> So, the checkpoint-as-a-corefile idea sounds good to me, but it >> definitely leaves a lot of questions about exactly how we'll need to do >> the restore. > > Talking with Dave over irc, I kind of liked the idea of creating a new > fs/binfmt_cr.c that executes a checkpoint-as-a-coredump file. > > One thing I do not like about the checkpoint-as-coredump is that it begs > us to dump all memory out into the file. Our plan/hope was to save > ourselves from writing out most memory by: > > 1. associating a separate swapfile with each container > 2. doing a swapfile snapshot at each checkpoint > 3. dumping the pte entries (/proc/self/) > > If we do checkpoint-as-a-coredump, then we need userspace to coordinate > a kernel-generated coredump with a user-generated (?) swapfile snapshot. > But I guess we figure that out later. I'm not sure how this approach integrates with (a) live migration (and the iterative process of sending over memory modified since previous iteration), and (b) incremental checkpoint (where except for the first snapshot, additional snapshots only save what changed since the previous one). Oren. > > -serge _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Oren Laadan
date: 2008-07-17T23:16:10-00:00

Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation with a specified id)
type: item
description: Dave Hansen wrote: > On Thu, 2008-07-10 at 12:21 -0700, Eric W. Biederman wrote: >>> Are we talking about the VMA itself, or the memory backing the VMA? >> The memory backing the VMA. We need to store the page protections >> that the memory was mapped with as well now that you point it out. A >> VMA is not user space visible, which is why we can arbitrarily split >> and merge VMAs. > > It is visible with /proc/$pid/{maps,smaps,numamaps?}. That's the only > efficient way I know of from userspace to figure out where userspace's > memory is and if it *is* file backed, and what the permissions are. > > We also can't truly split them arbitrarily because the memory cost of > the VMAs themselves might become too heavy (the remap_file_pages() > problem). > > It gets trickier when things are also private mappings in addition to > being in a file-backed VMA. We *do* need to checkpoint those, but only > the pages to which there was a write. > > There's also the problem of restoring things read-only, but VM_MAYWRITE. > If there's a high level of (non-COW'd) sharing of these anonymous areas, > we may not be able to even restore the set of processes unless we can > replicate the sharing. We might run out of memory if the sharing isn't > replicated. > > Memory is fun! :) Heh .. and it can get much worse. By all means, the functions that analyze and save the VMAs during checkpoint and later reconstruct them during restart are the most complicated logic. The good news, however, is that it works :) Oren. > > -- Dave > _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Oren Laadan
date: 2008-07-17T23:19:51-00:00