OpenVZ Forum - RDF feed
type: channel
items: [RFC][PATCH 1/5] Virtualization/containers: startup
items: [RFC][PATCH 2/5] Virtualization/containers: UIDs
items: [RFC][PATCH 3/5] Virtualization/containers: UTSNAME
items: Re: [RFC][PATCH 1/5] Virtualization/containers: startup
items: Summary: PID virtualization , Containers, Migration
items: Re: [RFC][PATCH 3/5] Virtualization/containers: UTSNAME
items: Re: Re: [RFC][PATCH 1/5] Virtualization/containers: startup
items: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
items: Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
items: Re: Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)

[RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: This patch introduces some abstract container/VPS kernel structure and tiny amount of operations on it. Patches following this one will be used for virtualization of some resources based on this container infrastructure, including those VPIDs patches I sent before. What is important here is that: - each VPS has unique ID - each process in kernel can belong to one VPS only Kirill --- ./include/linux/vps_info.h.vps_info 2006-02-03 16:38:33.000000000 +0300 +++ ./include/linux/vps_info.h 2006-02-03 16:49:26.000000000 +0300 @@ -0,0 +1,41 @@ +#ifndef __LINUX_VPS_INFO_H_ +#define __LINUX_VPS_INFO_H_ + +#include <asm/types.h> +#include <asm/atomic.h> + +struct task_struct; + +struct vps_info { + u32 id; + struct task_struct *init_task; + atomic_t refcnt; +}; + +extern struct vps_info host_vps_info; +typedef struct vps_info *vps_t; + +static inline vps_t get_vps(vps_t vps) +{ + atomic_inc(&vps->refcnt); + return vps; +} + +static inline void put_vps(vps_t vps) +{ + atomic_dec(&vps->refcnt); +} + +#include <linux/sched.h> +#include <asm/current.h> + +static inline vps_t set_current_vps(vps_t vps) +{ + vps_t ret; + + ret = current->owner_vps; + current->owner_vps = vps; + return ret; +} + +#endif --- ./include/linux/sched.h.vps_info 2006-02-03 16:38:33.000000000 +0300 +++ ./include/linux/sched.h 2006-02-03 16:43:42.000000000 +0300 @@ -687,6 +687,7 @@ struct audit_context; /* See audit.c */ struct mempolicy; +struct vps_info; struct task_struct { volatile long state; /* -1 unrunnable, 0 runnable, >0 stopped */ @@ -845,6 +846,7 @@ struct backing_dev_info *backing_dev_info; struct io_context *io_context; + struct vps_info *owner_vps; unsigned long ptrace_message; siginfo_t *last_siginfo; /* For ptrace use. */ @@ -875,6 +877,8 @@ struct rcu_head rcu; }; +#define current_vps() (current->owner_vps) + static inline pid_t process_group(struct task_struct *tsk) { return tsk->signal->pgrp; --- ./include/linux/init_task.h.vps_info 2006-02-03 16:38:33.000000000 +0300 +++ ./include/linux/i…
creator: Kirill Korotaev
date: 2006-02-03T16:57:18-00:00

[RFC][PATCH 2/5] Virtualization/containers: UIDs
type: item
description: The simplest virtualization of UID hashes, just to demonstrate our approach. Each container has it's own set of uids and should simply allocate hash/initialize it on startup. Kirill --- ./include/linux/vps_info.h.vps_uid_hash 2006-02-03 16:49:26.000000000 +0300 +++ ./include/linux/vps_info.h 2006-02-03 16:49:51.000000000 +0300 @@ -5,11 +5,14 @@ #include <asm/atomic.h> struct task_struct; +struct list_head; struct vps_info { u32 id; struct task_struct *init_task; atomic_t refcnt; + + struct list_head *vps_uid_hash; }; extern struct vps_info host_vps_info; --- ./kernel/user.c.vps_uid_hash 2006-02-03 16:49:08.000000000 +0300 +++ ./kernel/user.c 2006-02-03 16:49:51.000000000 +0300 @@ -14,6 +14,7 @@ #include <linux/bitops.h> #include <linux/key.h> #include <linux/interrupt.h> +#include <linux/vps_info.h> /* * UID task count cache, to get fast user lookup in "alloc_uid" @@ -24,7 +25,8 @@ #define UIDHASH_SZ (1 << UIDHASH_BITS) #define UIDHASH_MASK (UIDHASH_SZ - 1) #define __uidhashfn(uid) (((uid >> UIDHASH_BITS) + uid) & UIDHASH_MASK) -#define uidhashentry(uid) (uidhash_table + __uidhashfn((uid))) +#define uidhashentry(uid) (current_vps()->vps_uid_hash + \ + __uidhashfn((uid))) static kmem_cache_t *uid_cachep; static struct list_head uidhash_table[UIDHASH_SZ]; @@ -200,6 +202,7 @@ /* Insert the root user immediately (init already runs as root) */ spin_lock_irq(&uidhash_lock); + host_vps_info.vps_uid_hash = uidhash_table; uid_hash_insert(&root_user, uidhashentry(0)); spin_unlock_irq(&uidhash_lock);
creator: dev
date: 2006-02-03T17:01:30-00:00

[RFC][PATCH 3/5] Virtualization/containers: UTSNAME
type: item
description: Virtualization of UTSNAME. As simple as UID hashes, just virtualizes system_utsname variable (diff-vps-uts-name-core) and replaces access to it with vps_utsname in required places (diff-vps-uts-name-kern). Kirill --- ./include/linux/vps_info.h.vps_uts_core 2006-02-03 16:49:51.000000000 +0300 +++ ./include/linux/vps_info.h 2006-02-03 16:50:39.000000000 +0300 @@ -6,6 +6,7 @@ struct task_struct; struct list_head; +struct new_utsname; struct vps_info { u32 id; @@ -13,6 +14,7 @@ atomic_t refcnt; struct list_head *vps_uid_hash; + struct new_utsname *vps_uts_name; }; extern struct vps_info host_vps_info; --- ./include/linux/utsname.h.vps_uts_core 2006-02-03 16:38:17.000000000 +0300 +++ ./include/linux/utsname.h 2006-02-03 16:50:26.000000000 +0300 @@ -1,6 +1,8 @@ #ifndef _LINUX_UTSNAME_H #define _LINUX_UTSNAME_H +#include <linux/vps_info.h> + #define __OLD_UTS_LEN 8 struct oldold_utsname { @@ -31,6 +33,7 @@ }; extern struct new_utsname system_utsname; +#define vps_utsname (*(current_vps()->vps_uts_name)) extern struct rw_semaphore uts_sem; #endif --- ./init/main.c.vps_uts_core 2006-02-03 16:43:18.000000000 +0300 +++ ./init/main.c 2006-02-03 16:50:26.000000000 +0300 @@ -439,6 +439,7 @@ .id = 0, .init_task = &init_task, .refcnt = ATOMIC_INIT(1), + .vps_uts_name = &system_utsname, }; EXPORT_SYMBOL(host_vps_info); --- ./arch/alpha/kernel/osf_sys.c.vps_uts 2006-02-03 11:56:22.000000000 +0300 +++ ./arch/alpha/kernel/osf_sys.c 2006-02-03 13:35:25.000000000 +0300 @@ -402,15 +402,15 @@ down_read(&uts_sem); error = -EFAULT; - if (copy_to_user(name + 0, system_utsname.sysname, 32)) + if (copy_to_user(name + 0, vps_utsname.sysname, 32)) goto out; - if (copy_to_user(name + 32, system_utsname.nodename, 32)) + if (copy_to_user(name + 32, vps_utsname.nodename, 32)) goto out; - if (copy_to_user(name + 64, system_utsname.release, 32)) + if (copy_to_user(name + 64, vps_utsname.release, 32)) goto out; - if (copy_to_user(name + 96, system_utsname.version, 32)) + if (copy_to_user(name + 96, vps_…
creator: dev
date: 2006-02-03T17:04:50-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Fri, 3 Feb 2006, Kirill Korotaev wrote: > > This patch introduces some abstract container/VPS kernel structure and tiny > amount of operations on it. Please don't use things like "vps_t". It's a _mistake_ to use typedef for structures and pointers. When you see a vps_t a; in the source, what does it mean? In contrast, if it says struct virtual_container *a; you can actually tell what "a" is. Lots of people think that typedefs "help readability". Not so. They are useful only for (a) totally opaque objects (where the typedef is actively used to _hide_ what the object is). Example: "pte_t" etc opaque objects that you can only access using the proper accessor functions. NOTE! Opaqueness and "accessor functions" are not good in themselves. The reason we have them for things like pte_t etc is that there really is absolutely _zero_ portably accessible information there. (b) Clear integer types, where the abstraction _helps_ avoid confusion whether it is "int" or "long". u8/u16/u32 are perfectly fine typedefs. NOTE! Again - there needs to be a _reason_ for this. If something is "unsigned long", then there's no reason to do typedef long myflags_t; but if there is a clear reason for why it under certain circumstances might be an "unsigned int" and under other configurations might be "unsigned long", then by all means go ahead and use a typedef. (c) when you use sparse to literally create a _new_ type for type-checking. Maybe there are other cases too, but the rule should basically be to NEVER EVER use a typedef unless you can clearly match one of those rules. In general, a pointer, or a struct that has elements that can reasonably be directly accessed should _never_ be a typedef. Linus
creator: Linus Torvalds
date: 2006-02-03T17:15:19-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Linus, Not a problem and fully agree with you. Just had to better review patch before sending. Do you have any other ideas/comments on this? I will send additional IPC/filesystems virtualization patches a bit later. Kirill > On Fri, 3 Feb 2006, Kirill Korotaev wrote: >> This patch introduces some abstract container/VPS kernel structure and tiny >> amount of operations on it. > > Please don't use things like "vps_t". > > It's a _mistake_ to use typedef for structures and pointers. When you see > a > > vps_t a; > > in the source, what does it mean? > > In contrast, if it says > > struct virtual_container *a; > > you can actually tell what "a" is. > > Lots of people think that typedefs "help readability". Not so. They are > useful only for > > (a) totally opaque objects (where the typedef is actively used to _hide_ > what the object is). > > Example: "pte_t" etc opaque objects that you can only access using > the proper accessor functions. > > NOTE! Opaqueness and "accessor functions" are not good in themselves. > The reason we have them for things like pte_t etc is that there > really is absolutely _zero_ portably accessible information there. > > (b) Clear integer types, where the abstraction _helps_ avoid confusion > whether it is "int" or "long". > > u8/u16/u32 are perfectly fine typedefs. > > NOTE! Again - there needs to be a _reason_ for this. If something is > "unsigned long", then there's no reason to do > > typedef long myflags_t; > > but if there is a clear reason for why it under certain circumstances > might be an "unsigned int" and under other configurations might be > "unsigned long", then by all means go ahead and use a typedef. > > (c) when you use sparse to literally create a _new_ type for > type-checking. > > Maybe there are other cases too, but the rule should basically be to NEVER > EVER use a typedef unless you can clearly match one of those rules. > > In general, a pointer, or a struct that has elements that can reasonably > be directly accessed …
creator: dev
date: 2006-02-03T17:22:34-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Fri, 3 Feb 2006, Kirill Korotaev wrote: > > Do you have any other ideas/comments on this? > I will send additional IPC/filesystems virtualization patches a bit later. I think that a patch like this - particularly just the 1/5 part - makes total sense, because regardless of any other details of virtualization, every single scheme is going to need this. So I think at least 1/5 (and quite frankly, 2-3/5 look that way too) are things that we can (and probably should) merge quickly, so that people can then actually look at the differences in the places that they may actually disagree about. One thing I don't particularly like is some of the naming. To me "vps" doesn't sound particularly generic or logical. I realize that it probably makes perfect sense to you (and I assume it just means "virtual private servers"), but especially if you see patches 1-3 to really be independent of any "actual" virtualization code that is totally generic, I'd actually prefer a less specialized name. I realize that what things like this is used for is VPS web hosting etc, but that's really naming by current common usage, not by any real "logic". In other words, I personally would have called them "container" or something similar, rather than "vps_info". See? From a logical implementation standpoint, the fact that it is right now most commonly used for VPS hosting is totally irrelevant to the code, no? (And hey, maybe your "vps" means something different. In which case my argument makes even more sense ;) Linus
creator: Linus Torvalds
date: 2006-02-03T17:49:56-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Fri, 2006-02-03 at 09:49 -0800, Linus Torvalds wrote: > One thing I don't particularly like is some of the naming. To me "vps" > doesn't sound particularly generic or logical. I realize that it probably > makes perfect sense to you (and I assume it just means "virtual private > servers"), but especially if you see patches 1-3 to really be independent > of any "actual" virtualization code that is totally generic, I'd actually > prefer a less specialized name. I just did a global s/vps/container/ and it looks pretty reasonable, at least from my point of view. Couple of minor naming nitpick questions, though. Is vps/container_info really what we want to call it? It seems to me to be the basis for a real "container", without the _info part. "tsk->owner_container" That makes it sound like a pointer to the "task owner's container". How about "owning_container"? The "container owning this task". Or, maybe just "container"? Any particular reason for the "u32 id" in the vps_info struct as opposed to one of the more generic types? Do we want to abstract this one in the same way we do pid_t? The "host" in "host_container_info" doesn't mean much to me. Though, I guess it has some context in the UML space. Would "init_container_info" or "root_container_info" be more descriptive? Lastly, is this a place for krefs? I don't see a real need for a destructor yet, but the idea is fresh in my mind. How does the attached patch look? -- Dave memhotplug-dave/include/linux/container.h | 40 +++++++++++++++++++++++++ memhotplug-dave/include/linux/init_task.h | 2 + memhotplug-dave/include/linux/sched.h | 4 ++ memhotplug-dave/init/main.c | 8 +++++ memhotplug-dave/kernel/exit.c | 2 + memhotplug-dave/kernel/fork.c | 2 + 6 files changed, 58 insertions(+) diff -puN include/linux/container.h~container-base include/linux/container.h --- memhotplug/include/linux/container.h~container-base 2006-02-03 10:21:36.000000000 -0800 +++ memhotplug-dave/include/linux/container.h 2006-02-03 10:21:36.00000000…
creator: Dave Hansen
date: 2006-02-03T18:34:01-00:00

Summary: PID virtualization , Containers, Migration
type: item
description: Seems lots of interest is happening in this direction now with various ideas and patches having been submitted. I think the terms should be clarified a bit, even in our own discussion we are using them loosely. Let's keep focusing on the PID issues of containers and migration. - user have come to expect that pids range from [0..PIDMAX] - we can only migrate a self contained process tree (with some glue at the top) - when we migrate we expect that the pid is preserved to the user - we expect that UNIX semantics of pid visibility (e.g. kill etc) do not cross the container boundary (with some glue at the top) Now there are two views to achieve the preservation that go at the heart of the implementation only (not at the concept at all). Two fundamental concepts are thrown around here: (a) PID virtualization (b) PID spaces Let's be clear what I mean by these concepts: PID virtualization: =================== The kernel continues to use its current internal pid management. Each task has a unique internal pid, throughout its life on that particular kernel image. At the user/kernel boundary these pids are virtualized on a per container base. Hence pids coming from user space are converted to an internal pid and internal pids handed to the user are converted to the virtual pid. As a result, all places along the user/kernel boundary that deal with pids need to be identified and the conversion function needs to be called. Also the vpid needs to be allocated/freed on fork/exit. How this virtualization is maintained is an implementation detail. One can keep a full "pid->vpid" and "vpi->pid" hash with the container or play optimizations such as the OpenVZ folks to keep extra virtual numbers in the task->pids[type] structure and or making vpid == internal pid for non-containerized processes. Right now the OpenVZ patch supplies a nice and clean means for this. In a sense our patch also followes this approach because we explicitely have these conversion functions and we too have simi…
creator: Hubertus Franke
date: 2006-02-03T18:36:35-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Fri, 3 Feb 2006, Linus Torvalds wrote: > So I think at least 1/5 (and quite frankly, 2-3/5 look that way too) are > things that we can (and probably should) merge quickly, so that people > can then actually look at the differences in the places that they may > actually disagree about. Agreed. There are a lot of common bits between the various virtualization solutions, if we can get those out of the way it should be a lot easier to get the last bits done... -- All Rights Reversed
creator: Rik van Riel
date: 2006-02-03T18:36:56-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Dave Hansen wrote: > On Fri, 2006-02-03 at 09:49 -0800, Linus Torvalds wrote: > >>One thing I don't particularly like is some of the naming. To me "vps" >>doesn't sound particularly generic or logical. I realize that it probably >>makes perfect sense to you (and I assume it just means "virtual private >>servers"), but especially if you see patches 1-3 to really be independent >>of any "actual" virtualization code that is totally generic, I'd actually >>prefer a less specialized name. > > > I just did a global s/vps/container/ and it looks pretty reasonable, at > least from my point of view. I would have chosen the much shorter "box" or "jar", but that's just me :) > "tsk->owner_container" That makes it sound like a pointer to the "task > owner's container". How about "owning_container"? The "container > owning this task". Or, maybe just "container"? slip 'parent' in there... > Any particular reason for the "u32 id" in the vps_info struct as opposed > to one of the more generic types? Do we want to abstract this one in > the same way we do pid_t? > > The "host" in "host_container_info" doesn't mean much to me. Though, I > guess it has some context in the UML space. Would "init_container_info" > or "root_container_info" be more descriptive? probably Jeff
creator: Jeff Garzik
date: 2006-02-03T18:55:13-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Dave Hansen wrote: > On Fri, 2006-02-03 at 09:49 -0800, Linus Torvalds wrote: > >>One thing I don't particularly like is some of the naming. To me "vps" >>doesn't sound particularly generic or logical. I realize that it probably >>makes perfect sense to you (and I assume it just means "virtual private >>servers"), but especially if you see patches 1-3 to really be independent >>of any "actual" virtualization code that is totally generic, I'd actually >>prefer a less specialized name. > > > I just did a global s/vps/container/ and it looks pretty reasonable, at > least from my point of view. > > Couple of minor naming nitpick questions, though. Is vps/container_info > really what we want to call it? It seems to me to be the basis for a > real "container", without the _info part. > > "tsk->owner_container" That makes it sound like a pointer to the "task > owner's container". How about "owning_container"? The "container > owning this task". Or, maybe just "container"? > > Any particular reason for the "u32 id" in the vps_info struct as opposed > to one of the more generic types? Do we want to abstract this one in > the same way we do pid_t? > > The "host" in "host_container_info" doesn't mean much to me. Though, I > guess it has some context in the UML space. Would "init_container_info" > or "root_container_info" be more descriptive? > > Lastly, is this a place for krefs? I don't see a real need for a > destructor yet, but the idea is fresh in my mind. > > How does the attached patch look? > Looks good to me ... Similar to parts of our patch set, so overlap is good that means on large parts we agree. Let's go with Dave's adaption, since it already addresses some of Linus's concerns and I start moving the pid isolation (in contrast to pid virtualization) over this. -- Hubertus
creator: Hubertus Franke
date: 2006-02-03T19:18:03-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: missed the typedef on container .. I am still on 2.6.15.2 > Couple of minor naming nitpick questions, though. Is vps/container_info > really what we want to call it? It seems to me to be the basis for a > real "container", without the _info part. > > "tsk->owner_container" That makes it sound like a pointer to the "task > owner's container". How about "owning_container"? The "container > owning this task". Or, maybe just "container"? > > Any particular reason for the "u32 id" in the vps_info struct as opposed > to one of the more generic types? Do we want to abstract this one in > the same way we do pid_t? > > The "host" in "host_container_info" doesn't mean much to me. Though, I > guess it has some context in the UML space. Would "init_container_info" > or "root_container_info" be more descriptive? > > Lastly, is this a place for krefs? I don't see a real need for a > destructor yet, but the idea is fresh in my mind. > > How does the attached patch look? here is that adapted .... Index: linux-2.6.15/include/linux/container.h ============================================================ ======= --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ linux-2.6.15/include/linux/container.h 2006-02-03 14:52:12.000000000 -0500 @@ -0,0 +1,42 @@ +#ifndef __LINUX_CONTAINER_H_ +#define __LINUX_CONTAINER_H_ + +#include <asm/types.h> +#include <asm/atomic.h> + +struct task_struct; + +struct container { + u32 id; + struct task_struct *init_task; + atomic_t refcnt; +}; + +extern struct container root_container; + +static inline struct container* +get_container(struct container *container) +{ + atomic_inc(&container->refcnt); + return container; +} + +static inline void put_container(struct container *container) +{ + atomic_dec(&container->refcnt); +} + +#include <linux/sched.h> +#include <asm/current.h> + +static inline struct container* +set_current_container(struct container *container) +{ + struct container *ret; + + ret = current->container; + current->container = container; +…
creator: Hubertus Franke
date: 2006-02-03T19:56:02-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Fri, Feb 03, 2006 at 10:34:01AM -0800, Dave Hansen wrote: > > Lastly, is this a place for krefs? I don't see a real need for a > destructor yet, but the idea is fresh in my mind. Well, what happens when you drop the last reference to this container? Right now, your patch doesn't cause anything to happen, and if that's acceptable, then fine, you don't need to use a struct kref. But if not, then why have a reference count at all? :) thanks, greg k-h
creator: Greg KH
date: 2006-02-03T20:19:45-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Greg KH wrote: > On Fri, Feb 03, 2006 at 10:34:01AM -0800, Dave Hansen wrote: > >>Lastly, is this a place for krefs? I don't see a real need for a >>destructor yet, but the idea is fresh in my mind. > > > Well, what happens when you drop the last reference to this container? > Right now, your patch doesn't cause anything to happen, and if that's > acceptable, then fine, you don't need to use a struct kref. > > But if not, then why have a reference count at all? :) > > thanks, > > greg k-h > Greg ... In our pid virtualization patches we removed the object automatically. I presume that Kirill does the same .... although the code for that is not released. I am working on converting our pid stuff so to follow the recommendation of Linux/Alan to just do <container,pid> isolation through pidspaces. I could release an allocation/ before hand. Based on Kirill's 1/5 patch adoption (which took Linus's & Dave's naming comments into account) How do we want to create the container? In our patch we did it through a /proc/container filesystem. Which created the container object and then on fork/exec switched over. How about an additional sys_exec_container( exec_args + "container_name"). This does all the work like exec, but creates new container with name "..." and attaches task to new container. If name exists, an error -EEXIST will be raised ! -- Hubertus
creator: Hubertus Franke
date: 2006-02-03T20:34:45-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: >> Do you have any other ideas/comments on this? >> I will send additional IPC/filesystems virtualization patches a bit later. > > I think that a patch like this - particularly just the 1/5 part - makes > total sense, because regardless of any other details of virtualization, > every single scheme is going to need this. > > So I think at least 1/5 (and quite frankly, 2-3/5 look that way too) are > things that we can (and probably should) merge quickly, so that people can > then actually look at the differences in the places that they may actually > disagree about. Can we merge also proc/sysfs/network/netfilters virtualization? > In other words, I personally would have called them "container" or > something similar, rather than "vps_info". See? From a logical > implementation standpoint, the fact that it is right now most commonly > used for VPS hosting is totally irrelevant to the code, no? > > (And hey, maybe your "vps" means something different. In which case my > argument makes even more sense ;) virtual private sandbox :) Actually, we call them "virtual environments" (VE) in OpenVZ. It is more than abstract and have a nice brief name. If this suits you - I will be happy to commit patches as is :) other variants: virtual context (vc, vctx), virtual containers (vc). I personally don't like "container", since it is too long and I see no good abreviations for this... Kirill
creator: dev
date: 2006-02-05T14:52:53-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: > I just did a global s/vps/container/ and it looks pretty reasonable, at > least from my point of view. > > Couple of minor naming nitpick questions, though. Is vps/container_info > really what we want to call it? It seems to me to be the basis for a > real "container", without the _info part. Can be ommited. > "tsk->owner_container" That makes it sound like a pointer to the "task > owner's container". How about "owning_container"? The "container > owning this task". Or, maybe just "container"? This is why I don't like "container" name. Please, also note, in OpenVZ we have 2 pointers on task_struct: One is owner of a task (owner_env), 2nd is a current context (exec_env). exec_env pointer is used to avoid adding of additional argument to all the functions where current context is required. Linus, does such approach makes sense to you or you prefer us to add additional args to functions where container pointer is needed? This looks undersiable for embedded guys and increases stack usage/code size when virtualization is off, which doesn't look good for me. > Any particular reason for the "u32 id" in the vps_info struct as opposed > to one of the more generic types? Do we want to abstract this one in > the same way we do pid_t? VPS ID is passed to/from user space APIs and when you have a cluster with different archs and VPSs it is better to have something in common for managing this. > The "host" in "host_container_info" doesn't mean much to me. Though, I > guess it has some context in the UML space. Would "init_container_info" > or "root_container_info" be more descriptive? init_container? (Again, this is why container doesn't sound for me :) ) > Lastly, is this a place for krefs? I don't see a real need for a > destructor yet, but the idea is fresh in my mind. I don't see much need for krefs, do you? In OpenVZ we have 2-level refcounting (mentioned recently by Linus as in mm). Process counter is used to decide when container should collapse/cleanuped and real refcoun…
creator: dev
date: 2006-02-05T15:05:44-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: > On Fri, Feb 03, 2006 at 10:34:01AM -0800, Dave Hansen wrote: >> Lastly, is this a place for krefs? I don't see a real need for a >> destructor yet, but the idea is fresh in my mind. > > Well, what happens when you drop the last reference to this container? > Right now, your patch doesn't cause anything to happen, and if that's > acceptable, then fine, you don't need to use a struct kref. > > But if not, then why have a reference count at all? :) Please note, this patch introduces only small parts of it. It doesn't introduce the code which creates containers/destroys them etc. As I mentioned in another email: In OpenVZ we have 2-level refcounting (mentioned recently by Linus as in mm). Process counter is used to decide when container should collapse/cleanuped and real refcounter is used to free the structures which can be referenced from somewhere else. Kirill
creator: dev
date: 2006-02-05T15:10:04-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: > How do we want to create the container? > In our patch we did it through a /proc/container filesystem. > Which created the container object and then on fork/exec switched over. this doesn't look good for a full virtualization solution, since proc should be virtualized as well :) > How about an additional sys_exec_container( exec_args + "container_name"). > This does all the work like exec, but creates new container > with name "..." and attaches task to new container. > If name exists, an error -EEXIST will be raised ! Why do you need exec? Kirill
creator: dev
date: 2006-02-05T15:11:54-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Kirill Korotaev wrote: >> How do we want to create the container? >> In our patch we did it through a /proc/container filesystem. >> Which created the container object and then on fork/exec switched over. > > this doesn't look good for a full virtualization solution, since proc > should be virtualized as well :) Just lazy's man's development version of a faked sys_call to create the container without having to go through all architectures ... Nothing permanent.. > >> How about an additional sys_exec_container( exec_args + >> "container_name"). >> This does all the work like exec, but creates new container >> with name "..." and attaches task to new container. >> If name exists, an error -EEXIST will be raised ! > > Why do you need exec? (a) how do you create/destroy a container (b) how do you attach yourself to it? -- Hubertus
creator: Hubertus Franke
date: 2006-02-05T15:39:48-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Fri, 2006-02-03 at 19:58 +0300, Kirill Korotaev wrote: > +static inline vps_t get_vps(vps_t vps) > +{ > + atomic_inc(&vps->refcnt); > + return vps; > +} > + > +static inline void put_vps(vps_t vps) > +{ > + atomic_dec(&vps->refcnt); > +} I'm not too sure about the refcounting here .. you never destroy the object ? Also, why introduce your own refcounting mecanism instead of using existing ones ? You could probably use at least a kref to get a nice refcount + destructor instead of home made atomics based. Maybe some higher level structure if you think it makes sense (not too sure what this virtualization stuff is about so I can't comment on what data structure is appropriate here). Cheers, Ben.
creator: Benjamin Herrenschmid
date: 2006-02-06T00:56:31-00:00

Re: [RFC][PATCH 3/5] Virtualization/containers: UTSNAME
type: item
description: I am disturbed by the introduction of #defines like current_vps() and vps_utsname. Magic lower case #defines are usually a bad idea. These defines hide the cost of the operations you are performing. At that point you might as well name the thing system_utsname so you don't have to change the code. And of course you failed to change several references to system_utsname. Eric
creator: ebiederm
date: 2006-02-06T08:21:13-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Kirill Korotaev <dev@openvz.org> writes: > This patch introduces some abstract container/VPS kernel structure and tiny > amount of operations on it. > > Patches following this one will be used for virtualization of some resources > based on this container infrastructure, including those VPIDs patches I sent > before. > > What is important here is that: > - each VPS has unique ID > - each process in kernel can belong to one VPS only Next time could you please a diffstat for your patches, Thanks Eric
creator: ebiederm
date: 2006-02-06T08:31:49-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Linus Torvalds <torvalds@osdl.org> writes: > On Fri, 3 Feb 2006, Kirill Korotaev wrote: >> >> Do you have any other ideas/comments on this? >> I will send additional IPC/filesystems virtualization patches a bit later. > > I think that a patch like this - particularly just the 1/5 part - makes > total sense, because regardless of any other details of virtualization, > every single scheme is going to need this. I strongly disagree with this approach. I think Al Viro got it right when he created a separate namespace for filesystems. First this presumes an all or nothing interface. But that is not what people are doing. Different people want different subsets of the functionality. For the migration work I am doing having multiple meanings for the same uid isn't interesting. Secondly by implementing this in one big chunk there is no migration path when you want to isolate an additional part of the kernel interface. So I really think an approach that allows for incremental progress that allows for different subsets of this functionality to be used is a better approach. In clone we already have a perfectly serviceable interface for that and I have seen no one refute that. I'm not sure I have seen anyone get it though. My apologies for the late reply I didn't see this thread until just a couple of minutes ago. linux-kernel can be hard to follow when you aren't cc'd. Patches hopefully sometime in the next 24hours. So hopefully conversation can be carried forward in a productive manner. Eric
creator: ebiederm
date: 2006-02-06T08:39:03-00:00

Re: [RFC][PATCH 3/5] Virtualization/containers: UTSNAME
type: item
description: > I am disturbed by the introduction of #defines like current_vps() and > vps_utsname. > > Magic lower case #defines are usually a bad idea. It is not magic defines, this is done intentionally. You can take a more detailed view into OpenVZ sources, but the idea is to make kernel compilable without virtualization. When virtualization is OFF all this macros are defined to trivial variables/defines which make it an old good kernel. For example current_vps() should be (&init_vps), i.e. host system environment only. vps_utsname will be defined as system_utsname and so on. > These defines hide the cost of the operations you are performing. > At that point you might as well name the thing system_utsname > so you don't have to change the code. You mean to have variable and define with the same names? it is not always good. It works fine, when both are defined in the same file, but poorly when it is scattered all around... > And of course you failed to change several references to > system_utsname. which one? Maybe intentionally? ;-) Kirill
creator: dev
date: 2006-02-06T08:51:32-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: >>I think that a patch like this - particularly just the 1/5 part - makes >>total sense, because regardless of any other details of virtualization, >>every single scheme is going to need this. > I strongly disagree with this approach. I think Al Viro got it > right when he created a separate namespace for filesystems. These patch set introduces separate namespaces as those in filesystems. What exactly you don't like in this approach? Can you be more specific? > First this presumes an all or nothing interface. But that is not > what people are doing. Different people want different subsets > of the functionality. For the migration work I am doing having > multiple meanings for the same uid isn't interesting. What do you mean by that? That you don't care about virtualization of UIDs? So your migration doesn't care at all whether 2 systems have same uids? Do you keep /etc/passwd in sync when do migration? Only full virtualization allows to migrate applications without bugs and different effects. > Secondly by implementing this in one big chunk there is no > migration path when you want to isolate an additional part of the > kernel interface. > > So I really think an approach that allows for incremental progress > that allows for different subsets of this functionality to > be used is a better approach. In clone we already have > a perfectly serviceable interface for that and I have > seen no one refute that. I'm not sure I have seen anyone > get it though. Just introduce config option for each virtualization functionality. That's it. > My apologies for the late reply I didn't see this thread until > just a couple of minutes ago. linux-kernel can be hard to > follow when you aren't cc'd. > > > Patches hopefully sometime in the next 24hours. So hopefully > conversation can be carried forward in a productive manner. Ok. I will remake them either :) Kirill
creator: dev
date: 2006-02-06T08:58:39-00:00

Re: Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: >>+static inline vps_t get_vps(vps_t vps) >>+{ >>+ atomic_inc(&vps->refcnt); >>+ return vps; >>+} >>+ >>+static inline void put_vps(vps_t vps) >>+{ >>+ atomic_dec(&vps->refcnt); >>+} > > > I'm not too sure about the refcounting here .. you never destroy the > object ? Also, why introduce your own refcounting mecanism instead of > using existing ones ? You could probably use at least a kref to get a > nice refcount + destructor instead of home made atomics based. Maybe > some higher level structure if you think it makes sense (not too sure > what this virtualization stuff is about so I can't comment on what data > structure is appropriate here). I replied to this in another email. Briefly: this patch set doesn't introduce VPS/container creation/destroy interface yet. Only small parts. krefs are not needed here. Kirill
creator: dev
date: 2006-02-06T09:01:02-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: > Just lazy's man's development version of a faked sys_call to create the > container > without having to go through all architectures ... > Nothing permanent.. >>> How about an additional sys_exec_container( exec_args + >>> "container_name"). >>> This does all the work like exec, but creates new container >>> with name "..." and attaches task to new container. >>> If name exists, an error -EEXIST will be raised ! >> >> >> Why do you need exec? > > > (a) how do you create/destroy a container > (b) how do you attach yourself to it? a) create via syscall. destroyed when the last process dies in container. b) syscall which changes container. you need to close unneeded resources, change context and fork(). That's it. The whole process is exactly the same as if you changes UID.
creator: dev
date: 2006-02-06T09:06:40-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Kirill Korotaev <dev@sw.ru> writes: >>> I think that a patch like this - particularly just the 1/5 part - makes total >>> sense, because regardless of any other details of virtualization, every >>> single scheme is going to need this. >> I strongly disagree with this approach. I think Al Viro got it >> right when he created a separate namespace for filesystems. > These patch set introduces separate namespaces as those in filesystems. > What exactly you don't like in this approach? Can you be more specific? That you placed the namespaces in a separate structure from task_struct. That part seems completely unnecessary, that and the addition of a global id in a completely new namespace that will be a pain to virtualize when it's time comes. >> First this presumes an all or nothing interface. But that is not >> what people are doing. Different people want different subsets >> of the functionality. For the migration work I am doing having >> multiple meanings for the same uid isn't interesting. > What do you mean by that? That you don't care about virtualization of UIDs? So > your migration doesn't care at all whether 2 systems have same uids? Do you keep > /etc/passwd in sync when do migration? It is already in sync. When the only interesting target is inside of a cluster it is trivial to handle. In addition frequently there is only one job on a given machine at a time so there is no other interesting user of UIDs. > Only full virtualization allows to migrate applications without bugs and > different effects. The requirement is that for every global identifier you have a namespace where you will not have a conflict in definition when you restore that identifier. (Either that or you have to update all references to that identifier which is usually to hideous to take seriously) Virtualization has nothing to do with it. It is all about non-conflicting names. The kernel already has everything virtualized. >> Secondly by implementing this in one big chunk there is no >> migr…
creator: ebiederm
date: 2006-02-06T09:19:37-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Sun, 2006-02-05 at 18:05 +0300, Kirill Korotaev wrote: > > "tsk->owner_container" That makes it sound like a pointer to the "task > > owner's container". How about "owning_container"? The "container > > owning this task". Or, maybe just "container"? > This is why I don't like "container" name. I worry that using something like "vps" obfuscates the real meaning a bit. The reason that "owner_vps" doesn't sound weird is that people, by default, usually won't understand what a "vps" is. (if you like acronyms a lot, I'm sure I can find a job for you at IBM or in the US military :) > Please, also note, in OpenVZ we have 2 pointers on task_struct: > One is owner of a task (owner_env), 2nd is a current context (exec_env). > exec_env pointer is used to avoid adding of additional argument to all > the functions where current context is required. That makes sense. However, are there many cases in the kernel where a task ends up doing something temporary like this: tsk->exec_vnc = bar; do_something_here(task); tsk->exec_vnc = foo; If that's the case very often, we probably want to change the APIs, just to make the common action explicit. If it never happens, or is a rarity, I think it should be just fine. > > Any particular reason for the "u32 id" in the vps_info struct as opposed > > to one of the more generic types? Do we want to abstract this one in > > the same way we do pid_t? > VPS ID is passed to/from user space APIs and when you have a cluster > with different archs and VPSs it is better to have something in common > for managing this. I guess it does keep you from running into issues with mixing 32 and 64-bit processes. But, haven't we solved those problems already? Is it just a pain? > > Lastly, is this a place for krefs? I don't see a real need for a > > destructor yet, but the idea is fresh in my mind. > I don't see much need for krefs, do you? > In OpenVZ we have 2-level refcounting (mentioned recently by Linus as in > mm). Process counter is used to decide when…
creator: Dave Hansen
date: 2006-02-06T16:35:33-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Mon, 2006-02-06 at 02:19 -0700, Eric W. Biederman wrote: > That you placed the namespaces in a separate structure from > task_struct. > That part seems completely unnecessary, that and the addition of a > global id in a completely new namespace that will be a pain to > virtualize > when it's time comes. Could you explain a bit why the container ID would need to be virtualized? -- Dave
creator: Dave Hansen
date: 2006-02-06T16:37:54-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Hello, > I worry that using something like "vps" obfuscates the real meaning a > bit. The reason that "owner_vps" doesn't sound weird is that people, by > default, usually won't understand what a "vps" is. container or context sounds the same :) it is impossible to feel this notion naturally without getting into details. IMHO. > (if you like acronyms a lot, I'm sure I can find a job for you at IBM or > in the US military :) We can talk about it separetely :))) >>Please, also note, in OpenVZ we have 2 pointers on task_struct: >>One is owner of a task (owner_env), 2nd is a current context (exec_env). >>exec_env pointer is used to avoid adding of additional argument to all >>the functions where current context is required. > > That makes sense. However, are there many cases in the kernel where a > task ends up doing something temporary like this: > > tsk->exec_vnc = bar; > do_something_here(task); > tsk->exec_vnc = foo; > > If that's the case very often, we probably want to change the APIs, just > to make the common action explicit. If it never happens, or is a > rarity, I think it should be just fine. It is quite rare. In IRQ, softIRQ, TCP/IP stack and some timers. Not much. >>VPS ID is passed to/from user space APIs and when you have a cluster >>with different archs and VPSs it is better to have something in common >>for managing this. > I guess it does keep you from running into issues with mixing 32 and > 64-bit processes. But, haven't we solved those problems already? Is it > just a pain? VPSs can live in clusters. It is good to have one VPS ID space. Kirill
creator: dev
date: 2006-02-06T16:50:08-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: On Sun, 5 Feb 2006, Kirill Korotaev wrote: > > Please, also note, in OpenVZ we have 2 pointers on task_struct: > One is owner of a task (owner_env), 2nd is a current context (exec_env). > exec_env pointer is used to avoid adding of additional argument to all the > functions where current context is required. That naming _has_ to change. "exec" has a very clear meaning in unix: it talks about the notion of switching to another process image, or perhaps the bit that says that a file contains an image that can be executed. It has nothing to do with "current". What you seem to be talking about is the _effective_ environment. Ie the same way we have "uid" and "euid", you'd have a "container" and the "effective container". The "owner" name also makes no sense. The security context doesn't "own" tasks. A task is _part_ of a context. So if some people don't like "container", how about just calling it "context"? The downside of that name is that it's very commonly used in the kenel, because a lot of things have "contexts". That's why "container" would be a lot better. I'd suggest current->container - the current EFFECTIVE container current->master_container - the "long term" container. (replace "master" with some other non-S&M term if you want) (It would make sense to just have the prepend-"e" semantics of uid/gid, but the fact is, "euid/egid" has a long unix history and is readable only for that reason. The same wouldn't be true of containers. And "effective_container" is probably too long to use for the field that is actually the _common_ case. Thus the above suggestion). Linus
creator: Linus Torvalds
date: 2006-02-06T16:56:52-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: >>Please, also note, in OpenVZ we have 2 pointers on task_struct: >>One is owner of a task (owner_env), 2nd is a current context (exec_env). >>exec_env pointer is used to avoid adding of additional argument to all the >>functions where current context is required. > > > That naming _has_ to change. I agree. > "exec" has a very clear meaning in unix: it talks about the notion of > switching to another process image, or perhaps the bit that says that a > file contains an image that can be executed. It has nothing to do with > "current". > What you seem to be talking about is the _effective_ environment. Ie the > same way we have "uid" and "euid", you'd have a "container" and the > "effective container". agree on this either. Good point. > The "owner" name also makes no sense. The security context doesn't "own" > tasks. A task is _part_ of a context. > So if some people don't like "container", how about just calling it > "context"? The downside of that name is that it's very commonly used in > the kenel, because a lot of things have "contexts". That's why "container" > would be a lot better. > > I'd suggest > > current->container - the current EFFECTIVE container > current->master_container - the "long term" container. > > (replace "master" with some other non-S&M term if you want) maybe task_container? i.e. where task actually is. Sounds good for you? The only problem with such names I see, that task will be an exception then compared to other objects. I mean, on other objects field "container" will mean the container which object is part of. But for tasks this will mean effective one. Only tasks need these 2 containers pointers and I would prefer having the common one to be called simply "container". Maybe then current->econtainer - effective container current->container - "long term" container > (It would make sense to just have the prepend-"e" semantics of uid/gid, > but the fact is, "euid/egid" has a long unix history and is readable only > for that reason. The sa…
creator: dev
date: 2006-02-06T17:19:54-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Dave Hansen <haveblue@us.ibm.com> writes: > On Mon, 2006-02-06 at 02:19 -0700, Eric W. Biederman wrote: >> That you placed the namespaces in a separate structure from >> task_struct. >> That part seems completely unnecessary, that and the addition of a >> global id in a completely new namespace that will be a pain to >> virtualize >> when it's time comes. > > Could you explain a bit why the container ID would need to be > virtualized? As someone said to me a little bit ago, for migration or checkpointing ultimately you have to capture the entire user/kernel interface if things are going to work properly. Now if we add this facility to the kernel and it is a general purpose facility. It is only a matter of time before we need to deal with nested containers. Not considering the case of having nested containers now is just foolish. Maybe we don't have to implement it yet but not considering it is silly. As far as I can tell there is a very reasonable chance that when we are complete there is a very reasonable chance that software suspend will just be a special case of migration, done complete in user space. That is one of the more practical examples I can think of where this kind of functionality would be used. Eric
creator: ebiederm
date: 2006-02-06T18:37:07-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: > As someone said to me a little bit ago, for migration or checkpointing > ultimately you have to capture the entire user/kernel interface if > things are going to work properly. Now if we add this facility to > the kernel and it is a general purpose facility. It is only a matter > of time before we need to deal with nested containers. Fully virtualized container is not a matter of virtualized ID - it is the easiest thing to do actually, but a whole global problem of other resources virtualization. We can ommit ID for now, if you like it more. > Not considering the case of having nested containers now is just foolish. > Maybe we don't have to implement it yet but not considering it is silly. No one told that it is not considered. In fact PID virtualization send both by IBM/us is abstract and doesn't care whether containers are nested or not. > As far as I can tell there is a very reasonable chance that when we > are complete there is a very reasonable chance that software suspend > will just be a special case of migration, done complete in user space. > That is one of the more practical examples I can think of where this > kind of functionality would be used. Kirill
creator: dev
date: 2006-02-06T19:30:31-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Kirill Korotaev wrote: >> How do we want to create the container? >> In our patch we did it through a /proc/container filesystem. >> Which created the container object and then on fork/exec switched over. > > this doesn't look good for a full virtualization solution, since proc > should be virtualized as well :) Well, /proc should be "virtualized" or "isolated", how do you expect a container to work correctly ? plenty of user space tools depend on it. C.
creator: Cedric Le Goater
date: 2006-02-06T22:31:49-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Eric W. Biederman wrote: > As someone said to me a little bit ago, for migration or checkpointing > ultimately you have to capture the entire user/kernel interface if > things are going to work properly. Now if we add this facility to > the kernel and it is a general purpose facility. It is only a matter > of time before we need to deal with nested containers. > > Not considering the case of having nested containers now is just foolish. > Maybe we don't have to implement it yet but not considering it is silly. That could be restricted. Today, process groups are not nested. Why do you think nested containers are inevitable ? > As far as I can tell there is a very reasonable chance that when we > are complete there is a very reasonable chance that software suspend > will just be a special case of migration, done complete in user space. Being able to sofware suspend one container among many would be a very interesting feature to have. C.
creator: Cedric Le Goater
date: 2006-02-06T22:40:55-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Linus Torvalds wrote: > So if some people don't like "container", how about just calling it > "context"? The downside of that name is that it's very commonly used in > the kenel, because a lot of things have "contexts". That's why "container" > would be a lot better. > > I'd suggest > > current->container - the current EFFECTIVE container > current->master_container - the "long term" container. > > (replace "master" with some other non-S&M term if you want) Hmm. You actually need a linked list, otherwise you have replaced a one level flat structure with a two level one, and you miss out on some of the applications. VServer uses a special structure for this. Sam.
creator: Sam Vilain
date: 2006-02-07T00:28:08-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Cedric Le Goater <clg@fr.ibm.com> writes: > Eric W. Biederman wrote: > >> As someone said to me a little bit ago, for migration or checkpointing >> ultimately you have to capture the entire user/kernel interface if >> things are going to work properly. Now if we add this facility to >> the kernel and it is a general purpose facility. It is only a matter >> of time before we need to deal with nested containers. >> >> Not considering the case of having nested containers now is just foolish. >> Maybe we don't have to implement it yet but not considering it is silly. > > That could be restricted. Today, process groups are not nested. Why do you > think nested containers are inevitable ? process groups are a completely different kind of beast. A closer analogy are hierarchical name spaces and mounts. If we didn't need things like waitpid outside one pid namespace to wait for a nested namespace they would be complete disjoint and the implementation would be trivial. >> As far as I can tell there is a very reasonable chance that when we >> are complete there is a very reasonable chance that software suspend >> will just be a special case of migration, done complete in user space. > > Being able to sofware suspend one container among many would be a very > interesting feature to have. That is what checkpointing. And that is simply the persistent form of migration. Eric
creator: ebiederm
date: 2006-02-07T01:57:05-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: >> So if some people don't like "container", how about just calling it >> "context"? The downside of that name is that it's very commonly used >> in the kenel, because a lot of things have "contexts". That's why >> "container" would be a lot better. >> >> I'd suggest >> >> current->container - the current EFFECTIVE container >> current->master_container - the "long term" container. >> >> (replace "master" with some other non-S&M term if you want) > > > Hmm. You actually need a linked list, otherwise you have replaced a one > level flat structure with a two level one, and you miss out on some of > the applications. VServer uses a special structure for this. Nope! :) This is pointer to current/effective container, which can be anywhere in the hierarchy. list should be inside container struct. Kirill
creator: dev
date: 2006-02-07T12:19:14-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: >>>How do we want to create the container? >>>In our patch we did it through a /proc/container filesystem. >>>Which created the container object and then on fork/exec switched over. >> >>this doesn't look good for a full virtualization solution, since proc >>should be virtualized as well :) > > > Well, /proc should be "virtualized" or "isolated", how do you expect a > container to work correctly ? plenty of user space tools depend on it. Sorry, not actually understand your question... :( There is not much problems with virtualization of proc. It can be virtualized correctly, so that tools are still working. For example, in OpenVZ /proc has 2 trees - global and local. global tree contains the entries which are visiable in all containers. and local tree - only those which are visible to containers. PIDs are shown also only those which present in container. Kirill
creator: dev
date: 2006-02-07T12:25:51-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Kirill Korotaev wrote: >>> I'd suggest >>> >>> current->container - the current EFFECTIVE container >>> current->master_container - the "long term" container. >>> >>> (replace "master" with some other non-S&M term if you want) >> Hmm. You actually need a linked list, otherwise you have replaced a one >> level flat structure with a two level one, and you miss out on some of >> the applications. VServer uses a special structure for this. > > Nope! :) This is pointer to current/effective container, which can be > anywhere in the hierarchy. list should be inside container struct. So why store anything other than the effective container in the task? Sam.
creator: Sam Vilain
date: 2006-02-07T22:21:57-00:00

swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
type: item
description: Hi! > > Could you explain a bit why the container ID would need to be > > virtualized? > > As someone said to me a little bit ago, for migration or checkpointing > ultimately you have to capture the entire user/kernel interface if > things are going to work properly. Now if we add this facility to > the kernel and it is a general purpose facility. It is only a matter > of time before we need to deal with nested containers. > > Not considering the case of having nested containers now is just foolish. > Maybe we don't have to implement it yet but not considering it is silly. > > As far as I can tell there is a very reasonable chance that when we > are complete there is a very reasonable chance that software suspend > will just be a special case of migration, done complete in user space. > That is one of the more practical examples I can think of where this > kind of functionality would be used. Well, for now software suspend is done at very different level (it snapshots complete kernel state), but being able to use migration for this is certainly nice option. BTW you could do whole-machine-migration now with uswsusp; but you'd need identical hardware and it would take a bit long... Pavel -- Thanks, Sharp!
creator: Pavel Machek
date: 2006-02-08T21:54:13-00:00

Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
type: item
description: Pavel Machek <pavel@ucw.cz> writes: > Well, for now software suspend is done at very different level > (it snapshots complete kernel state), but being able to use > migration for this is certainly nice option. > > BTW you could do whole-machine-migration now with uswsusp; but you'd > need identical hardware and it would take a bit long... Right part of the goal is with doing it as we are doing it is that we can define what the interesting state is. Replacing software suspend is not an immediate goal but I think it is a worthy thing to target. In part because if we really can rip things out of the kernel store them in a portable format and restore them we will also have the ability to upgrade the kernel with out stopping user space applications... But being able to avoid the uninteresting parts, and having the policy complete controlled outside the kernel are the big wins we are shooting for. Eric
creator: ebiederm
date: 2006-02-09T18:20:13-00:00

Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
type: item
description: On Feb 09, 2006, at 13:20, Eric W. Biederman wrote: > Pavel Machek <pavel@ucw.cz> writes: >> Well, for now software suspend is done at very different level (it >> snapshots complete kernel state), but being able to use migration >> for this is certainly nice option. >> >> BTW you could do whole-machine-migration now with uswsusp; but >> you'd need identical hardware and it would take a bit long... > > Right part of the goal is with doing it as we are doing it is that > we can define what the interesting state is. > > Replacing software suspend is not an immediate goal but I think it > is a worthy thing to target. In part because if we really can rip > things out of the kernel store them in a portable format and > restore them we will also have the ability to upgrade the kernel > with out stopping user space applications... > > But being able to avoid the uninteresting parts, and having the > policy complete controlled outside the kernel are the big wins we > are shooting for. <wishful thinking> I can see another extension to this functionality. With appropriate changes it might also be possible to have a container exist across multiple computers using some cluster code for synchronization and fencing. The outermost container would be the system boot container, and multiple inner containers would use some sort of network- container-aware cluster filesystem to spread multiple vservers across multiple servers, distributing CPU and network load appropriately. </wishful thinking> Cheers, Kyle Moffett -- I have yet to see any problem, however complicated, which, when you looked at it in the right way, did not become still more complicated. -- Poul Anderson
creator: Kyle Moffett
date: 2006-02-10T00:21:50-00:00

Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
type: item
description: Kyle Moffett wrote: > <wishful thinking> > I can see another extension to this functionality. With appropriate > changes it might also be possible to have a container exist across > multiple computers using some cluster code for synchronization and > fencing. The outermost container would be the system boot container, > and multiple inner containers would use some sort of network- > container-aware cluster filesystem to spread multiple vservers across > multiple servers, distributing CPU and network load appropriately. > </wishful thinking> Yeah. If you fudged/virtualised /dev/random, the system clock, etc you could even have Tandem-style transparent High Availability. </more wishful thinking> Actually there is relatively little difference between a NUMA system and a cluster... Sam.
creator: Sam Vilain
date: 2006-02-10T04:31:08-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Hi. On Tuesday 07 February 2006 04:37, Eric W. Biederman wrote: > Dave Hansen <haveblue@us.ibm.com> writes: > > On Mon, 2006-02-06 at 02:19 -0700, Eric W. Biederman wrote: > >> That you placed the namespaces in a separate structure from > >> task_struct. > >> That part seems completely unnecessary, that and the addition of a > >> global id in a completely new namespace that will be a pain to > >> virtualize > >> when it's time comes. > > > > Could you explain a bit why the container ID would need to be > > virtualized? > > As someone said to me a little bit ago, for migration or checkpointing > ultimately you have to capture the entire user/kernel interface if > things are going to work properly. Now if we add this facility to > the kernel and it is a general purpose facility. It is only a matter > of time before we need to deal with nested containers. > > Not considering the case of having nested containers now is just foolish. > Maybe we don't have to implement it yet but not considering it is silly. > > As far as I can tell there is a very reasonable chance that when we > are complete there is a very reasonable chance that software suspend > will just be a special case of migration, done complete in user space. > That is one of the more practical examples I can think of where this > kind of functionality would be used. Am I missing something? I though migration referred only to userspace processes. Software suspend on the other hand, deals with the whole system, of which process data/context is only a part. Regards, Nigel
creator: Nigel Cunningham
date: 2006-02-10T05:40:56-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: Nigel Cunningham <ncunningham@cyclades.com> writes: > Am I missing something? I though migration referred only to userspace > processes. Software suspend on the other hand, deals with the whole system, > of which process data/context is only a part. The problem domain is user process and the kernel state they depend on. Implementation wise we are looking at two totally different problems. However the effects should be similar if the set of processes to migrate are all of the processes in the system. For most of the interesting cases migration does not need to be that ambitious. Eric
creator: ebiederm
date: 2006-02-10T06:01:26-00:00

Re: Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
type: item
description: Sam Vilain wrote: > Kyle Moffett wrote: >> <wishful thinking> >> I can see another extension to this functionality. With appropriate >> changes it might also be possible to have a container exist across >> multiple computers using some cluster code for synchronization and >> fencing. The outermost container would be the system boot container, >> and multiple inner containers would use some sort of network- >> container-aware cluster filesystem to spread multiple vservers across >> multiple servers, distributing CPU and network load appropriately. >> </wishful thinking> > > Yeah. If you fudged/virtualised /dev/random, the system clock, etc you > could even have Tandem-style transparent High Availability. > </more wishful thinking> Could you please explain, why you want to virtualize /dev/random? Tnank you, Vasily Averin Virtuozzo Linux Kernel Team
creator: vaverin
date: 2006-02-10T06:23:56-00:00

Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
type: item
description: On Feb 09, 2006, at 23:31, Sam Vilain wrote: > Kyle Moffett wrote: >> <wishful thinking> >> I can see another extension to this functionality. With >> appropriate changes it might also be possible to have a container >> exist across multiple computers using some cluster code for >> synchronization and fencing. The outermost container would be >> the system boot container, and multiple inner containers would >> use some sort of network- container-aware cluster filesystem to >> spread multiple vservers across multiple servers, distributing >> CPU and network load appropriately. >> </wishful thinking> > > Yeah. If you fudged/virtualised /dev/random, the system clock, etc > you could even have Tandem-style transparent High Availability. > </more wishful thinking> > > Actually there is relatively little difference between a NUMA > system and a cluster... Yeah, a cluster is just a multi-tiered multi-address-space RNUMA (*Really* Non-Uniform Memory Architecture) :-D. With some kind of RDMA infiniband card and the right kernel and userspace tools, that kind of cluster could be practical. I _suspect_ (never really considered the issue before) that a properly virtualized container could even achieve extremely high fault tolerance by allowing systems to "vote" on correct output. If you synchronize /dev/random and network IO across the system correctly such that each instance of each userspace process on each system sees _exactly_ the same virtual inputs and virtual clock in the exact same order, then you could binary-compare the output of 3 different servers. If one didn't agree, it could be discarded and marked as failing. Cheers, Kyle Moffett -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. -- C.A.R. Hoare
creator: Kyle Moffett
date: 2006-02-10T08:29:57-00:00

Re: Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
type: item
description: On Fri, 2006-02-10 at 09:23 +0300, Vasily Averin wrote: > >> <wishful thinking> > >> I can see another extension to this functionality. With appropriate > >> changes it might also be possible to have a container exist across > >> multiple computers using some cluster code for synchronization and > >> fencing. The outermost container would be the system boot container, > >> and multiple inner containers would use some sort of network- > >> container-aware cluster filesystem to spread multiple vservers across > >> multiple servers, distributing CPU and network load appropriately. > >> </wishful thinking> > > Yeah. If you fudged/virtualised /dev/random, the system clock, etc you > > could even have Tandem-style transparent High Availability. > > </more wishful thinking> > Could you please explain, why you want to virtualize /dev/random? When checkpointing it is important to preserve all state. If you are doing transparent highly available computing, you need to make sure all system calls get the same answers in the clones. So you would need to virtualise the entropy pool. There are likely to be dozens of other quite hard problems in the way first. Like I said, wishful thinking :-). Sam.
creator: Sam Vilain
date: 2006-02-11T02:38:18-00:00

Re: Re: swsusp done by migration (was Re: [RFC][PATCH 1/5] Virtualization/containers: startup)
type: item
description: Sam Vilain wrote: > On Fri, 2006-02-10 at 09:23 +0300, Vasily Averin wrote: >>>Yeah. If you fudged/virtualised /dev/random, the system clock, etc you >>>could even have Tandem-style transparent High Availability. >>></more wishful thinking> >>Could you please explain, why you want to virtualize /dev/random? > > When checkpointing it is important to preserve all state. If you are > doing transparent highly available computing, you need to make sure all > system calls get the same answers in the clones. So you would need to > virtualise the entropy pool. >From my point of view it is important to preserve only all the determinated state. Ok, lets we've checkpointed and saved current entropy pool. But we have not any guarantee that pool will be in the same state at the moment of first access to it after wakeuping. Because a new entropy can change it unpredictable. Am I right? Thank you, Vasily Averin Virtuozzo Linux kernel Team
creator: vaverin
date: 2006-02-11T17:29:02-00:00

Re: [RFC][PATCH 1/5] Virtualization/containers: startup
type: item
description: > So why store anything other than the effective container in the task? Effective container is used for temporary context change, e.g. when processing interrupts and need to handle skb. it is effective container for this code. just like get_fs()/set_fs() works. Original container pointer is used for external process identification, e.g. whether to show task in /proc in context of another task. Kirill
creator: dev
date: 2006-02-20T11:54:02-00:00