OpenVZ Forum - RDF feed
type: channel
items: [PATCH 1/4] Virtualization/containers: introduction
items: [PATCH 2/4] Virtualization/containers: CONFIG_CONTAINER
items: [PATCH 3/4] Virtualization/containers: UID hash
items: [PATCH 4/4] Virtualization/containers: uts name
items: Re: [PATCH 1/4] Virtualization/containers: introduction
items: The issues for agreeing on a virtualization/namespaces implementation.
items: Re: The issues for agreeing on a virtualization/namespaces implementation.

[PATCH 1/4] Virtualization/containers: introduction
type: item
description: Hello, I tried to take into account all the comments from you guys (thanks a lot for them!) and prepared a new version of virtualization patches. I will send only 4 patches today, just not to overflow everyone and keep it clear/tidy/possible to review. This patch introduces some abstract container kernel structure and a number of operations on it. The important properties of the proposed container implementation: - each container has unique ID in the system - each process in the kernel can belong to one container only - effective container pointer (econtainer()) is used on the task to avoid insertion of additional argument "container" to all functions where it is required. - kernel compilation with disabled virtualization should result in old good linux kernel Patches following this one will be used for virtualization of the kernel resources based on this container infrastructure, including those VPID patches I sent before. Every virtualized resource can be given separate config option if needed (just give me to know if it is desired). Signed-Off-By: Kirill Korotaev <dev@openvz.org> Kirill P.S. I understand that this virtualization spam can be unintersting for some of you, just give me to know if you want to be removed from CC. P.P.S. All patches are against 2.6.16-rc2-git2 and compile when virtualization=n. --- ./include/linux/container.h.vpsinfo 2006-02-06 22:35:36.000000000 +0300 +++ ./include/linux/container.h 2006-02-07 00:52:57.000000000 +0300 @@ -0,0 +1,59 @@ +#ifndef __LINUX_CONTAINER_H__ +#define __LINUX_CONTAINER_H__ + +#include <linux/config.h> +#include <asm/types.h> +#include <asm/atomic.h> + +struct task_struct; + +struct container { + u32 id; + struct task_struct *init_task; + atomic_t refcnt; +}; + +extern struct container init_container; + +#ifdef CONFIG_CONTAINER + +static inline struct container *get_container(struct container *cont) +{ + atomic_inc(&cont->refcnt); + return cont; +} + +static inline void put_container(struct container *cont) +{ + …
creator: Kirill Korotaev
date: 2006-02-06T21:55:44-00:00

[PATCH 2/4] Virtualization/containers: CONFIG_CONTAINER
type: item
description: This patch simply adds CONFIG_CONTAINER option for virtualization/containerss functionality. Per-resource config options can be added later if needed. Signed-Off-By: Kirill Korotaev <dev@openvz.org> Kirill --- ./arch/alpha/Kconfig.vkconfig 2006-02-06 22:14:50.000000000 +0300 +++ ./arch/alpha/Kconfig 2006-02-06 23:26:35.000000000 +0300 @@ -621,6 +621,8 @@ source "arch/alpha/Kconfig.debug" source "security/Kconfig" +source "kernel/Kconfig.container" + source "crypto/Kconfig" source "lib/Kconfig" --- ./arch/arm/Kconfig.vkconfig 2006-02-06 22:14:50.000000000 +0300 +++ ./arch/arm/Kconfig 2006-02-06 23:27:06.000000000 +0300 @@ -794,6 +794,8 @@ source "arch/arm/Kconfig.debug" source "security/Kconfig" +source "kernel/Kconfig.container" + source "crypto/Kconfig" source "lib/Kconfig" --- ./arch/arm26/Kconfig.vkconfig 2006-02-06 22:14:51.000000000 +0300 +++ ./arch/arm26/Kconfig 2006-02-06 23:27:14.000000000 +0300 @@ -232,6 +232,8 @@ source "arch/arm26/Kconfig.debug" source "security/Kconfig" +source "kernel/Kconfig.container" + source "crypto/Kconfig" source "lib/Kconfig" --- ./arch/cris/Kconfig.vkconfig 2006-02-06 22:14:51.000000000 +0300 +++ ./arch/cris/Kconfig 2006-02-06 23:30:51.000000000 +0300 @@ -175,6 +175,8 @@ source "arch/cris/Kconfig.debug" source "security/Kconfig" +source "kernel/Kconfig.container" + source "crypto/Kconfig" source "lib/Kconfig" --- ./arch/frv/Kconfig.vkconfig 2006-02-06 22:14:51.000000000 +0300 +++ ./arch/frv/Kconfig 2006-02-06 23:30:51.000000000 +0300 @@ -341,6 +341,8 @@ source "arch/frv/Kconfig.debug" source "security/Kconfig" +source "kernel/Kconfig.container" + source "crypto/Kconfig" source "lib/Kconfig" --- ./arch/h8300/Kconfig.vkconfig 2006-02-06 22:14:51.000000000 +0300 +++ ./arch/h8300/Kconfig 2006-02-06 23:30:51.000000000 +0300 @@ -189,6 +189,8 @@ source "arch/h8300/Kconfig.debug" source "security/Kconfig" +source "kernel/Kconfig.container" + source "crypto/Kconfig" source "lib/Kconfig" --- ./arch/i386/Kconfig.vkconfig 2006-02-06 22:15:1…
creator: Kirill Korotaev
date: 2006-02-06T22:10:05-00:00

[PATCH 3/4] Virtualization/containers: UID hash
type: item
description: This patch virtualizes UID hash, so that processes in container can use it's own UID set. Can be done as an option if some virtualization solutions do not require it. Signed-Off-By: Kirill Korotaev <dev@openvz.org> Kirill --- ./include/linux/container.h.uids 2006-02-06 23:46:40.000000000 +0300 +++ ./include/linux/container.h 2006-02-07 00:05:33.000000000 +0300 @@ -6,11 +6,14 @@ #include <asm/atomic.h> struct task_struct; +struct list_head; struct container { u32 id; struct task_struct *init_task; atomic_t refcnt; + + struct list_head *c_uid_hash; }; extern struct container init_container; --- ./kernel/user.c.uids 2006-02-06 22:15:06.000000000 +0300 +++ ./kernel/user.c 2006-02-06 23:58:06.000000000 +0300 @@ -14,6 +14,7 @@ #include <linux/bitops.h> #include <linux/key.h> #include <linux/interrupt.h> +#include <linux/container.h> /* * UID task count cache, to get fast user lookup in "alloc_uid" @@ -24,7 +25,12 @@ #define UIDHASH_SZ (1 << UIDHASH_BITS) #define UIDHASH_MASK (UIDHASH_SZ - 1) #define __uidhashfn(uid) (((uid >> UIDHASH_BITS) + uid) & UIDHASH_MASK) + +#ifdef CONFIG_CONTAINER +#define uidhashentry(uid) (econtainer()->c_uid_hash + __uidhashfn((uid))) +#else #define uidhashentry(uid) (uidhash_table + __uidhashfn((uid))) +#endif static kmem_cache_t *uid_cachep; static struct list_head uidhash_table[UIDHASH_SZ]; @@ -200,6 +206,9 @@ static int __init uid_cache_init(void) /* Insert the root user immediately (init already runs as root) */ spin_lock_irq(&uidhash_lock); +#ifdef CONFIG_CONTAINER + init_container.c_uid_hash = uidhash_table; +#endif uid_hash_insert(&root_user, uidhashentry(0)); spin_unlock_irq(&uidhash_lock);
creator: Kirill Korotaev
date: 2006-02-06T22:15:04-00:00

[PATCH 4/4] Virtualization/containers: uts name
type: item
description: This patch virtualizes uts name. main changes are done in container.h, uts_name.h, all other places are just replacement of system_utsname with uts_name. Signed-Off-By: Kirill Korotaev <dev@openvz.org> Kirill --- ./arch/alpha/kernel/osf_sys.c.utsnamex 2006-02-07 01:18:42.000000000 +0300 +++ ./arch/alpha/kernel/osf_sys.c 2006-02-07 01:18:50.000000000 +0300 @@ -402,15 +402,15 @@ osf_utsname(char __user *name) down_read(&uts_sem); error = -EFAULT; - if (copy_to_user(name + 0, system_utsname.sysname, 32)) + if (copy_to_user(name + 0, uts_name.sysname, 32)) goto out; - if (copy_to_user(name + 32, system_utsname.nodename, 32)) + if (copy_to_user(name + 32, uts_name.nodename, 32)) goto out; - if (copy_to_user(name + 64, system_utsname.release, 32)) + if (copy_to_user(name + 64, uts_name.release, 32)) goto out; - if (copy_to_user(name + 96, system_utsname.version, 32)) + if (copy_to_user(name + 96, uts_name.version, 32)) goto out; - if (copy_to_user(name + 128, system_utsname.machine, 32)) + if (copy_to_user(name + 128, uts_name.machine, 32)) goto out; error = 0; @@ -449,8 +449,8 @@ osf_getdomainname(char __user *name, int down_read(&uts_sem); for (i = 0; i < len; ++i) { - __put_user(system_utsname.domainname[i], name + i); - if (system_utsname.domainname[i] == '\0') + __put_user(uts_name.domainname[i], name + i); + if (uts_name.domainname[i] == '\0') break; } up_read(&uts_sem); @@ -608,11 +608,11 @@ asmlinkage long osf_sysinfo(int command, char __user *buf, long count) { static char * sysinfo_table[] = { - system_utsname.sysname, - system_utsname.nodename, - system_utsname.release, - system_utsname.version, - system_utsname.machine, + uts_name.sysname, + uts_name.nodename, + uts_name.release, + uts_name.version, + uts_name.machine, "alpha", /* instruction set architecture */ "dummy", /* hardware serial number */ "dummy", /* hardware manufacturer */ --- ./arch/i386/kernel/sys_i386.c.utsnamex 2006-02-07 01:18:42.000000000 +0300 +++ ./arch/i386/kernel/sys_i386.c 2006-02-07 01…
creator: Kirill Korotaev
date: 2006-02-06T22:20:49-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: On Tue, 2006-02-07 at 00:57 +0300, Kirill Korotaev wrote: > @@ -1132,6 +1133,7 @@ static task_t *copy_process(unsigned lon > p->ioprio = current->ioprio; > > SET_LINKS(p); > + (void)get_container(p->container); > if (unlikely(p->ptrace & PT_PTRACED)) > __ptrace_link(p, current->parent); This entire patch looks nice and very straightforward, except for this bit. :) The "(void)" bit isn't usual kernel coding style. You can probably kill it. BTW, why does get_container() return the container argument? get_task_struct(), for instance is just a do{}while(0) loop, so it doesn't have a return value. Is there some magic later on in your patch set that utilizes this? One other really minor thing: I usually try to do is keep the ! CONFIG_FOO functions static inlines, just like the full versions. The advantage is that you get some compile-time type checking, even when your CONFIG option is off. -- Dave
creator: Dave Hansen
date: 2006-02-06T23:00:58-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Kirill Korotaev <dev@openvz.org> writes: > Hello, > > I tried to take into account all the comments from you guys (thanks a lot for > them!) and prepared a new version of virtualization patches. I will send only 4 > patches today, just not to overflow everyone and keep it clear/tidy/possible to > review. > > This patch introduces some abstract container kernel structure and a number of > operations on it. > > The important properties of the proposed container implementation: > - each container has unique ID in the system What namespace does this ID live in? > - each process in the kernel can belong to one container only Reasonable. > - effective container pointer (econtainer()) is used on the task to avoid > insertion of additional argument "container" to all functions where it is > required. Why is that desirable? > - kernel compilation with disabled virtualization should result in old good > linux kernel A reasonable goal. Why do we need a container structure to hold pointers to other pointers? > Patches following this one will be used for virtualization of the kernel > resources based on this container infrastructure, including those VPID patches I > sent before. Every virtualized resource can be given separate config option if > needed (just give me to know if it is desired). > > Signed-Off-By: Kirill Korotaev <dev@openvz.org> > > Kirill > > P.S. I understand that this virtualization spam can be unintersting for some of > you, just give me to know if you want to be removed from CC. May I please be added to the CC list. We are never going to form a consensus if all of the people doing implementations don't talk. Eric
creator: ebiederm
date: 2006-02-07T03:34:33-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: On Mon, 6 Feb 2006, Eric W. Biederman wrote: > We are never going to form a consensus if all of the people doing > implementations don't talk. Speaking of which - it would be interesting to get Kirill's comments on Eric's patchset ;) Once we know what's good and bad about both patchsets, we'll be a lot closer to knowing what exactly should go upstream. -- All Rights Reversed
creator: Rik van Riel
date: 2006-02-07T03:40:01-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Rik van Riel wrote: > On Mon, 6 Feb 2006, Eric W. Biederman wrote: > > >>We are never going to form a consensus if all of the people doing >>implementations don't talk. > > > Speaking of which - it would be interesting to get Kirill's > comments on Eric's patchset ;) > > Once we know what's good and bad about both patchsets, we'll > be a lot closer to knowing what exactly should go upstream. Let's compare approaches of patchsets before the patchsets themselves. It seems to be, should we: A) make a general form of virtualising PIDs, and hope this assists later virtualisation efforts (Eric's patch) B) make a general form of containers/jails/vservers/vpses, and layer PID virtualisation on top of it somewhere (as in openvz, vserver) I can't think of any real use cases where you would specifically want A) without B). Also, the problem space in B) is now very well explored. To start with A) would mean to throw away 4+ years of experience at this approach (just counting vserver and variants - not FreeBSD Jail, etc). Trying to re-base B) atop a massive refactoring and new patch like A) would incur a lot of work; however fitting it into B) is natural and solved conceptually and in practice, with the only drawback I see being that the use cases mentioned above wouldn't suffer from the side-effects of B). Perhaps I'm wrong there, but that's my gut feeling. Sam.
creator: Sam Vilain
date: 2006-02-07T06:30:34-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: >>> We are never going to form a consensus if all of the people doing >>> implementations don't talk. >> >> Speaking of which - it would be interesting to get Kirill's >> comments on Eric's patchset ;) I'll do comment. >> Once we know what's good and bad about both patchsets, we'll >> be a lot closer to knowing what exactly should go upstream. I'm starting to think that nothing in upstream can be better for all of us :) > Let's compare approaches of patchsets before the patchsets themselves. > It seems to be, should we: > > A) make a general form of virtualising PIDs, and hope this assists > later virtualisation efforts (Eric's patch) > > B) make a general form of containers/jails/vservers/vpses, and layer > PID virtualisation on top of it somewhere (as in openvz, vserver) > > I can't think of any real use cases where you would specifically want A) > without B). Exactly! All these patches for A) look weird for me without containers itself. A try to make half-solution which is bad. > Also, the problem space in B) is now very well explored. To start with > A) would mean to throw away 4+ years of experience at this approach > (just counting vserver and variants - not FreeBSD Jail, etc). Trying to > re-base B) atop a massive refactoring and new patch like A) would incur > a lot of work; however fitting it into B) is natural and solved > conceptually and in practice, with the only drawback I see being that > the use cases mentioned above wouldn't suffer from the side-effects of > B). Have you saw my patches? This is B) :) This is what we should start with IMHO. Having a containers and isolation all these talks about A) will be much more precise. Kirill
creator: dev
date: 2006-02-07T11:49:18-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: >>The important properties of the proposed container implementation: >>- each container has unique ID in the system > What namespace does this ID live in? global namespace. can be virtualized later. can be optional. But the idea is simple. Eventually you will need some management tools anyway. And they should be able to refer to containers. >>- each process in the kernel can belong to one container only > Reasonable. > > >>- effective container pointer (econtainer()) is used on the task to avoid >>insertion of additional argument "container" to all functions where it is >>required. > Why is that desirable? It was discussed with Linus and the reason is provided in this text actually. There are 2 ways: - to add additional argument "container" to all the functions where it is required. Drawbacks: a) lot's of changes, b) compilation without virtualization is not the same. c) increased stakc usage - to add effective container pointer on the task. i.e. context which kernel should be in when works with virtualized resources. Drawbacks: a) there are some places where you need to change effective container context explicitly such as TCP/IP. >>- kernel compilation with disabled virtualization should result in old good >>linux kernel > > A reasonable goal. > > Why do we need a container structure to hold pointers to other pointers? can't catch what you mean :) is it prohibited somehow? :)))) > May I please be added to the CC list. > We are never going to form a consensus if all of the people doing implementations don't > talk. To make a consensus people need to make mutual concessions... Otherwise these talks are useless. Kirill
creator: dev
date: 2006-02-07T12:12:43-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: >>@@ -1132,6 +1133,7 @@ static task_t *copy_process(unsigned lon >> p->ioprio = current->ioprio; >> >> SET_LINKS(p); >>+ (void)get_container(p->container); >> if (unlikely(p->ptrace & PT_PTRACED)) >> __ptrace_link(p, current->parent); > > > This entire patch looks nice and very straightforward, except for this > bit. :) The "(void)" bit isn't usual kernel coding style. You can > probably kill it. it is to avoid warning message the value has no effect. > BTW, why does get_container() return the container argument? > get_task_struct(), for instance is just a do{}while(0) loop, so it > doesn't have a return value. Is there some magic later on in your patch > set that utilizes this? ok, I will remake it without a return value. not a real problem at all. > One other really minor thing: I usually try to do is keep the ! > CONFIG_FOO functions static inlines, just like the full versions. The > advantage is that you get some compile-time type checking, even when > your CONFIG option is off. it is not always appropriate :( I try to follow this as well :) Kirill
creator: dev
date: 2006-02-07T12:22:18-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Kirill Korotaev <dev@sw.ru> writes: >>>The important properties of the proposed container implementation: >>>- each container has unique ID in the system >> What namespace does this ID live in? > global namespace. can be virtualized later. > can be optional. So a totally new kind of number in the kernel, that doesn't fall in any of the traditional namespaces. That is something I would like to avoid. > But the idea is simple. Eventually you will need some management tools > anyway. And they should be able to refer to containers. In my implementation I refer to things indirectly via the task in it since I have an appropriate pid. As does the current kernel code for the filesystem mount namespace. So far except a tool to create my things I don't need special management tools the traditional unix ones suffice as I am simply extending the traditional unix model. The best counter argument I have is that filesystem namespaces have already been implemented and we have not needed an additional id in the kernel. >>>- each process in the kernel can belong to one container only >> Reasonable. >> >>>- effective container pointer (econtainer()) is used on the task to avoid >>>insertion of additional argument "container" to all functions where it is >>>required. >> Why is that desirable? Ok. I can finally see the effective thing. Writing an argument onto the task structure instead of onto the stack seems weird to me. > It was discussed with Linus and the reason is provided in this text actually. > There are 2 ways: > - to add additional argument "container" to all the functions where it is > required. > Drawbacks: > a) lot's of changes, I believe most of those changes make other places in the kernel where misuse happens easier to find. Turning places that need changes into compile errors seems a virtue to me. I do know there are places that store identifiers that might be used in a different context later on. Finding all of those places seems more important than generating compile…
creator: ebiederm
date: 2006-02-07T14:06:50-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Kirill Korotaev <dev@sw.ru> writes: >>>> We are never going to form a consensus if all of the people doing >>>> implementations don't talk. >>> >>> Speaking of which - it would be interesting to get Kirill's >>> comments on Eric's patchset ;) > I'll do comment. Thank you I will look forward to your comments. >>> Once we know what's good and bad about both patchsets, we'll >>> be a lot closer to knowing what exactly should go upstream. > I'm starting to think that nothing in upstream can be better for all of us :) In a thread voicing the concerns for maintaining out of tree patches that is a natural concern. >> Let's compare approaches of patchsets before the patchsets themselves. >> It seems to be, should we: >> A) make a general form of virtualising PIDs, and hope this assists >> later virtualisation efforts (Eric's patch) >> B) make a general form of containers/jails/vservers/vpses, and layer >> PID virtualisation on top of it somewhere (as in openvz, vserver) > > >> I can't think of any real use cases where you would specifically want A) >> without B). > Exactly! All these patches for A) look weird for me without containers itself. A > try to make half-solution which is bad. I am willing to contend that my approach also leads to a complete solution. In fact I believe my network virtualization has actually gone much farther than yours. Although I admit there is still some work to do before the code is in shape to be merged. You notice in the kernel there is also not a struct process? To me having a container structure while an obvious approach to the problem seems to add unnecessary policy to the kernel. Lumping together the implementation of multiple instances of different namespaces in a way that the implementation does not require. Eric
creator: ebiederm
date: 2006-02-07T14:31:33-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: On Tue, 7 Feb 2006, Eric W. Biederman wrote: > Yes people have to be willing to bend to work together to find the > best solution. I am willing. At the same time I am not easy to > convince that other solutions are necessarily better. It's not about "better", it's about different requirements. -- All Rights Reversed
creator: Rik van Riel
date: 2006-02-07T14:52:17-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Rik van Riel <riel@redhat.com> writes: > On Tue, 7 Feb 2006, Eric W. Biederman wrote: > >> Yes people have to be willing to bend to work together to find the >> best solution. I am willing. At the same time I am not easy to >> convince that other solutions are necessarily better. > > It's not about "better", it's about different requirements. In part but I think it is possible to largely agree on what the requirements for the kernel are, by seeing what the requirements of the different users are. That is part of the discussion. Once those requirements are agreed upon it is about the best technical solution. Maintainability is one of the more important requirements is it not? Eric
creator: ebiederm
date: 2006-02-07T15:13:28-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Sam Vilain <sam@vilain.net> writes: > Rik van Riel wrote: >> On Mon, 6 Feb 2006, Eric W. Biederman wrote: >> >>> We are never going to form a consensus if all of the people doing >>> implementations don't talk. >> Speaking of which - it would be interesting to get Kirill's >> comments on Eric's patchset ;) >> Once we know what's good and bad about both patchsets, we'll >> be a lot closer to knowing what exactly should go upstream. > > Let's compare approaches of patchsets before the patchsets themselves. > > It seems to be, should we: > > A) make a general form of virtualising PIDs, and hope this assists > later virtualisation efforts (Eric's patch) > > B) make a general form of containers/jails/vservers/vpses, and layer > PID virtualisation on top of it somewhere (as in openvz, vserver) > > I can't think of any real use cases where you would specifically want A) > without B). You misrepresent my approach. First there is a huge commonality in the code bases between the different implementations and I have already gotten preliminary acceptance from the vserver developers, that my approach is sane. The major difference is what user interface does the kernel export, and I posted my user interface. What user interface to export is a debate worth having. For a lot of things getting the details just so is very important to long term maintainability and it is not my impression that anyone has done that yet. Second I am not trying to just implement a form of virtualizing PIDs. Heck I don't intend to virtualize anything. The kernel has already virtualized everything I require. I want to implement multiple instances of the current kernel global namespaces. All I want is to be able to use the same name twice in user space and not have a conflict. Beyond getting multiple instance of all of the kernel namespaces (which is the hard requirement for migration) my approach is to see what is needed for projects like vserver and vps and see how their needs can be met as well. I disagr…
creator: ebiederm
date: 2006-02-07T15:42:12-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: >>I can't think of any real use cases where you would specifically want A) >>without B). > You misrepresent my approach. [...] > Second I am not trying to just implement a form of virtualizing PIDs. > Heck I don't intend to virtualize anything. The kernel has already > virtualized everything I require. I want to implement multiple > instances of the current kernel global namespaces. All I want is > to be able to use the same name twice in user space and not have > a conflict. if you want not virtualize anything, what is this discussion about? :) can you provide an URL to your sources? you makes lot's of statements about that your network virtualization solution is better/more complete, so I'd like to see your solution in whole rather than only words. Probably this will help. > I disagree with a struct container simply because I do not see what > value it happens to bring to the table. I have yet to see a problem > that it solves that I have not solved yet. again, source would help to understand your solution and problem you solved and not solved yet. > In addition I depart from vserver and other implementations in another > regard. It is my impression a lot of their work has been done so > those projects are maintainable outside of the kernel, which makes > sense as that is where those code bases live. But I don't think that > gives the best solution for an in kernel implementation, which is > what we are implementing. These soltuions are in kernel implementations actually. Kirill
creator: dev
date: 2006-02-07T16:16:36-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Sam Vilain wrote: > Rik van Riel wrote: > >> On Mon, 6 Feb 2006, Eric W. Biederman wrote: >> >> >>> We are never going to form a consensus if all of the people doing >>> implementations don't talk. >> >> >> Speaking of which - it would be interesting to get Kirill's >> comments on Eric's patchset ;) >> >> Once we know what's good and bad about both patchsets, we'll >> be a lot closer to knowing what exactly should go upstream. > > > Let's compare approaches of patchsets before the patchsets themselves. > > It seems to be, should we: > > A) make a general form of virtualising PIDs, and hope this assists > later virtualisation efforts (Eric's patch) > > B) make a general form of containers/jails/vservers/vpses, and layer > PID virtualisation on top of it somewhere (as in openvz, vserver) > > I can't think of any real use cases where you would specifically want A) > without B). > > Also, the problem space in B) is now very well explored. To start with > A) would mean to throw away 4+ years of experience at this approach > (just counting vserver and variants - not FreeBSD Jail, etc). Trying to > re-base B) atop a massive refactoring and new patch like A) would incur > a lot of work; however fitting it into B) is natural and solved > conceptually and in practice, with the only drawback I see being that > the use cases mentioned above wouldn't suffer from the side-effects of > B). > Sam, that is a bit far fetched. I looked and experienced myself with both approaches and there is a lot of functional overlap, with both of them having advantages and disadvantages. What Eric provides is an alternative to the PID virtualization part of openvz. Indeed it is a pid isolation more then anything else (with some dealing at the boundary condition). I personally don't see much problem in replacing the pid virtualization of openvz with that of pidspaces. So the correct thing to do here is as RvR points out, simple discuss the merits and drawbacks of each PID approach for now and settle…
creator: Hubertus Franke
date: 2006-02-07T16:57:04-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Kirill Korotaev <dev@sw.ru> writes: >>>I can't think of any real use cases where you would specifically want A) >>>without B). > >> You misrepresent my approach. > [...] > >> Second I am not trying to just implement a form of virtualizing PIDs. >> Heck I don't intend to virtualize anything. The kernel has already >> virtualized everything I require. I want to implement multiple >> instances of the current kernel global namespaces. All I want is >> to be able to use the same name twice in user space and not have >> a conflict. > if you want not virtualize anything, what is this discussion about? :) > can you provide an URL to your sources? you makes lot's of statements about that > your network virtualization solution is better/more complete, so I'd like to see > your solution in whole rather than only words. > Probably this will help. Sure. I think it is more an accident of time, and the fact that I am quite proud of where I am at. You quite likely have improved things in openvz since last I looked as well. Currently my code quality is only a proof of concept but the tree is below. git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/linux -2.6-ns.git/ Basically the implementation appears to the user as a separate instance of the network stack. Since the tree was experimental the path is absolutely horrible. I am in the process of cleaning and redoing all of that now in preparation for kernel inclusion. But I suspect I am in the lead as no one else had noticed the ipv6 reference counting bugs. >> I disagree with a struct container simply because I do not see what >> value it happens to bring to the table. I have yet to see a problem >> that it solves that I have not solved yet. > again, source would help to understand your solution and problem you solved and > not solved yet. Above. But at least with pids it has all been posted on the mailing list. I think I have solved most of the code structural issues and the big kernel API issues. A lot of the little things I …
creator: ebiederm
date: 2006-02-07T17:20:46-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Quoting Hubertus Franke (frankeh@watson.ibm.com): > The vpid approach has the drawbacks of having to identify the conversion > spots > of all vpid vs. pid semantics. On the otherhand it does take advantage > of the fact that no virtualization has to take place until a "container" > has been migrated, thus rendering most of the vpid<->pid calls to be > noops. > > What I like about the pspace approach is that it explicitely defines in the > code > when I am using a different pspace for the lookup. That is kind of hidden in > the vpid/pid approach. I agree with this. From a maintenance pov, imagining making a minor change to some pid-related code, if I see something doing effectively "if (pspace1 == pspace2 && pid1==pid2)" that is clear, whereas trying to remember whether I'm supposed to return the pid or vpid can get really confusing. We actually had some errors with that while we were developing the first vpid patchset we posted in december. I believe that from a vserver point of view, either approach will work. You either create a new pspace and make 'init' pid 1 in that pspace, or, in the openvz approach, you start virtualizing with a hashtable so userspace in the new vserver/container/vz/whateveritscalled sees that init as pid1, while the rest of the system sees it as pid 3270 or something. Likewise, for checkpoint/restore and migration, either approach works. All we really need is, on restore/migrate, to be able to create processes with their original pids, so we can do that either with real pids in a new container, or virtualized pids faked for process in the same vz. Are there other uses of pid virtualization which one approach or the other cannot accomodate? If not, then I for one lean towards the more maintainable code. (which I'm sure we're not agreed on is Eric's, but imvho it is) -serge
creator: serue
date: 2006-02-07T20:19:08-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Serge E. Hallyn wrote: > Quoting Hubertus Franke (frankeh@watson.ibm.com): > >>The vpid approach has the drawbacks of having to identify the conversion >>spots >>of all vpid vs. pid semantics. On the otherhand it does take advantage >>of the fact that no virtualization has to take place until a "container" >>has been migrated, thus rendering most of the vpid<->pid calls to be >>noops. >> >>What I like about the pspace approach is that it explicitely defines in the >>code >>when I am using a different pspace for the lookup. That is kind of hidden in >>the vpid/pid approach. > > > I agree with this. From a maintenance pov, imagining making a minor > change to some pid-related code, if I see something doing effectively > "if (pspace1 == pspace2 && pid1==pid2)" that is clear, whereas trying > to remember whether I'm supposed to return the pid or vpid can get > really confusing. We actually had some errors with that while we were > developing the first vpid patchset we posted in december. > > I believe that from a vserver point of view, either approach will work. > You either create a new pspace and make 'init' pid 1 in that pspace, or, > in the openvz approach, you start virtualizing with a hashtable so > userspace in the new vserver/container/vz/whateveritscalled sees that > init as pid1, while the rest of the system sees it as pid 3270 or > something. > > Likewise, for checkpoint/restore and migration, either approach works. > All we really need is, on restore/migrate, to be able to create > processes with their original pids, so we can do that either with real > pids in a new container, or virtualized pids faked for process in the > same vz. > > Are there other uses of pid virtualization which one approach or the > other cannot accomodate? Kirill brought up that VPS can span a cluster.. if so how do you (Kirill) do that? You pre-partition the pids into allocation ranges for each container? Eitherway, if this is an important feature, then one needs to look at how that…
creator: Hubertus Franke
date: 2006-02-07T20:46:11-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Hubertus Franke <frankeh@watson.ibm.com> writes: > Kirill brought up that VPS can span a cluster.. > if so how do you (Kirill) do that? You pre-partition the pids into allocation > ranges for each container? > Eitherway, if this is an important feature, then one needs to look at > how that is achieved in pspace (e.g. mod the pidmap_alloc() function > to take legal ranges into account). Should still be straight forward. Actually legal ranges already exist in the form of min/max values. So that is trivial to implement. Eric
creator: ebiederm
date: 2006-02-07T22:00:04-00:00

The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: I think I can boil the discussion down into some of the fundamental questions that we are facing. Currently everyone seems to agree that we need something like my namespace concept that isolates multiple resources. We need these for PIDS UIDS SYSVIPC NETWORK UTSNAME FILESYSTEM etc. The questions seem to break down into: 1) Where do we put the references to the different namespaces? - Do we put the references in a struct container that we reference from struct task_struct? - Do we put the references directly in struct task_struct? 2) What is the syscall interface to create these namespaces? - Do we add clone flags? (Plan 9 style) - Do we add a syscall (similar to setsid) per namespace? (Traditional unix style)? - Do we in addition add syscalls to manipulate containers generically? I don't think having a single system call to create a container and a new instance of each namespace is reasonable as that does not give us a path into the future when we create yet another namespace. If we have one syscall per each namespace why would we need a container structure? 3) How do we refer to namespaces and containers when we are not members? - Do we refer to them indirectly by processes or other objects that we can see and are members? - Do we assign some kind of unique id to the containers? 4) How do we implement each of these namespaces? Besides being maintainable are there other constraints? 5) How do we control the resource inside a namespace starting from a process that is outside of that namespace? - The filesystem mount namespace gave an interesting answer. So it is quite possible other namespaces will give equally interesting and surprising answers. 6) How do we do all of this efficiently without a noticeable impact on performance? - I have already heard concerns that I might be introducing cache line bounces and thus increasing tasklist_lock hold time. Which on big way systems can be a problem. 7) How do we allow a process inside a container to create containers for it…
creator: ebiederm
date: 2006-02-07T22:06:51-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Eric W. Biederman wrote: > Hubertus Franke <frankeh@watson.ibm.com> writes: > > > >>Kirill brought up that VPS can span a cluster.. >>if so how do you (Kirill) do that? You pre-partition the pids into allocation >>ranges for each container? >>Eitherway, if this is an important feature, then one needs to look at >>how that is achieved in pspace (e.g. mod the pidmap_alloc() function >>to take legal ranges into account). Should still be straight forward. > > > Actually legal ranges already exist in the form of min/max values. > So that is trivial to implement. > Yipp, didn't want to state the obvious, but also give Kirrill a chance to explain how its done in OpenVZ. Ultimately, the same "partitioning" that works on vps_info, should work on pspace. -- Hubertus
creator: Hubertus Franke
date: 2006-02-07T22:19:27-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Eric W. Biederman wrote [note: quoting sections out of order]: > Sam Vilain <sam@vilain.net> writes: >>Let's compare approaches of patchsets before the patchsets themselves. >>It seems to be, should we: >> A) make a general form of virtualising PIDs, and hope this assists >> later virtualisation efforts (Eric's patch) >>I can't think of any real use cases where you would specifically want A) >>without B). > You misrepresent my approach. ok, after reading more of your post, agreed. > What user interface to export is a debate worth having. This is the bit that needs a long period of prototyping and experimental use IMHO. So in essence, we're agreeing on that point. > First there is a huge commonality in the code bases between the > different implementations and I have already gotten preliminary > acceptance from the vserver developers, that my approach is sane. The > major difference is what user interface does the kernel export, > and I posted my user interface. > Second I am not trying to just implement a form of virtualizing PIDs. > Heck I don't intend to virtualize anything. The kernel has already > virtualized everything I require. I want to implement multiple > instances of the current kernel global namespaces. All I want is > to be able to use the same name twice in user space and not have > a conflict. Right, well, I think our approaches might have more in common than I previously thought. Indeed, it seems that at least one of the features of Linux-VServer I am preparing for consideration for inclusion into Linus' tree are your work :-). > Beyond getting multiple instance of all of the kernel namespaces > (which is the hard requirement for migration) my approach is to > see what is needed for projects like vserver and vps and see how > their needs can be met as well. ok, but the question is - doesn't this just constitute a refactoring once the stable virtualisation code is in? I'm just a bit nervous about trying to refactor-approach-and-concepts-as-we-go. But …
creator: Sam Vilain
date: 2006-02-07T22:43:08-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Hubertus Franke wrote: > The container is just an umbrella object that ties every "virtualized" > subsystem together. I like this description; it matches roughly with the concepts as presented by vserver; there is the process virtualisation (vx_info), and the network virtualisation (nx_info) of Eric's that has been integrated to the vserver 2.1.x development branch. However the vx_info has become the de facto umbrella object space as well. These could almost certainly be split out without too much pain or incurring major rethinks. Sam.
creator: Sam Vilain
date: 2006-02-07T22:58:10-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Sam Vilain wrote: > Hubertus Franke wrote: > >> The container is just an umbrella object that ties every "virtualized" >> subsystem together. > > > I like this description; it matches roughly with the concepts as > presented by vserver; there is the process virtualisation (vx_info), and > the network virtualisation (nx_info) of Eric's that has been integrated > to the vserver 2.1.x development branch. However the vx_info has become > the de facto umbrella object space as well. These could almost > certainly be split out without too much pain or incurring major > rethinks. > > Sam. > Agreed.. here are some issued we learned from other projects that had similar interception points. Having a central umbrella object (let's stick to the name container) is useful, but being the only object through which every access has to pass may have drawbacks.. task->container->pspace->pidmap[offset].page implies potential cachemisses etc. If overhead becomes too large, then we can stick (cache) the pointer additionally in the task struct. But ofcourse that should be carefully examined on a per subsystem base... == Another thing to point out is that container's can have overlaps. C/R should be a policy thing. So if each "subsystem" > Quote Eric>>> > PIDS > UIDS > SYSVIPC > NETWORK > UTSNAME > FILESYSTEM is represented as a NAMESPACE, then one can pick and choose as a policy how these constitute at a conceptual level as a container. You want something migratable you better make sure that container implies unique subsystems. Maybe you want to nest containers, but only want to create a separate pidspaces for performance isolation (see planetlab work with vserver). So, there are many possibilities, that might make perfect sense for different desired solutions and it seems with the clone ( CLONE_FLAGS_NSPACE_[PIDS/UIDS/SYS.../FS] ) one gets a solution that is flexible, yet embodies may requirements..... -- Hubertus
creator: Hubertus Franke
date: 2006-02-07T23:18:31-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Eric W. Biederman wrote: > I think I can boil the discussion down into some of the fundamental > questions that we are facing. > Man, bearly can keep up with this email load. Addressed some in previous thread, but will reiterate under this context. > Currently everyone seems to agree that we need something like > my namespace concept that isolates multiple resources. > > We need these for > PIDS > UIDS > SYSVIPC > NETWORK > UTSNAME > FILESYSTEM > etc. > > The questions seem to break down into: > 1) Where do we put the references to the different namespaces? > - Do we put the references in a struct container that we reference from struct task_struct? > - Do we put the references directly in struct task_struct? You "cache" task_struct->container->hotsubsys under task_struct->hotsubsys. We don't change containers other then at clone time, so no coherency issue here !!!! Which subsystems pointers to "cache", should be agreed by the experts, but first approach should always not to cache and go through the container. > > 2) What is the syscall interface to create these namespaces? > - Do we add clone flags? > (Plan 9 style) Like that approach .. flexible .. particular when one has well specified namespaces. > - Do we add a syscall (similar to setsid) per namespace? > (Traditional unix style)? Where does that approach end .. what's wrong with doing it at clone() time ? Mainly the naming issue. Just providing a flag does not give me name. > - Do we in addition add syscalls to manipulate containers generically? > > I don't think having a single system call to create a container and a new > instance of each namespace is reasonable as that does not give us a > path into the future when we create yet another namespace. > Agreed. > If we have one syscall per each namespace why would we need a container > structure? > > 3) How do we refer to namespaces and containers when we are not members? > - Do we refer to them indirectly by processes or other objects that > we can see and ar…
creator: Hubertus Franke
date: 2006-02-07T23:35:52-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Hello! > >2) What is the syscall interface to create these namespaces? > > - Do we add clone flags? > > (Plan 9 style) > > Like that approach .. flexible .. particular when one has well specified > namespaces. > > > - Do we add a syscall (similar to setsid) per namespace? > > (Traditional unix style)? > > Where does that approach end .. what's wrong with doing it at clone() time ? That most of those namespaces need a special setup rather than a plain copy? F.e. what are you going to do with NETWORK namespace? The only valid thing to do is to prepare a new context and to configure its content (addresses, routing tables, iptables...) later. So that, in this case it is natural to inherit the context through clone() and to create new context with a separate syscall. Seems, only PID space needs to be setup at clone time. All the rest of suggested namespaces are more convenient to change with separate syscalls. I would suggest to combine both approaches. Those namespaces, which can be naturally copied while clone() (f.e. the best example is already existing CLONE_NEWNS) deserve a clone() flag. The rest are preserved through clone() and forked and configured later. Alexey
creator: Alexey Kuznetsov
date: 2006-02-08T00:43:25-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Kevin Fox wrote: >>>The container is just an umbrella object that ties every "virtualized" >>>subsystem together. >>I like this description; it matches roughly with the concepts as >>presented by vserver; there is the process virtualisation (vx_info), and >>the network virtualisation (nx_info) of Eric's that has been integrated >>to the vserver 2.1.x development branch. However the vx_info has become >>the de facto umbrella object space as well. These could almost >>certainly be split out without too much pain or incurring major >>rethinks. > How does all of this tie in with CPU Sets? It seems to me, they have > something not unlike a container already that supports nesting. Yes, I saw that. AIUI that's mainly about binding groups of processes to CPUs, to defeat Amdahl's law when it rears its head. It fits into the containers model as a hard partitioning feature, but is a lot more crude than the CPU Token Bucket scheduler in Linux-VServer. No doubt both CPU allocation strategies will be useful. Sam.
creator: Sam Vilain
date: 2006-02-08T01:16:01-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: On Wed, 2006-02-08 at 11:58 +1300, Sam Vilain wrote: > Hubertus Franke wrote: > > The container is just an umbrella object that ties every "virtualized" > > subsystem together. > > I like this description; it matches roughly with the concepts as > presented by vserver; there is the process virtualisation (vx_info), and > the network virtualisation (nx_info) of Eric's that has been integrated > to the vserver 2.1.x development branch. However the vx_info has become > the de facto umbrella object space as well. These could almost > certainly be split out without too much pain or incurring major > rethinks. > > Sam. How does all of this tie in with CPU Sets? It seems to me, they have something not unlike a container already that supports nesting. Kevin > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/
creator: Kevin Fox
date: 2006-02-08T02:08:03-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> writes: > Hello! > >> >2) What is the syscall interface to create these namespaces? >> > - Do we add clone flags? >> > (Plan 9 style) >> >> Like that approach .. flexible .. particular when one has well specified >> namespaces. >> >> > - Do we add a syscall (similar to setsid) per namespace? >> > (Traditional unix style)? >> >> Where does that approach end .. what's wrong with doing it at clone() time ? > > That most of those namespaces need a special setup rather than a plain copy? > > F.e. what are you going to do with NETWORK namespace? The only valid thing > to do is to prepare a new context and to configure its content (addresses, > routing tables, iptables...) later. So that, in this case it is natural > to inherit the context through clone() and to create new context > with a separate syscall. With a NETWORK namespace what I implemented was that you get a empty namespace with a loopback interface. But setting up the namespace from the inside is clearly the sane thing todo. > Seems, only PID space needs to be setup at clone time. All the rest of > suggested namespaces are more convenient to change with separate syscalls. Actually I think I can setup a PID space in a syscall as well. It is a little odd that your session, and process group change but since I was keeping 2 pids on the PID space leader I could easily do it. The fact that getpid() would start returning 1 might be confusing to a some processes though so clone seems to be the natural time to do it. > I would suggest to combine both approaches. Those namespaces, which can be > naturally copied while clone() (f.e. the best example is already existing > CLONE_NEWNS) deserve a clone() flag. The rest are preserved through clone() > and forked and configured later. Sounds reasonable. We make the decision on a case by case base whatever make sense for the patch and the namespace. The only real advantage to clone is you can create a bunch of namespaces all in one shot.…
creator: ebiederm
date: 2006-02-08T02:49:08-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Quoting Eric W. Biederman (ebiederm@xmission.com): > Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> writes: > > > Hello! > > > >> >2) What is the syscall interface to create these namespaces? > >> > - Do we add clone flags? > >> > (Plan 9 style) > >> > >> Like that approach .. flexible .. particular when one has well specified > >> namespaces. > >> > >> > - Do we add a syscall (similar to setsid) per namespace? > >> > (Traditional unix style)? > >> > >> Where does that approach end .. what's wrong with doing it at clone() time ? > > > > That most of those namespaces need a special setup rather than a plain copy? > > > > F.e. what are you going to do with NETWORK namespace? The only valid thing > > to do is to prepare a new context and to configure its content (addresses, > > routing tables, iptables...) later. So that, in this case it is natural > > to inherit the context through clone() and to create new context > > with a separate syscall. > > With a NETWORK namespace what I implemented was that you get a empty > namespace with a loopback interface. > > But setting up the namespace from the inside is clearly the sane thing > todo. What I tried to do in a proof of concept long ago was to have CLONE_NETNS mean that you get access to all the network devices, but then you could drop/add them. Conceptually I prefer that to getting an empty namespace, but I'm not sure whether there's any practical use where you'd want that... -serge
creator: serue
date: 2006-02-08T03:36:33-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: "Serge E. Hallyn" <serue@us.ibm.com> writes: > > What I tried to do in a proof of concept long ago was to have > CLONE_NETNS mean that you get access to all the network devices, but > then you could drop/add them. Conceptually I prefer that to getting an > empty namespace, but I'm not sure whether there's any practical use > where you'd want that... My observation was that the network stack does not come out cleanly as a namespace unless you adopt the rule that a network device belongs to exactly one network namespace. With that rule dealing with the network stack is just a matter of making some currently global variables/data structures per container. A pain to do the first round but easy to maintain once you are there and the logic of the code doesn't need to change. Eric
creator: ebiederm
date: 2006-02-08T03:52:15-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: The driving force for cpusets are NUMA architectures. Cpusets represent the topologies of NUMA systems, with hierarchies of cabinets, drawers, boards, packages, cores, hyperthreads, and with chunks of main memory associated usually with the board, but sometimes a layer or two up or down. Since not all cpus have the same access performance (delay and bandwidth) to all memory chunks (nodes), for optimum performance one wants to bind tasks, cpus and memory together, so as to run tasks on sets of cpus and memory that are "near" to each other, and to size the sets appropriately for the workload presented by the tasks. Cpusets have no explicit awareness of topology; they just provides a file system style hierarchy of named, permissioned sets, where each set has: mems - the memory nodes in that set cpus - the cpus in that set tasks - the tasks running on these cpus and mems For any cpuset, the 'cpus' and 'mems' are a subset of its parent in the hierarchy, and the root of the hierarchy (usually mounted at /dev/cpuset) contains all the online cpus and mems in the system. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2006-02-08T04:21:37-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: On Tue, Feb 07, 2006 at 08:52:15PM -0700, Eric W. Biederman wrote: > "Serge E. Hallyn" <serue@us.ibm.com> writes: > > > > > What I tried to do in a proof of concept long ago was to have > > CLONE_NETNS mean that you get access to all the network devices, but > > then you could drop/add them. Conceptually I prefer that to getting an > > empty namespace, but I'm not sure whether there's any practical use > > where you'd want that... > > My observation was that the network stack does not come out cleanly > as a namespace unless you adopt the rule that a network device > belongs to exactly one network namespace. yep, that's what the first network virtualization for Linux-VServer aimed at, but found too complicated the second one uses 'pairs' of communicating devices to send between guests/host > With that rule dealing with the network stack is just a matter of > making some currently global variables/data structures per container. yep, like the universal loopback and so ... > A pain to do the first round but easy to maintain once you are there > and the logic of the code doesn't need to change. best, Herbert > Eric
creator: Herbert Poetzl
date: 2006-02-08T04:37:21-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Herbert Poetzl <herbert@13thfloor.at> writes: > yep, that's what the first network virtualization for > Linux-VServer aimed at, but found too complicated > the second one uses 'pairs' of communicating devices > to send between guests/host Well you need the pairs of course, to communication between the two stack ``instances''. >> With that rule dealing with the network stack is just a matter of >> making some currently global variables/data structures per container. > > yep, like the universal loopback and so ... :) Eric
creator: ebiederm
date: 2006-02-08T04:46:26-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: On Tue, Feb 07, 2006 at 03:06:51PM -0700, Eric W. Biederman wrote: > > I think I can boil the discussion down into some of the fundamental > questions that we are facing. > > Currently everyone seems to agree that we need something like > my namespace concept that isolates multiple resources. > > We need these for > PIDS > UIDS > SYSVIPC > NETWORK > UTSNAME > FILESYSTEM > etc. > > The questions seem to break down into: > 1) Where do we put the references to the different namespaces? > - Do we put the references in a struct container that we > reference from struct task_struct? no, just let the tasks be in groups of disjunct spaces so that they can have shared or private structures for each of the identified spaces > - Do we put the references directly in struct task_struct? yes, IMHO that's the way to do it .. Linux-VServer is moving in this direction for some time now, but we need to add a special space for context capabilities and context flags, basically a context struct, similar to a namespace ... > 2) What is the syscall interface to create these namespaces? > - Do we add clone flags? > (Plan 9 style) I'd definitely prefer that, maybe if necessary with a 'new' clone syscall which allows to do a little more than clone does now, e.g. - clone into container/context/guest - set space initializations and limits, etc ... > - Do we add a syscall (similar to setsid) per namespace? > (Traditional unix style)? doesn't make sense for the creation, but a syscall for moving between and management of spaces are very important ... > - Do we in addition add syscalls to manipulate containers > generically? > > I don't think having a single system call to create a container > and a new instance of each namespace is reasonable as that does > not give us a path into the future when we create yet another > namespace. > > If we have one syscall per each namespace why would we need a > container structure? for the beforementioned permissions and flags, but you can as well see it as s…
creator: Herbert Poetzl
date: 2006-02-08T04:56:14-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Hubertus Franke <frankeh@watson.ibm.com> writes: > > Agreed.. here are some issued we learned from other projects that had > similar interception points. > > Having a central umbrella object (let's stick to the name container) > is useful, but being the only object through which every access has to > pass may have drawbacks.. > > task->container->pspace->pidmap[offset].page implies potential > cachemisses etc. > > If overhead becomes too large, then we can stick (cache) the pointer > additionally in the task struct. But ofcourse that should be carefully > examined on a per subsystem base... Ok. After talking with the vserver guys on IRC. I think grasp the importance. The key feature is to have a place to put limits and the like for your entire container. Look at all of the non-signal stuff in struct signal for an example. The nested namespaces seem to be just an implementation detail. For OpenVZ having the other namespaces nested may have some importance. I haven't gotten their yet. The task->container->pspace->.... thing feels very awkward to me, and feels like it increases our chance getting a cache miss. So I support the concept of a place to put all of the odd little things like rlimits for containers. But I would like to flatten it in the task_struct if we can. Eric
creator: ebiederm
date: 2006-02-08T05:03:50-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Hubertus Franke <frankeh@watson.ibm.com> writes: > Eric W. Biederman wrote: >> I think I can boil the discussion down into some of the fundamental >> questions that we are facing. >> > Man, bearly can keep up with this email load. Addressed some in > previous thread, but will reiterate under this context. :) >> Currently everyone seems to agree that we need something like >> my namespace concept that isolates multiple resources. >> We need these for PIDS >> UIDS >> SYSVIPC >> NETWORK >> UTSNAME >> FILESYSTEM >> etc. >> The questions seem to break down into: >> 1) Where do we put the references to the different namespaces? >> - Do we put the references in a struct container that we reference from struct > task_struct? >> - Do we put the references directly in struct task_struct? > > You "cache" task_struct->container->hotsubsys under task_struct->hotsubsys. > We don't change containers other then at clone time, so no coherency issue here > !!!! > Which subsystems pointers to "cache", should be agreed by the experts, > but first approach should always not to cache and go through the container. > >> 2) What is the syscall interface to create these namespaces? >> - Do we add clone flags? (Plan 9 style) > > Like that approach .. flexible .. particular when one has well specified > namespaces. > >> - Do we add a syscall (similar to setsid) per namespace? >> (Traditional unix style)? > > Where does that approach end .. what's wrong with doing it at clone() time ? > Mainly the naming issue. Just providing a flag does not give me name. It really is a fairly even toss up. The usual argument for doing it this way is that you will get a endless stream of arguments added to fork+exec other wise. Look of posix_spawn or the windows version if you want an example. Bits to clone are skirting the edge of a slippery slope. >> 3) How do we refer to namespaces and containers when we are not members? >> - Do we refer to them indirectly by processes or other objects that >> we can see and…
creator: ebiederm
date: 2006-02-08T05:23:15-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Eric W. Biederman wrote: > Hubertus Franke <frankeh@watson.ibm.com> writes: > >>Agreed.. here are some issued we learned from other projects that had >>similar interception points. >> >>Having a central umbrella object (let's stick to the name container) >>is useful, but being the only object through which every access has to >>pass may have drawbacks.. >> >>task->container->pspace->pidmap[offset].page implies potential >>cachemisses etc. >> >>If overhead becomes too large, then we can stick (cache) the pointer >>additionally in the task struct. But ofcourse that should be carefully >>examined on a per subsystem base... > > > Ok. After talking with the vserver guys on IRC. I think grasp the > importance. The key feature is to have a place to put limits and the > like for your entire container. Look at all of the non-signal stuff > in struct signal for an example. The nested namespaces seem to > be just an implementation detail. > > For OpenVZ having the other namespaces nested may have some > importance. I haven't gotten their yet. > > The task->container->pspace->.... thing feels very awkward to me, > and feels like it increases our chance getting a cache miss. > > So I support the concept of a place to put all of the odd little > things like rlimits for containers. But I would like to flatten > it in the task_struct if we can. > My point was to mainly identify the performance culprits and provide an direct access to those "namespaces" for performance reasons. So we all agreed on that we need to do that.. Question now (see other's note as well), should we provide a pointer to each and every namespace in struct task. Seem rather wasteful to me as certain path/namespaces are not exercise heavily. Having one object "struct container" that still embodies all namespace still seems a reasonable idea. Otherwise identifying the respective namespace of subsystems will have to go through container->init->subsys_namespace or similar. Not necessarily bad either.. -- Hubertus
creator: Hubertus Franke
date: 2006-02-08T14:13:25-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Quoting Herbert Poetzl (herbert@13thfloor.at): > > 3) How do we refer to namespaces and containers when we are not members? > > - Do we refer to them indirectly by processes or other objects that > > we can see and are members? > > the process will be an unique identifier to the > namespace, but it might not be easy to use it, so > IMHO it might at least make sense to ... Especially from userspace. If I want to start a checkpoint on a container, but I have to use the process to identify the container/namespace, well I can't uniquely specify the process by pid anymore... -serge
creator: serue
date: 2006-02-08T14:38:13-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Eric W. Biederman wrote: > Hubertus Franke <frankeh@watson.ibm.com> writes: > > >>Eric W. Biederman wrote: >> >>>2) What is the syscall interface to create these namespaces? >>> - Do we add clone flags? (Plan 9 style) >> >>Like that approach .. flexible .. particular when one has well specified >>namespaces. >> >> >>> - Do we add a syscall (similar to setsid) per namespace? >>> (Traditional unix style)? >> >>Where does that approach end .. what's wrong with doing it at clone() time ? >>Mainly the naming issue. Just providing a flag does not give me name. > > > It really is a fairly even toss up. The usual argument for doing it > this way is that you will get a endless stream of arguments added to > fork+exec other wise. Look of posix_spawn or the windows version if > you want an example. Bits to clone are skirting the edge of a slippery > slope. > So it seems the clone( flags ) is a reasonable approach to create new namespaces. Question is what is the initial state of each namespace? In pidspace we know we should be creating an empty pidmap ! In network, someone suggested creating a loopback device In uts, create "localhost" Are there examples where we rather inherit ? Filesystem ? Can we iterate the assumption for each subsystem what people thing is right? IMHO, there is only a need to refer to a namespace from the global context. Since one will be moving into a new container, but getting out of one could be prohibitive (e.g. after migration) It does not make sense therefore to know the name of a namespace in a different container. The example you used below by using the pid comes natural, because that already limits visibility. I am still struggling with why we need new sys_calls. sys_calls already exist for changing certain system parameters (e.g. utsname ) so to me it boils down to identifying a proper initial state when the namespace is created. > >>>3) How do we refer to namespaces and containers when we are not members? >>> - Do we refer to them indirectly by…
creator: Hubertus Franke
date: 2006-02-08T14:40:17-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Serge E. Hallyn wrote: > Quoting Herbert Poetzl (herbert@13thfloor.at): > >>>3) How do we refer to namespaces and containers when we are not members? >>> - Do we refer to them indirectly by processes or other objects that >>> we can see and are members? >> >>the process will be an unique identifier to the >>namespace, but it might not be easy to use it, so >>IMHO it might at least make sense to ... > > > Especially from userspace. If I want to start a checkpoint on a > container, but I have to use the process to identify the > container/namespace, well I can't uniquely specify the process by pid > anymore... > > -serge > Well we first can agree that througout all processes/tasks of a container the namespaces used are the same. Hence looking at the init_task of each container is sufficient. Restricting visibility to the default container makes sense to me, because one is not supposed to be able to look into another container. However, in the global context we do have pid that we can use. -- Hubertus
creator: Hubertus Franke
date: 2006-02-08T14:51:03-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Quoting Hubertus Franke (frankeh@watson.ibm.com): > Eric W. Biederman wrote: > So it seems the clone( flags ) is a reasonable approach to create new > namespaces. Question is what is the initial state of each namespace? > In pidspace we know we should be creating an empty pidmap ! > In network, someone suggested creating a loopback device > In uts, create "localhost" > Are there examples where we rather inherit ? Filesystem ? Of course filesystem is already implemented, and does inheret a full copy. -serge
creator: serue
date: 2006-02-08T15:17:26-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: >>Eric W. Biederman wrote: >>So it seems the clone( flags ) is a reasonable approach to create new >>namespaces. Question is what is the initial state of each namespace? >>In pidspace we know we should be creating an empty pidmap ! >>In network, someone suggested creating a loopback device >>In uts, create "localhost" >>Are there examples where we rather inherit ? Filesystem ? > Of course filesystem is already implemented, and does inheret a full > copy. why do we want to use clone()? Just because of its name and flags? I think it is really strange to fork() to create network context. What has process creation has to do with it? After all these clone()'s are called, some management actions from host system are still required, to add these IPs/routings/etc. So? Why mess it up? Why not create a separate clean interface for container management? Kirill
creator: dev
date: 2006-02-08T15:34:41-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: > The pid-namespace (pspace) provides an approach of fully separate > the allocation and maintenance of the pids and treating the <pspace,pid> > tuple as an entity to uniquely identify a task and vice versa. > As a result the logic of lookup can be pushed down the find_task_by_pid() > lookup. There are specific cases where the init_task of a container or > pspace needs to be checked to ensure that signals/waits and alike are > properly > handled across pspace boundaries. I think this is an intuitive and clean > way. > It also completely avoids the problem of having to think about all the > locations > at the user/kernel boundary where a vpid/pid conversion needs to take > place. > It also avoids the problems that logically vpids and pids are different > types and > therefore it would have been good to have type checking automatically > identify > problem spots. > On the negative side, it does require to maintain a pidmap per pidspace. Additional negative sides: - full isolation can be inconvinient from containers management point of view. You will need to introduce new modified tools such as top/ps/kill and many many others. You won't be able to strace/gdb processes from the host also. - overhead when virtualization is off, result is not the same. - additional args everywhere (stack usage, etc.) > The vpid approach has the drawbacks of having to identify the conversion > spots > of all vpid vs. pid semantics. On the otherhand it does take advantage > of the fact that no virtualization has to take place until a "container" > has been migrated, thus rendering most of the vpid<->pid calls to be > noops. It has some other additional advantages: - flexible: you can select full isolation or weak is required. I really believe this is very important. > The container is just an umbrella object that ties every "virtualized" > subsystem > together. Yep. And containers were what I wanted to start with actually. Not VPIDs. Kirill
creator: dev
date: 2006-02-08T15:35:10-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: > My point was to mainly identify the performance culprits and provide > an direct access to those "namespaces" for performance reasons. > So we all agreed on that we need to do that.. After having looked at Eric's patch, I can tell that he does all these dereferences in quite the same amount. For example, lot's of skb->sk->host->... while in OpenVZ it would be econtainer()->... which is essentially current->container->... So until someone did measurements it looks doubtfull that one solution is better than the another. > Question now (see other's note as well), should we provide > a pointer to each and every namespace in struct task. > Seem rather wasteful to me as certain path/namespaces are not > exercise heavily. > Having one object "struct container" that still embodies all > namespace still seems a reasonable idea. > Otherwise identifying the respective namespace of subsystems will > have to go through container->init->subsys_namespace or similar. > Not necessarily bad either.. why not simply container->subsys_namespace? Kirill
creator: dev
date: 2006-02-08T15:42:05-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Kirill Korotaev wrote: >>> Eric W. Biederman wrote: >>> So it seems the clone( flags ) is a reasonable approach to create new >>> namespaces. Question is what is the initial state of each namespace? >>> In pidspace we know we should be creating an empty pidmap ! >>> In network, someone suggested creating a loopback device >>> In uts, create "localhost" >>> Are there examples where we rather inherit ? Filesystem ? >> >> Of course filesystem is already implemented, and does inheret a full >> copy. > > > why do we want to use clone()? Just because of its name and flags? > I think it is really strange to fork() to create network context. What > has process creation has to do with it? > > After all these clone()'s are called, some management actions from host > system are still required, to add these IPs/routings/etc. > So? Why mess it up? Why not create a separate clean interface for > container management? > > Kirill > We need a "init" per container, which represents the context of the system represented by the container. If that is the case, then why not create the container such that we specify what namespaces need to be new for a container at the container creation time and initialize them to a well understood state that makes sense (e.g. copy namespace (FS, uts) , new fresh state (pid) ). Then use the standard syscall to modify state (now "virtualized" through the task->xxx_namespace access ). Do you see a need to change the namespace of a container after it has been created. I am not referring to the state of the namespace but truely moving to a completely different namespace after the container has been created. Obviously you seem to have some other usage in mind, beyond what my limited vision can see. Can you share some of those examples, because that would help this discussion along ... Thanks a 10^6. -- Hubertus
creator: Hubertus Franke
date: 2006-02-08T15:57:22-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Kirill Korotaev <dev@sw.ru> writes: >> My point was to mainly identify the performance culprits and provide >> an direct access to those "namespaces" for performance reasons. >> So we all agreed on that we need to do that.. > After having looked at Eric's patch, I can tell that he does all these > dereferences in quite the same amount. > > For example, lot's of skb->sk->host->... > while in OpenVZ it would be econtainer()->... which is essentially > current->container->... Except at that point in time I cannot use current, because it does not have necessarily have the appropriate context. So doubt you could correctly use econtainer there. > So until someone did measurements it looks doubtfull that one solution is better > than the another. I agree, exactly where we look is a minor matter, unless we can find an instance where there are good reasons for preferring one representation over another. Performance currently does not look a sufficient reason. My basis for preferring a flat layout is essentially because that is how other similar interfaces are currently implemented in the kernel. In linux we have a plan9 inspired system call called clone. One of the ideas with clone is that when you create a new task you either share or you don't share resources. Namespaces are one of those resources. Since we are dealing with concepts that appear to fit the existing model beautifully my feeling is to use the existing model of how things are currently implemented in the kernel. The only other reason I can think of is namespace implementation independence. If we have an additional structure that we collect up the pointers it feels like it causes unnecessary tying. The best justification I can think of for putting it all in one structure seems like the current->container current->econtainer thing that comes out of OpenVZ. Currently I have followed the threads enough to know it exists but I yet understand it. If there is a good reason for the container/econtainer thing I can cer…
creator: ebiederm
date: 2006-02-08T16:39:51-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Kirill Korotaev <dev@sw.ru> writes: >>>Eric W. Biederman wrote: >>>So it seems the clone( flags ) is a reasonable approach to create new >>>namespaces. Question is what is the initial state of each namespace? >>>In pidspace we know we should be creating an empty pidmap ! >>>In network, someone suggested creating a loopback device >>>In uts, create "localhost" >>>Are there examples where we rather inherit ? Filesystem ? >> Of course filesystem is already implemented, and does inheret a full >> copy. > > why do we want to use clone()? Just because of its name and flags? > I think it is really strange to fork() to create network context. What has > process creation has to do with it? Agreed. Although clones brother unshare takes process creation out of the picture, but otherwise preserves the same interface. > After all these clone()'s are called, some management actions from host system > are still required, to add these IPs/routings/etc. > So? Why mess it up? Why not create a separate clean interface for container > management? If we need additional arguments besides create the thing. We have a clear argument that clone is completely the wrong interface. However. So far I have not seen an instance where using the existing standard configuration mechanisms from inside the namespace is not the proper way to set things up. The only thing I know that needs to happen from outside is to pass the container a network interface. And if it is a physical interface that is all that must happen. Eric
creator: ebiederm
date: 2006-02-08T16:48:14-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Kirill Korotaev <dev@sw.ru> writes: > Additional negative sides: > - full isolation can be inconvinient from containers management point of > view. You will need to introduce new modified tools such as top/ps/kill and many > many others. You won't be able to strace/gdb processes from the host also. Ok. I have to pick a nit here. Needing all new tools top/ps/kill was an artifact of your implementation. Mine does not suffer from it. > - overhead when virtualization is off, result is not the same. > - additional args everywhere (stack usage, etc.) Agreed. When using a PID namespace the code always behaves the same. Which is with more arguments than the code used to have. However the code always behaving the same is a tremendous advantage for maintainability. >> The vpid approach has the drawbacks of having to identify the conversion spots >> of all vpid vs. pid semantics. On the otherhand it does take advantage >> of the fact that no virtualization has to take place until a "container" >> has been migrated, thus rendering most of the vpid<->pid calls to be >> noops. > It has some other additional advantages: > - flexible: you can select full isolation or weak is required. I really believe > this is very important. I am willing to be convinced but so far there seem to be other solutions like running gdb_stubs inside your container. To address most of the issues. In a fairly real sense mucking with a container from the outside appears to be a sysadmin layering violation. >> The container is just an umbrella object that ties every "virtualized" >> subsystem >> together. > Yep. And containers were what I wanted to start with actually. Not VPIDs. The historical linux approach is to build things out of tasks sharing resources that give the impression of containers not out containers and their children. I am still trying to understand why that model does not work in this instance. Eric
creator: ebiederm
date: 2006-02-08T17:16:45-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Hubertus Franke <frankeh@watson.ibm.com> writes: > > So it seems the clone( flags ) is a reasonable approach to create new > namespaces. Question is what is the initial state of each namespace? > In pidspace we know we should be creating an empty pidmap ! > In network, someone suggested creating a loopback device > In uts, create "localhost" "localhost" is wrong. Either copy or set it to the value from system initialization time. > Are there examples where we rather inherit ? Filesystem ? > Can we iterate the assumption for each subsystem what people thing is right? I think this needs to happen on a pure subsystem basis as we merge the patches. > IMHO, there is only a need to refer to a namespace from the global context. > Since one will be moving into a new container, but getting out of one > could be prohibitive (e.g. after migration) > It does not make sense therefore to know the name of a namespace in > a different container. > > The example you used below by using the pid comes natural, because > that already limits visibility. > > I am still struggling with why we need new sys_calls. > sys_calls already exist for changing certain system parameters (e.g. utsname ) > so to me it boils down to identifying a proper initial state when the > namespace is created. Agreed. But we can't always count on everything having a useful state that we can modify from the inside. So it is important to leave the option open at least for those case. And again there is always the idea that by adding flags we will transform fork or fork+exec into a spaghetti system call. I think that is a reasonable concern. Also there is always the danger that we run out of clone flags. >> What I have done which seems easier than creating new names is to refer >> to the process which has the namespace I want to manipulate. > > Is then the idea to only allow the container->init to manipulate > or is there need to allow other priviliged processes to perform namespace > manipulation? > Also after thin…
creator: ebiederm
date: 2006-02-08T17:46:14-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Quoting Hubertus Franke (frankeh@watson.ibm.com): > IMHO, there is only a need to refer to a namespace from the global context. > Since one will be moving into a new container, but getting out of one > could be prohibitive (e.g. after migration) > It does not make sense therefore to know the name of a namespace in > a different container. Not sure I agree. What if we are using a private namespace for a vserver, and then we want to create a private namespace in there for a mobile application. Since we're talking about nested namespaces, this should be possible. Now I believe Eric's code so far would make it so that you can only refer to a namespace from it's *creating* context. Still restrictive, but seems acceptable. (right?) -serge
creator: serue
date: 2006-02-08T18:03:09-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Serge E. Hallyn wrote: > Quoting Hubertus Franke (frankeh@watson.ibm.com): > >>IMHO, there is only a need to refer to a namespace from the global context. >>Since one will be moving into a new container, but getting out of one >>could be prohibitive (e.g. after migration) >>It does not make sense therefore to know the name of a namespace in >>a different container. > > > Not sure I agree. What if we are using a private namespace for a > vserver, and then we want to create a private namespace in there for a > mobile application. Since we're talking about nested namespaces, this > should be possible. > > Now I believe Eric's code so far would make it so that you can only > refer to a namespace from it's *creating* context. Still restrictive, > but seems acceptable. > That's what I meant .. as usually used the wrong word.. s/global context/spawing context/g .. because that's the only place where you have a pid to refer to the newly created container ! > (right?) > Yes, seriii .. ahmm serue > -serge >
creator: Hubertus Franke
date: 2006-02-08T18:31:08-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: On Wed, Feb 08, 2006 at 10:57:24AM -0500, Hubertus Franke wrote: >Kirill Korotaev wrote: >>>>Eric W. Biederman wrote: >>>>So it seems the clone( flags ) is a reasonable approach to create new >>>>namespaces. Question is what is the initial state of each namespace? >>>>In pidspace we know we should be creating an empty pidmap ! >>>>In network, someone suggested creating a loopback device >>>>In uts, create "localhost" >>>>Are there examples where we rather inherit ? Filesystem ? >>> >>>Of course filesystem is already implemented, and does inheret a full >>>copy. I try to comment on both mails here because I thing that clone() is basically a good interface, but will require some redesign and/or extension ... >> why do we want to use clone()? because it is a natural and existing interface for this purpose at least in Linux-VServer it would work (to some extend). why? because we aleady use tools like chbind and chcontext which do similar things as chroot, and chroot could, in theory, use clone() and rbind to do it's job ... >> Just because of its name and flags? extending the flags seems natural to me, but the problem might actually be that there are not enough of them left >> I think it is really strange to fork() to create network context. if you look at it as network namespace and sharing existing spaces and/or creating new ones, then clone() and unshare() make pretty much sense there ... >> What has process creation has to do with it? it is a natural interface where you can decide whether to share a space or acquire a new one ... IMHO it would make sense to get trivial userspace tools to create those new spaces in one go, so that the user can use those 'building blocks' to create new spaces whenever she needs >> After all these clone()'s are called, some management actions >> from host system are still required, to add these IPs/routings/etc. not necessarily, for example Linux-VServer uses some kind of 'priviledged' mode, in which the initial guest process can modif…
creator: Herbert Poetzl
date: 2006-02-08T19:02:30-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: On Wed, 2006-02-08 at 12:03 -0600, Serge E. Hallyn wrote: > Now I believe Eric's code so far would make it so that you can only > refer to a namespace from it's *creating* context. Still restrictive, > but seems acceptable. The same goes for filesystem namespaces. You can't see into random namespaces, just the ones underneath your own. Sounds really reasonable to me. -- Dave
creator: Dave Hansen
date: 2006-02-08T20:21:39-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: On Wed, 2006-02-08 at 18:36 +0300, Kirill Korotaev wrote: > - full isolation can be inconvinient from containers management point of > view. You will need to introduce new modified tools such as top/ps/kill > and many many others. You won't be able to strace/gdb processes from the > host also. I'd like to put a theory out there: the more isolation we perform, the easier checkpointing and migration become to guarantee. Agree? Disagree? But, full isolation is hard to code. The right approach is very likely somewhere in the middle where we require some things to happen underneath us. For instance, requiring that the filesystem be made consistent if a container is moved across systems. -- Dave
creator: Dave Hansen
date: 2006-02-08T20:43:55-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Dave Hansen <haveblue@us.ibm.com> writes: > On Wed, 2006-02-08 at 18:36 +0300, Kirill Korotaev wrote: >> - full isolation can be inconvinient from containers management point of >> view. You will need to introduce new modified tools such as top/ps/kill >> and many many others. You won't be able to strace/gdb processes from the >> host also. > > I'd like to put a theory out there: the more isolation we perform, the > easier checkpointing and migration become to guarantee. > > Agree? Disagree? Agree. But that does not address the reasons OpenVZ and Vserver exist. > But, full isolation is hard to code. Disagree. If you limit your self to just changing the code that translates from names to objects it is a very narrow slice of code, and there are very few surprises. There is a lot of grunt work involved but it easy to tell if you got everything and did it correctly. Other approaches are more adhoc, take short cuts, and seem prone to missing the corner cases. > The right approach is very likely > somewhere in the middle where we require some things to happen > underneath us. For instance, requiring that the filesystem be made > consistent if a container is moved across systems. Possibly. That is very out from where we are at the moment. Let's get the isolation and see where we are at. Eric
creator: ebiederm
date: 2006-02-08T21:04:14-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Quoting Dave Hansen (haveblue@us.ibm.com): > On Wed, 2006-02-08 at 12:03 -0600, Serge E. Hallyn wrote: > > Now I believe Eric's code so far would make it so that you can only > > refer to a namespace from it's *creating* context. Still restrictive, > > but seems acceptable. > > The same goes for filesystem namespaces. You can't see into random > namespaces, just the ones underneath your own. Sounds really reasonable > to me. Hmmm? I suspect I'm misreading what you're saying, but to be clear: Let's say I start a screen session. In one of those shells, I clone, specify CLONE_NEWNS, and exec a shell. now i do a bunch of mounting. Other shells in the screen session won't see the results of those mounts, and if i ctrl-d, the shell which started the screen session can't either. Each of these is in the "parent filesystem namespace". OTOH, shared subtrees specified in the parent shell could make it such that the parent ns, but not others, see the results. Is that what you're referring to? thanks, -serge
creator: serue
date: 2006-02-08T21:22:12-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Hubertus Franke <frankeh@watson.ibm.com> writes: > Eric W. Biederman wrote: >> Hubertus Franke <frankeh@watson.ibm.com> writes: >> >>>Eric W. Biederman wrote: >>> >> >>>>3) How do we refer to namespaces and containers when we are not members? >>>> - Do we refer to them indirectly by processes or other objects that >>>> we can see and are members? >>>> - Do we assign some kind of unique id to the containers? >>> >>> >> What I have done which seems easier than creating new names is to refer >> to the process which has the namespace I want to manipulate. > > Is then the idea to only allow the container->init to manipulate > or is there need to allow other priviliged processes to perform namespace > manipulation? > Also after thinking about it.. why is there a need to have an external name > for a namespace ? There are several cases. Passing network devices to a childs namespace, as usually the loopback interface is not enough. Monitoring the namespace from outside, so among other things you aren't required to checkpoint and migrate your monitoring daemon. There are several other control and monitoring operations that I am not quite as familiar. One of them is the vserver idea of entering a guest. To expand on things a little bit. If we have interfaces that take strings we can refer to an arbitrary child process as pid/pid/pid/.... So we should not be limited to what is at the init of the container. If that proves desirable. Permissions checks for most of these operations require some serious thinking before they are merged. Eric
creator: ebiederm
date: 2006-02-08T22:28:12-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Kirill Korotaev <dev@openvz.org> writes: > Hello, > > I tried to take into account all the comments from you guys (thanks a lot for > them!) and prepared a new version of virtualization patches. I will send only 4 > patches today, just not to overflow everyone and keep it clear/tidy/possible to > review. > > This patch introduces some abstract container kernel structure and a number of > operations on it. > > The important properties of the proposed container implementation: > - each container has unique ID in the system > - each process in the kernel can belong to one container only > - effective container pointer (econtainer()) is used on the task to avoid > insertion of additional argument "container" to all functions where it is > required. > - kernel compilation with disabled virtualization should result in old good > linux kernel > > Patches following this one will be used for virtualization of the kernel > resources based on this container infrastructure, including those VPID patches I > sent before. Every virtualized resource can be given separate config option if > needed (just give me to know if it is desired). After having digested this I think there is something sane with regard to this container idea. I still think the implementation is totally wrong but there is a potential problem the basic idea solves. In the traditional linux/plan9 style of namespaces the question is which resources different tasks share, and we pass clone bits to determine what we want to share and what we don't want to share. Semantically this is very clean and allows for a great deal of flexibility. However as the flexibility increases we get more code in do_fork more reference counts and ultimately the performance decreases. In addition it is not common for us to change which resources we share. So our traditional route is flexible but it does not optimize the common case where we share all of the same things. Containers can potentially optimize that case. So long as anyone can …
creator: ebiederm
date: 2006-02-09T00:24:16-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: On Wed, Feb 08, 2006 at 05:24:16PM -0700, Eric W. Biederman wrote: > To deal with networking there are currently a significant number of > variables with static storage duration. Making those variables global > and placing them in structures is neither as efficient as it could be > nor is it as maintainable as it should be. Other subsystems have > similar problems. BTW, there is another solution, which you may or may not consider to be clean. That is to load a separate copy of the subsystem (code and data) as a module when you want a new instance of it. The code doesn't change, but you probably have to move it around some and provide some sort of interface to it. I did this to the scheduler last year - see http://marc.theaimsgroup.com/?l=linux-kernel&m=111404726 721747&w=2 Jeff
creator: Jeff Dike
date: 2006-02-09T02:18:28-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Jeff Dike <jdike@addtoit.com> writes: > On Wed, Feb 08, 2006 at 05:24:16PM -0700, Eric W. Biederman wrote: >> To deal with networking there are currently a significant number of >> variables with static storage duration. Making those variables global >> and placing them in structures is neither as efficient as it could be >> nor is it as maintainable as it should be. Other subsystems have >> similar problems. > > BTW, there is another solution, which you may or may not consider to > be clean. > > That is to load a separate copy of the subsystem (code and data) as a > module when you want a new instance of it. The code doesn't change, > but you probably have to move it around some and provide some sort of > interface to it. There are some real drawbacks to that. Basically as things are structured you only want multiple copies of the user facing data structures. Pretty much everything else really does remain the same. > I did this to the scheduler last year - see > http://marc.theaimsgroup.com/?l=linux-kernel&m=111404726 721747&w=2 I will take a look but I don't expect to find anything. Eric
creator: ebiederm
date: 2006-02-09T03:16:38-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: On Feb 07, 2006, at 17:06, Eric W. Biederman wrote: > I think I can boil the discussion down into some of the fundamental > questions that we are facing. > > Currently everyone seems to agree that we need something like my > namespace concept that isolates multiple resources. > > We need these for > UIDS > FILESYSTEM I have one suggestion for this (it also covers capabilities to a certain extent). Could we use the kernel credentials system to abstract away the concept of a single UID/GID? We currently have uid, euid, gid, egid, groups, fsid. I'm thinking that there would be virtualized UID tables to determine ownership of processes/SHM/etc. Each process would have a (uid_container,uid) pair (or similar) as its "uid" and likewise for gid. Then the ability to send signals to any given (uid_container,uid) or (gid_container,gid) pair would be given by keys in the kernel keyring indexed by the "uid_container" part and containing the "uid" part (or maybe just a pointer). Likewise the filesystem access could be virtualized by using uid and gid keys in the kernel keyring indexed by vfsmount (Not superblock, so that it would be possible to have different UID representations on different mounts/parts of the same filesystem). I'm guessing that the performance implications of the above would not be quite so nice, as it would put a lot of code in the fastpath, but I would guess that it might be possible to use the existing fields for processes without any virtualization needs. Cheers, Kyle Moffett -- There is no way to make Linux robust with unreliable memory subsystems, sorry. It would be like trying to make a human more robust with an unreliable O2 supply. Memory just has to work. -- Andi Kleen
creator: Kyle Moffett
date: 2006-02-09T04:45:02-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: Kyle Moffett <mrmacman_g4@mac.com> writes: > On Feb 07, 2006, at 17:06, Eric W. Biederman wrote: >> I think I can boil the discussion down into some of the fundamental questions >> that we are facing. >> >> Currently everyone seems to agree that we need something like my namespace >> concept that isolates multiple resources. >> >> We need these for >> UIDS >> FILESYSTEM > > I have one suggestion for this (it also covers capabilities to a certain > extent). Could we use the kernel credentials system to abstract away the > concept of a single UID/GID? We currently have uid, euid, gid, egid, groups, > fsid. I'm thinking that there would be virtualized UID tables to determine > ownership of processes/SHM/etc. > > Each process would have a (uid_container,uid) pair (or similar) as its "uid" > and likewise for gid. Then the ability to send signals to any given > (uid_container,uid) or (gid_container,gid) pair would be given by keys in the > kernel keyring indexed by the "uid_container" part and containing the "uid" > part (or maybe just a pointer). > > Likewise the filesystem access could be virtualized by using uid and gid keys > in the kernel keyring indexed by vfsmount (Not superblock, so that it would be > possible to have different UID representations on different mounts/parts of the > same filesystem). > > I'm guessing that the performance implications of the above would not be quite > so nice, as it would put a lot of code in the fastpath, but I would guess that > it might be possible to use the existing fields for processes without any > virtualization needs. At least for signal sending it looks like it would be easier to just compare the pointers to struct user. At least in that context it looks like it would be as cheap as what we are doing now. I just don't know where to find a struct user for the euid, or is it the normal uid. Eric
creator: ebiederm
date: 2006-02-09T05:41:10-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Jeff Dike wrote: > On Wed, Feb 08, 2006 at 05:24:16PM -0700, Eric W. Biederman wrote: > >>To deal with networking there are currently a significant number of >>variables with static storage duration. Making those variables global >>and placing them in structures is neither as efficient as it could be >>nor is it as maintainable as it should be. Other subsystems have >>similar problems. > > > BTW, there is another solution, which you may or may not consider to > be clean. > > That is to load a separate copy of the subsystem (code and data) as a > module when you want a new instance of it. The code doesn't change, > but you probably have to move it around some and provide some sort of > interface to it. > > I did this to the scheduler last year - see > http://marc.theaimsgroup.com/?l=linux-kernel&m=111404726 721747&w=2 > > Jeff > Jeff, interesting, but won't that post some serious scalability issue? Imaging 100s of container/namespace ? The namespace is mainly there to identify which data needs to be private when multiple instances of a subsystem are considered and encapsulate that data in an object/datastructure ! -- Hubertus
creator: Hubertus Franke
date: 2006-02-09T16:38:31-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: On Thu, Feb 09, 2006 at 11:38:31AM -0500, Hubertus Franke wrote: > Jeff, interesting, but won't that post some serious scalability issue? > Imaging 100s of container/namespace ? In terms of memory? Running size on sched.o gives me this on x86_64: text data bss dec hex filename 35685 6880 28800 71365 116c5 sched.o and on i386 (actually UML/i386) text data bss dec hex filename 10010 36 2504 12550 3106 obj/kernel/sched.o I'm not sure why there's such a big difference, but 100 instances adds a meg or two (or three) to the kernel. This overstates things a bit because there are things in sched.c which wouldn't be duplicated, like the system calls. How big a deal is that on a system which you plan to have 100s of containers on anyway? It's heavier than your namespaces, but does have the advantage that it imposes no cost when it's not being used. > The namespace is mainly there to identify which data needs to be private > when multiple instances of a subsystem are considered and > encapsulate that data in an object/datastructure ! Sure, and that's a fine approach. It's just not the only one. Jeff
creator: Jeff Dike
date: 2006-02-09T17:47:58-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Ok digesting this some more I don't see a glaring case in the code where it is set_econtainer(...); do stuff ; restore_econtainer(..); has a tremendous advantage at the moment. Plus it removes the opportunity to catch old code that deals with multiple contexts at compile time, and has to be done by pain staking code review. That aside the truly important thing I realized is that for the optimization effects it is an implementation detail. Batched reference counting of shared resources is something we can go back in and add at any time. It will take a little refactoring to do but it something that can be done without changing the user space API. So since individual namespace pointers in the task_struct are simpler and have the required flexibility. I think we can table this part of the discussion for now. The other detail I brought up which was most interesting was the marking of specific variables that would go into a container or namespace. Well as long as it is implemented on the namespace level that is still possible, and may even be beneficial. Eric
creator: ebiederm
date: 2006-02-09T21:56:46-00:00

Re: [PATCH 1/4] Virtualization/containers: introduction
type: item
description: Jeff Dike wrote: > On Thu, Feb 09, 2006 at 11:38:31AM -0500, Hubertus Franke wrote: >>Jeff, interesting, but won't that post some serious scalability issue? >>Imaging 100s of container/namespace ? > In terms of memory? > Running size on sched.o gives me this on x86_64: > text data bss dec hex filename > 35685 6880 28800 71365 116c5 sched.o > > and on i386 (actually UML/i386) > > text data bss dec hex filename > 10010 36 2504 12550 3106 obj/kernel/sched.o > > I'm not sure why there's such a big difference, but 100 instances adds > a meg or two (or three) to the kernel. This overstates things a bit > because there are things in sched.c which wouldn't be duplicated, like > the system calls. > > How big a deal is that on a system which you plan to have 100s of > containers on anyway? Quite a big deal. You might have 2Gigs of main memory, but your CPU is unlikely to be more than a Megabyte in close reach. A meg or two of scheduler data and code means that your L1 and L2 cache will be cycling every scheduler round; which is OK if you have very short runqueues but as you get more and more processes it will really start to hurt. Remember, systems today are memory bound and anything you can do to reduce the amount of time the system sits around waiting for memory to fetch, the better. Compare that to the Token Bucket Scheduler of Linux-VServer; a tiny struct for each process umbrella, that will generally fit in one or two cachelines, to which the scheduling support adds four ints and a spinlock. With this it achieves fair CPU scheduling between vservers. Sam.
creator: Sam Vilain
date: 2006-02-09T22:09:00-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: ebiederm@xmission.com (Eric W. Biederman) writes: > I think I can boil the discussion down into some of the fundamental > questions that we are facing. > > Currently everyone seems to agree that we need something like > my namespace concept that isolates multiple resources. > > We need these for > PIDS > UIDS > SYSVIPC > NETWORK > UTSNAME > FILESYSTEM > etc. > > The questions seem to break down into: > 1) Where do we put the references to the different namespaces? > - Do we put the references in a struct container that we reference from > struct task_struct? > - Do we put the references directly in struct task_struct? Answer in the task_struct. It is the simplest and most flexible route and the other implementations are still possible. > 2) What is the syscall interface to create these namespaces? > - Do we add clone flags? > (Plan 9 style) > - Do we add a syscall (similar to setsid) per namespace? > (Traditional unix style)? > - Do we in addition add syscalls to manipulate containers generically? The answer seems to be we decide on a per namespace basis with additional syscalls being mandatory if we have any additional data to pass. > 3) How do we refer to namespaces and containers when we are not members? I have seen no arguments against referring to namespaces or containers by global ids. So it seems we do not need a container id. > 4) How do we implement each of these namespaces? > Besides being maintainable are there other constraints? Largely quite. But I have not heard additional constraints. > 5) How do we control the resource inside a namespace starting > from a process that is outside of that namespace? > - The filesystem mount namespace gave an interesting answer. > So it is quite possible other namespaces will give > equally interesting and surprising answers. Not yet resolved, but a bit of speculation. > 6) How do we do all of this efficiently without a noticeable impact on > performance? > - I have already heard concerns that I might be introducing cac…
creator: ebiederm
date: 2006-02-09T22:25:35-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: >> The questions seem to break down into: >> 1) Where do we put the references to the different namespaces? >> - Do we put the references in a struct container that we reference >> from struct task_struct? >> - Do we put the references directly in struct task_struct? > > > You "cache" task_struct->container->hotsubsys under > task_struct->hotsubsys. > We don't change containers other then at clone time, so no coherency > issue here !!!! > Which subsystems pointers to "cache", should be agreed by the experts, > but first approach should always not to cache and go through the container. agreed. I see no much reason to cache it and make tons of the same pointers in all the tasks. Only if needed. Also, in OpenVZ container has many fields intergrated inside, so there is no additional dereference, but task->container->subsys_field >> 2) What is the syscall interface to create these namespaces? >> - Do we add clone flags? (Plan 9 style) > Like that approach .. flexible .. particular when one has well specified > namespaces. mmm, how do you plan to pass additional flags to clone()? e.g. strong or weak isolation of pids? another questions: how do you plan to meet the dependancies between namespaces? e.g. conntracks require netfilters to be initialized. network requires sysctls and proc to be initialized and so on. do you propose to track all this in clone()? huh... >> - Do we add a syscall (similar to setsid) per namespace? >> (Traditional unix style)? can be so... >> - Do we in addition add syscalls to manipulate containers generically? >> >> I don't think having a single system call to create a container and >> a new >> instance of each namespace is reasonable as that does not give us a >> path into the future when we create yet another namespace. >> > Agreed. why do you think so? this syscalls will start handling this new namespace and that's all. this is not different from many syscalls approach. >> 4) How do we implement each of these namespaces? >> Besides being mainta…
creator: dev
date: 2006-02-20T12:08:58-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: On Mon, Feb 20, 2006 at 03:11:32PM +0300, Kirill Korotaev wrote: > >>The questions seem to break down into: > >>1) Where do we put the references to the different namespaces? > >> - Do we put the references in a struct container that we reference > >>from struct task_struct? > >> - Do we put the references directly in struct task_struct? > > > > > >You "cache" task_struct->container->hotsubsys under > >task_struct->hotsubsys. > >We don't change containers other then at clone time, so no coherency > >issue here !!!! > >Which subsystems pointers to "cache", should be agreed by the experts, > >but first approach should always not to cache and go through the container. > agreed. I see no much reason to cache it and make tons of the same > pointers in all the tasks. Only if needed. > Also, in OpenVZ container has many fields intergrated inside, so there > is no additional dereference, but task->container->subsys_field as does Linux-VServer currently, but do you have any proof that putting all the fields together in one big structure actually has any (dis)advantage over separate structures? > >>2) What is the syscall interface to create these namespaces? > >> - Do we add clone flags? (Plan 9 style) > >Like that approach .. flexible .. particular when one has well > >specified namespaces. > mmm, how do you plan to pass additional flags to clone()? > e.g. strong or weak isolation of pids? do you really have to pass them at clone() time? would shortly after be more than enough? what if you want to change those properties later? > another questions: > how do you plan to meet the dependancies between namespaces? > e.g. conntracks require netfilters to be initialized. > network requires sysctls and proc to be initialized and so on. > do you propose to track all this in clone()? huh... this is missing isolation/virtualization, and I guess it has to be done to make those spaces useful ... > >> - Do we add a syscall (similar to setsid) per namespace? > >> (Traditional unix style)?…
creator: Herbert Poetzl
date: 2006-02-20T12:40:55-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: > as does Linux-VServer currently, but do you have > any proof that putting all the fields together in > one big structure actually has any (dis)advantage > over separate structures? have no proof and don't mind if there are many pointers. Though this doesn't look helpful to me as well. >>mmm, how do you plan to pass additional flags to clone()? >>e.g. strong or weak isolation of pids? > do you really have to pass them at clone() time? > would shortly after be more than enough? > what if you want to change those properties later? I don't think it is always suiatable to do configuration later. We had races in OpenVZ on VPS create/stop against exec/enter etc. (even introduced flag is_running). So I have some experience to believe it will be painfull place. >>this syscalls will start handling this new namespace and that's all. >>this is not different from many syscalls approach. > well, let's defer the 'how amny syscalls' issue to > a later time, when we know what we want to implement :) agreed. Kirill
creator: dev
date: 2006-02-20T14:25:35-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: On Mon, Feb 20, 2006 at 05:26:13PM +0300, Kirill Korotaev wrote: >> as does Linux-VServer currently, but do you have >> any proof that putting all the fields together in >> one big structure actually has any (dis)advantage >> over separate structures? > have no proof and don't mind if there are many pointers. > Though this doesn't look helpful to me as well. well, my point is just that we don't know yet so we should not favor one over the other, just because somebody did it like that and it didn't hurt :) >>> mmm, how do you plan to pass additional flags to clone()? >>> e.g. strong or weak isolation of pids? >> do you really have to pass them at clone() time? >> would shortly after be more than enough? >> what if you want to change those properties later? > I don't think it is always suiatable to do configuration later. > We had races in OpenVZ on VPS create/stop against exec/enter etc. > (even introduced flag is_running). > So I have some experience to believe it will be painfull place. well, Linux-VServer uses a state called 'setup' which allows to change all kinds of things before the guest can be entered, this state is changed as the last operation of the setup, which in turn drops all the capabilities and makes the guest visible to the outside ... works quite well and seems to be free of those races you mentioned ... >>> this syscalls will start handling this new namespace and that's all. >>> this is not different from many syscalls approach. >> well, let's defer the 'how amny syscalls' issue to >> a later time, when we know what we want to implement :) > agreed. btw, maybe it's just me, but would it be possible to do the email quoting like this: >>> Text instead of > >>Text TIA, Herbert > Kirill
creator: Herbert Poetzl
date: 2006-02-20T15:16:14-00:00

Re: The issues for agreeing on a virtualization/namespaces implementation.
type: item
description: If you guys want to continue to argue this point forever please drop me from cc. Thanks. -Andi
creator: Andi Kleen
date: 2006-02-20T15:17:51-00:00