OpenVZ Forum - RDF feed
type: channel
items: [PATCH 00/10] Containers(V10): Generic Process Containers
items: [PATCH 07/10] Containers(V10): Make cpusets a client of containers
items: [PATCH 09/10] Containers(V10): Simple debug info subsystem
items: Re: [PATCH 00/10] Containers(V10): Generic Process Containers
items: Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers

[PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: This is an update to my multi-hierarchy multi-subsystem generic process containers patch. Changes since V9 (April 27th) include: - The patchset has been rebased over 2.6.22-rc2-mm1 - A lattice of lists linking tasks to their css_groups and css_groups to their containers has been added to support more efficient iteration across the member tasks of a container. - Support for the cpusets "release agent" functionality has been added back in; this is based on a workqueue concept similar to the changes that Cliff Wickman has been pushing for supporting CPU hot-unplug. - Several uses of tasklist_lock replaced by reliance on RCU - Misc cleanups - Tested with a tweaked version of PaulJ's cpuset_test script Still TODO: - decide whether "Containers" is an acceptable name for the system given its usage by some other development groups, or whether something else (ProcessSets? ResourceGroups? TaskGroups?) would be better. I'm inclined to leave this political decision to Andrew/Linus once they're happy with the technical aspects of the patches. - add a hash-table based lookup for css_group objects. - use seq_file properly in container tasks files to avoid having to allocate a big array for all the container's task pointers. - lots more testing - define standards for container file names -- Generic Process Containers -------------------------- There have recently been various proposals floating around for resource management/accounting and other task grouping subsystems in the kernel, including ResGroups, User BeanCounters, NSProxy containers, and others. These all need the basic abstraction of being able to group together multiple processes in an aggregate, in order to track/limit the resources permitted to those processes, or control other behaviour of the processes, and all implement this grouping in different ways. Already existing in the kernel is the cpuset subsystem; this has a process grouping mechanism that is mature, tested, and well documented (particularly with regards …
creator: Paul Menage
date: 2007-05-29T13:01:04-00:00

[PATCH 07/10] Containers(V10): Make cpusets a client of containers
type: item
description: This patch removes the filesystem support logic from the cpusets system and makes cpusets a container subsystem Signed-off-by: Paul Menage <menage@google.com> --- Documentation/cpusets.txt | 91 +-- fs/proc/base.c | 4 include/linux/container_subsys.h | 6 include/linux/cpuset.h | 12 include/linux/mempolicy.h | 12 include/linux/sched.h | 3 init/Kconfig | 6 kernel/cpuset.c | 1151 +++++---------------------------------- kernel/exit.c | 2 kernel/fork.c | 3 mm/mempolicy.c | 2 11 files changed, 241 insertions(+), 1051 deletions(-) Index: container-2.6.22-rc2-mm1/Documentation/cpusets.txt ============================================================ ======= --- container-2.6.22-rc2-mm1.orig/Documentation/cpusets.txt +++ container-2.6.22-rc2-mm1/Documentation/cpusets.txt @@ -7,6 +7,7 @@ Written by Simon.Derr@bull.net Portions Copyright (c) 2004-2006 Silicon Graphics, Inc. Modified by Paul Jackson <pj@sgi.com> Modified by Christoph Lameter <clameter@sgi.com> +Modified by Paul Menage <menage@google.com> CONTENTS: ========= @@ -16,10 +17,9 @@ CONTENTS: 1.2 Why are cpusets needed ? 1.3 How are cpusets implemented ? 1.4 What are exclusive cpusets ? - 1.5 What does notify_on_release do ? - 1.6 What is memory_pressure ? - 1.7 What is memory spread ? - 1.8 How do I use cpusets ? + 1.5 What is memory_pressure ? + 1.6 What is memory spread ? + 1.7 How do I use cpusets ? 2. Usage Examples and Syntax 2.1 Basic Usage 2.2 Adding/removing cpus @@ -43,18 +43,19 @@ hierarchy visible in a virtual file syst hooks, beyond what is already present, required to manage dynamic job placement on large systems. -Each task has a pointer to a cpuset. Multiple tasks may reference -the same cpuset. Requests by a task, using the sched_setaffinity(2) -system call to include CPUs in its CPU affinity mask, and using the -mbind(2) and set_mempolicy(2) system calls to include Memory Nodes -in its memory policy, are both filtered through that tasks cpuset, -filtering out any CPUs or Memory Nodes not in that cpuset…
creator: Paul Menage
date: 2007-05-29T13:01:11-00:00

[PATCH 09/10] Containers(V10): Simple debug info subsystem
type: item
description: This example subsystem exports debugging information as an aid to diagnosing refcount leaks, etc, in the container framework. Signed-off-by: Paul Menage <menage@google.com> --- include/linux/container_subsys.h | 4 + init/Kconfig | 10 ++++ kernel/Makefile | 1 kernel/container_debug.c | 89 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 104 insertions(+) Index: container-2.6.22-rc2-mm1/include/linux/container_subsys.h ============================================================ ======= --- container-2.6.22-rc2-mm1.orig/include/linux/container_subsys .h +++ container-2.6.22-rc2-mm1/include/linux/container_subsys.h @@ -19,4 +19,8 @@ SUBSYS(cpuset) /* */ +#ifdef CONFIG_CONTAINER_DEBUG +SUBSYS(debug) +#endif + /* */ Index: container-2.6.22-rc2-mm1/init/Kconfig ============================================================ ======= --- container-2.6.22-rc2-mm1.orig/init/Kconfig +++ container-2.6.22-rc2-mm1/init/Kconfig @@ -306,6 +306,16 @@ config LOG_BUF_SHIFT config CONTAINERS bool +config CONTAINER_DEBUG + bool "Example debug container subsystem" + select CONTAINERS + help + This option enables a simple container subsystem that + exports useful debugging information about the containers + framework + + Say N if unsure + config CPUSETS bool "Cpuset support" depends on SMP Index: container-2.6.22-rc2-mm1/kernel/container_debug.c ============================================================ ======= --- /dev/null +++ container-2.6.22-rc2-mm1/kernel/container_debug.c @@ -0,0 +1,89 @@ +/* + * kernel/ccontainer_debug.c - Example container subsystem that + * exposes debug info + * + * Copyright (C) Google Inc, 2007 + * + * Developed by Paul Menage (menage@google.com) + * + */ + +#include <linux/container.h> +#include <linux/fs.h> + +static int debug_create(struct container_subsys *ss, struct container *cont) +{ + struct container_subsys_state *css = kzalloc(sizeof(*css), GFP_KERNEL); + if (!css) + return -ENOMEM; + cont->subsys[debug_subsys_id] = css; + return 0; +} + +stat…
creator: Paul Menage
date: 2007-05-29T13:01:13-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: On Tue, 29 May 2007 06:01:04 -0700 menage@google.com wrote: > This is an update to my multi-hierarchy multi-subsystem generic > process containers patch. > > ... > > Still TODO: > > ... > > - lots more testing > So how do we do this? Is there any sneaky way in which we can modify the kernel so that this new code gets exercised more? Obviously, tossing init into some default system-wide container would be a start. But I wonder if we can be sneakier - for example, create a new container on each setuid(), toss the task into that. Or something along those lines?
creator: Andrew Morton
date: 2007-05-30T07:14:55-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: On Wed, May 30, 2007 at 12:14:55AM -0700, Andrew Morton wrote: > So how do we do this? > Is there any sneaky way in which we can modify the kernel so that this new > code gets exercised more? Obviously, tossing init into some default > system-wide container would be a start. But I wonder if we can be > sneakier - for example, create a new container on each setuid(), toss the > task into that. Or something along those lines? How about a container for each thread group, pgrp, session, and user? -- wli
creator: William Lee Irwin III
date: 2007-05-30T07:39:59-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Andrew Morton wrote: > On Tue, 29 May 2007 06:01:04 -0700 menage@google.com wrote: > >> This is an update to my multi-hierarchy multi-subsystem generic >> process containers patch. >> >> ... >> >> Still TODO: >> >> ... >> >> - lots more testing >> > > So how do we do this? > > Is there any sneaky way in which we can modify the kernel so that this new > code gets exercised more? Obviously, tossing init into some default > system-wide container would be a start. But I wonder if we can be > sneakier - for example, create a new container on each setuid(), toss the > task into that. Or something along those lines? Please, lets get the RSS controller in. It's ready, been tested and commented on widely. I'll also start porting my containerstats patches on top of -v10. -- Thanks, Balbir Singh Linux Technology Center IBM, ISTL
creator: Balbir Singh
date: 2007-05-30T08:09:37-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Balbir Singh wrote: > Andrew Morton wrote: >> On Tue, 29 May 2007 06:01:04 -0700 menage@google.com wrote: >> >>> This is an update to my multi-hierarchy multi-subsystem generic >>> process containers patch. >>> >>> ... >>> >>> Still TODO: >>> >>> ... >>> >>> - lots more testing >>> >> So how do we do this? >> >> Is there any sneaky way in which we can modify the kernel so that this new >> code gets exercised more? Obviously, tossing init into some default >> system-wide container would be a start. But I wonder if we can be >> sneakier - for example, create a new container on each setuid(), toss the >> task into that. Or something along those lines? > > Please, lets get the RSS controller in. It's ready, been tested It is not 100% ready yet actually :) I am working on it right now and hope to get ready till tomorrow. > and commented on widely. Yup :) Balbir, thanks for testing, your patches are already in. > I'll also start porting my containerstats patches on top of -v10. >
creator: Pavel Emelianov
date: 2007-05-30T08:58:29-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Pavel Emelianov wrote: >>> Is there any sneaky way in which we can modify the kernel so that this new >>> code gets exercised more? Obviously, tossing init into some default >>> system-wide container would be a start. But I wonder if we can be >>> sneakier - for example, create a new container on each setuid(), toss the >>> task into that. Or something along those lines? >> Please, lets get the RSS controller in. It's ready, been tested > > It is not 100% ready yet actually :) I am working on it right now > and hope to get ready till tomorrow. > By ready, I meant ready for inclusion as a concept/approach. >> and commented on widely. > > Yup :) Balbir, thanks for testing, your patches are already in. > Thanks for including them. -- Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
creator: Balbir Singh
date: 2007-05-30T09:02:47-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Hi Paul. I have faced a warning during testing your patches. The testcase is simple: # ssh to the node mount -t container none /cnt/rss/ -o rss mkdir /cnt/rss/0 /bin/echo $$ > /cnt/rss/0/tasks # exit with ^d and ssh again rmdir /cnt/rss/0 dmesg BUG: at mm/slab.c:777 __find_general_cachep() [<c04656c8>] __kmalloc+0x3f/0xa5 [<c0440e3a>] container_tasks_open+0x56/0x11f [<c0440bcc>] container_file_open+0x0/0x36 [<c0440bfb>] container_file_open+0x2f/0x36 [<c0467a12>] __dentry_open+0xc1/0x178 [<c0467b43>] nameidata_to_filp+0x24/0x33 [<c0467b84>] do_filp_open+0x32/0x39 [<c04678eb>] get_unused_fd+0x50/0xb6 [<c0467bcd>] do_sys_open+0x42/0xbe [<c0467c82>] sys_open+0x1c/0x1e [<c0404c12>] sysenter_past_esp+0x5f/0x85 [<c05b0000>] __xfrm_policy_check+0x11a/0x4f6 The bug seems to be here: static int container_tasks_open(struct inode *unused, struct file *file) { ... npids = container_task_count(cont); pidarray = kmalloc(npids * sizeof(pid_t), GFP_KERNEL); if (!pidarray) goto err1; ... } The npids happened to be 0 and kmalloc warns that size is zero. Thanks, Pavel.
creator: Pavel Emelianov
date: 2007-05-30T10:44:24-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Hi Paul, I've got two problems working with this patchset: 1. A task can't join a cpuset unless 'cpus' and 'mems' are set. These don't seem to automatically inherit the parent's values. So when I do mount -t container -o ns,cpuset nsproxy /containers (unshare a namespace) the unshare fails because container_clone() created a new cpuset container but the task couldn't automatically enter that new cpuset. 2. I can't delete containers because of the files they contain, and am not allowed to delete those files by hand. thanks, -serge
creator: serue
date: 2007-06-04T19:14:12-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: What you describe, Serge, sounds like semantics carried over from cpusets. Serge wrote: > A task can't join a cpuset unless 'cpus' and 'mems' are set. Yup - don't want to run a task in a cpuset that lacks cpu, or lacks memory. Hard to run without those. > These don't seem to automatically inherit the parent's values Yup - early in the life of cpusets, a created cpuset inherited the cpus and mems of its parent. But that broke the exclusive property big time. You will recall that a cpu_exclusive or mem_exclusive cpuset cannot overlap the cpus or memory, respectively, of any of its sibling cpusets. So we changed it to creating new cpusets empty of cpus or memory. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2007-06-04T19:31:51-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: On 6/4/07, Paul Jackson <pj@sgi.com> wrote: > > Yup - early in the life of cpusets, a created cpuset inherited the cpus > and mems of its parent. But that broke the exclusive property big > time. You will recall that a cpu_exclusive or mem_exclusive cpuset > cannot overlap the cpus or memory, respectively, of any of its sibling > cpusets. > Maybe we could make it a per-cpuset option whether children should inherit mems/cpus or not? Paul
creator: Paul Menage
date: 2007-06-04T20:30:41-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: On 6/4/07, Serge E. Hallyn <serue@us.ibm.com> wrote: > > 2. I can't delete containers because of the files they contain, and > am not allowed to delete those files by hand. > You should be able to delete a container with rmdir as long as it's not in use - its control files will get cleaned up automatically. If you're getting an EBUSY error that means that either there are still tasks running in the container (look in the "tasks" file) or else there's a reference counting bug somewhere. Can you post an example to reproduce the problem? Paul
creator: Paul Menage
date: 2007-06-04T20:32:06-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Paul M wrote: > Maybe we could make it a per-cpuset option whether children should > inherit mems/cpus or not? I suppose, if those needing inherited mems/cpus need it bad enough. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2007-06-04T20:37:29-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Menage (menage@google.com): > On 6/4/07, Paul Jackson <pj@sgi.com> wrote: > > > >Yup - early in the life of cpusets, a created cpuset inherited the cpus > >and mems of its parent. But that broke the exclusive property big > >time. You will recall that a cpu_exclusive or mem_exclusive cpuset > >cannot overlap the cpus or memory, respectively, of any of its sibling > >cpusets. > > > > Maybe we could make it a per-cpuset option whether children should > inherit mems/cpus or not? The values can be changed after the cpuset is populated, right? So really these are just defaults? Would it then make sense to just default to (parent_set - sibling_exclusive_set) for a new sibling's value? An option is fine with me, but without such an option at all, cpusets could not be applied to namespaces... thanks, -serge
creator: serue
date: 2007-06-04T20:41:31-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Menage (menage@google.com): > On 6/4/07, Serge E. Hallyn <serue@us.ibm.com> wrote: > > > >2. I can't delete containers because of the files they contain, and > >am not allowed to delete those files by hand. > > > > You should be able to delete a container with rmdir as long as it's > not in use - its control files will get cleaned up automatically. > > If you're getting an EBUSY error that means that either there are > still tasks running in the container (look in the "tasks" file) or > else there's a reference counting bug somewhere. > > Can you post an example to reproduce the problem? here is an excerpt: [root@linuz11 root]# mount -t container -ocpuset cpuset /containers/ [root@linuz11 root]# ls /containers/ cpu_exclusive memory_pressure mems tasks cpus memory_pressure_enabled notify_on_release mem_exclusive memory_spread_page releasable memory_migrate memory_spread_slab release_agent [root@linuz11 root]# mkdir /containers/1 [root@linuz11 root]# echo 0 > /containers/1/mems [root@linuz11 root]# echo 0 > /containers/1/cpus [root@linuz11 root]# sh sh-2.05b# echo $$ > /containers/1/tasks sh-2.05b# cat /containers/1/tasks 4325 4326 sh-2.05b# exit [root@linuz11 root]# ls /containers/1/tasks /containers/1/tasks [root@linuz11 root]# rm -rf /containers/1 rm: cannot remove `/containers/1/memory_spread_slab': Operation not permitted rm: cannot remove `/containers/1/memory_spread_page': Operation not permitted rm: cannot remove `/containers/1/memory_pressure': Operation not permitted rm: cannot remove `/containers/1/memory_migrate': Operation not permitted rm: cannot remove `/containers/1/mem_exclusive': Operation not permitted rm: cannot remove `/containers/1/cpu_exclusive': Operation not permitted rm: cannot remove `/containers/1/mems': Operation not permitted rm: cannot remove `/containers/1/cpus': Operation not permitted rm: cannot remove `/containers/1/releasable': Operation not permitted rm: cannot remove `/containers/1/notify_on_release': Operation not pe…
creator: serue
date: 2007-06-04T20:51:06-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: On 6/4/07, Serge E. Hallyn <serue@us.ibm.com> wrote: > root@linuz11 root]# rm -rf /containers/1 Just use "rmdir /containers/1" here. > > Ah, I see the second time I typed 'ls /containers/1/tasks' instead of > cat. When I then used cat, the file was empty, and I got an oops just > like Pavel reported. I bet if I solve the problem he reported, then I > solve my problem :) > As far as I could see, Pavel's problem wasn't actually an Oops, it was a WARN_ON() when allocating a zero length chunk of memory. There's ongoing discussion as to whether this counts as a problem with the allocators or the kmalloc() code, since it used to be OK to allocate a zero-length chunk. Paul
creator: Paul Menage
date: 2007-06-04T20:56:12-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: > Would it then make sense to just > default to (parent_set - sibling_exclusive_set) for a new sibling's > value? Which could well be empty, which in turn puts one back in the position of dealing with a newborn cpuset that is empty (of cpus or of memory), or else it introduces a new and odd constraint on when cpusets can be created (only when there are non-exclusive cpus and mems available.) > An option is fine with me, but without such an option at all, cpusets > could not be applied to namespaces... I wasn't paying close enough attention to understand why you couldn't do it in two steps - make the container, and then populate it with resources. But if indeed that's not possible, then I guess we need some sort of option specifying whether to create kids empty, or inheriting. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2007-06-04T21:05:33-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Menage (menage@google.com): > On 6/4/07, Serge E. Hallyn <serue@us.ibm.com> wrote: > >root@linuz11 root]# rm -rf /containers/1 > > Just use "rmdir /containers/1" here. Hmm. Ok, that works... Odd, I thought rm -rf used to work in the past, but i'm likely wrong. thanks, -serge > >Ah, I see the second time I typed 'ls /containers/1/tasks' instead of > >cat. When I then used cat, the file was empty, and I got an oops just > >like Pavel reported. I bet if I solve the problem he reported, then I > >solve my problem :) > > > > As far as I could see, Pavel's problem wasn't actually an Oops, it was > a WARN_ON() when allocating a zero length chunk of memory. There's > ongoing discussion as to whether this counts as a problem with the > allocators or the kmalloc() code, since it used to be OK to allocate a > zero-length chunk. > > Paul
creator: serue
date: 2007-06-04T21:11:03-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Serge wrote: > Odd, I thought rm -rf used to work in the past, > but i'm likely wrong. I'm pretty sure it never worked. And I've probably tested it myself, every few months, since the birth of cpusets, when I forget and type it again, and then stare dumbly at the screen wondering what all the complaining is about. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2007-06-04T21:16:17-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Jackson (pj@sgi.com): > > Would it then make sense to just > > default to (parent_set - sibling_exclusive_set) for a new sibling's > > value? > > Which could well be empty, which in turn puts one back in the position > of dealing with a newborn cpuset that is empty (of cpus or of memory), > or else it introduces a new and odd constraint on when cpusets can be > created (only when there are non-exclusive cpus and mems available.) > > > An option is fine with me, but without such an option at all, cpusets > > could not be applied to namespaces... > > I wasn't paying close enough attention to understand why you couldn't > do it in two steps - make the container, and then populate it with > resources. Sorry, please clarify - are you saying that now you do understand, or that I should explain? > But if indeed that's not possible, then I guess we need some sort of > option specifying whether to create kids empty, or inheriting. Paul (uh, Menage :) should I do a patch for this or have you got it already? thanks, -serge
creator: serue
date: 2007-06-06T22:39:52-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: > > I wasn't paying close enough attention to understand why you couldn't > > do it in two steps - make the container, and then populate it with > > resources. > > Sorry, please clarify - are you saying that now you do understand, or > that I should explain? Could you explain -- I still don't understand why you need this option. I still don't understand why you can't do it in two steps - make the container, then add cpu/mem separately. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2007-06-06T22:43:47-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Jackson (pj@sgi.com): > > > I wasn't paying close enough attention to understand why you couldn't > > > do it in two steps - make the container, and then populate it with > > > resources. > > > > Sorry, please clarify - are you saying that now you do understand, or > > that I should explain? > > Could you explain -- I still don't understand why you need this option. > I still don't understand why you can't do it in two steps - make the > container, then add cpu/mem separately. Sure - the key is that the ns subsystem uses container_clone() to automatically create a new container (on sys_unshare() or clone(2) with certain flags) and move the current task into it. Let's say we have done mount -t container -o ns,cpuset nsproxy /containers and we, as task 875, happen to be in the topmost container: /containers/ Now we fork task 999 which does an unshare(CLONE_NEWNS), or we just clone(CLONE_NEWNS). This will create /containers/node_999 and move task 999 into that container. Except that when it tries attach_task() it is refused by cpuset. So the container_clone() fails, and in turn the sys_unshare() or clone() fails. A login making use of the pam_namespace.so library would fail this way with the ns and cpuset subsystems composed. We could special case this by having kernel/container.c:container_clone() check whether one of the subsystems is cpusets and, if so, setting the defaults for mems and cpus, but that is kind of ugly. I suppose as a cleaner alternative we could add a container_subsys->inherit_defaults() handler, to be called at container_clone(), and for cpusets this would set cpus and mems to the parent values - sibling exclusive values. If that comes to nothing, then the attach_task() is still refused, and the unshare() or clone() fails, but this time with good reason. thanks, -serge
creator: serue
date: 2007-06-07T00:05:59-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: > I suppose as a cleaner alternative we could > add a container_subsys->inherit_defaults() handler, to be called at > container_clone(), and for cpusets this would set cpus and mems to > the parent values - sibling exclusive values. If that comes to nothing, > then the attach_task() is still refused, and the unshare() or clone() > fails, but this time with good reason. Unfortunately, I haven't spent the time I should thinking about container cloning, namespaces and such. I don't know, for the workloads that matter to me, when, how or if this container cloning will be used. I'm tempted to suggest the following. First, I am assuming that the classic method of creating cpuset children will still work, such as the following (which can fail for certain combinations of exclusive cpus or mems): cd /dev/cpuset/foobar mkdir foochild cp cpus foochild cp mems foochild echo $$ > foochild/tasks Second, given that, how about you fail the unshare() or clone() anytime that the instance to be cloned has any sibling cpusets with any exclusive flags set. The exclusive property is not really on friendly terms with cloning. Now if the above classic code must be encoded using cloning under the covers, then we've got problems, probably more problems than just this. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2007-06-07T00:46:09-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Jackson (pj@sgi.com): > > I suppose as a cleaner alternative we could > > add a container_subsys->inherit_defaults() handler, to be called at > > container_clone(), and for cpusets this would set cpus and mems to > > the parent values - sibling exclusive values. If that comes to nothing, > > then the attach_task() is still refused, and the unshare() or clone() > > fails, but this time with good reason. > > Unfortunately, I haven't spent the time I should thinking about > container cloning, namespaces and such. > > I don't know, for the workloads that matter to me, when, how or > if this container cloning will be used. > > I'm tempted to suggest the following. > > First, I am assuming that the classic method of creating cpuset > children will still work, such as the following (which can fail > for certain combinations of exclusive cpus or mems): > cd /dev/cpuset/foobar > mkdir foochild > cp cpus foochild > cp mems foochild > echo $$ > foochild/tasks > > Second, given that, how about you fail the unshare() or clone() > anytime that the instance to be cloned has any sibling cpusets > with any exclusive flags set. The below patch (on top of my previous patch) does basically that. But I wasn't able to test it, bc i wasn't able to set cpus_exclusive... For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0 also have to have it exclusively? If so, then clearly this approach won't work, since if any container has exclusive cpus, then every container will have siblings with exclusive cpus, and unshare still isn't possible on the system. > The exclusive property is not really on friendly terms with cloning. > > Now if the above classic code must be encoded using cloning under > the covers, then we've got problems, probably more problems than > just this. > > -- > I won't rest till it's the best ... > Programmer, Linux Scalability > Paul Jackson <pj@sgi.com> 1.925.600.0401 thanks, -serge >From 821de58b6ba446e50225606e907baac00130586c Mon Sep 17 00:00…
creator: serue
date: 2007-06-07T18:01:58-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: > For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0 > also have to have it exclusively? Yes. > If so, then clearly this approach won't work, since if any container has > exclusive cpus, then every container will have siblings with exclusive > cpus, and unshare still isn't possible on the system. Well, if I'm following you, not exactly. If we have some exclusive flags set, then every top level container will have exclusive siblings, but further down the hierarchy, some subtree might be entirely free of any exclusive settings. Then nodes below the top of that subtree would not have exclusive set, and would not have any exclusive siblings. But, overall, yeah, exclusive is no friend of container cloning. I just wish I had been thinking harder about how container cloning will impact my life, and the lives of the customers in my cpuset intensive corner of the world. There are certainly a whole bunch of people who will never have any need for exclusive cpusets. Perhaps (speculating wildly from great ignorance) there are a whole bunch of people who will never have need for container cloning. And perhaps, hoping to get lucky here, the set of people who need both at the same time on the same system is sufficiently close to empty that we can just tell them tough toenails - you cannot do both at once. How wide spread will be the use of container cloning, if it proceeds as envisioned? The set of people using exclusive cpusets is roughly some subset of those running multiple, cpuset isolated, non-cooperating jobs on big iron, usually with the aid of a batch scheduler. Well, that's what I am aware of anyway. If there are any other friends of exclusive cpusets lurking here, you might want to speak up, before I sell your interests down the river. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2007-06-07T19:21:21-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Jackson (pj@sgi.com): > > For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0 > > also have to have it exclusively? > > Yes. > > > If so, then clearly this approach won't work, since if any container has > > exclusive cpus, then every container will have siblings with exclusive > > cpus, and unshare still isn't possible on the system. > > Well, if I'm following you, not exactly. > > If we have some exclusive flags set, then every top level container > will have exclusive siblings, but further down the hierarchy, some > subtree might be entirely free of any exclusive settings. Then nodes > below the top of that subtree would not have exclusive set, and would > not have any exclusive siblings. > > But, overall, yeah, exclusive is no friend of container cloning. > > I just wish I had been thinking harder about how container cloning > will impact my life, and the lives of the customers in my cpuset > intensive corner of the world. > > There are certainly a whole bunch of people who will never have any > need for exclusive cpusets. > > Perhaps (speculating wildly from great ignorance) there are a whole > bunch of people who will never have need for container cloning. > > And perhaps, hoping to get lucky here, the set of people who need both > at the same time on the same system is sufficiently close to empty > that we can just tell them tough toenails - you cannot do both at once. > > How wide spread will be the use of container cloning, if it proceeds > as envisioned? It's not just container cloning, but all namespace unsharing. So uses include (1) providing 'polyinstantiated directory' functionality, i.e. private per-user /tmp's or per-security-level /tmp and /home's. (2) any virtual server usage (3) hpc checkpoint/restart users. > The set of people using exclusive cpusets is roughly some subset of > those running multiple, cpuset isolated, non-cooperating jobs on big > iron, usually with the aid of a batch scheduler. Unfortunately I would…
creator: serue
date: 2007-06-07T20:17:23-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: > > The set of people using exclusive cpusets is roughly some subset of > > those running multiple, cpuset isolated, non-cooperating jobs on big > > iron, usually with the aid of a batch scheduler. > > Unfortunately I would imagine these users to be very intereseted in > providing checkpoint/restart/migrate functionality. Yup - such customers are very interested in checkpoint, restart, and migrate functionality. > Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4 > later, those cpus can just be taken away from set3? Yeah - that works, so far as I know (which isn't all that far ..') But both: 1) that (using whatever cpus are still available) and 2) my other idea, of not allowing any cloning of cpusets with exclusive siblings at all, looked a little ugly to me. For example, such customers as above would not appreciate having their checkpoint/restart/migrate fail in any case where there weren't spare non-exclusive cpus, which for users of the exclusive flag, is often the more common case. My rule of thumb when doing ugly stuff is to constrain it as best I can -- minimize what it allows. This led me to prefer (2) above over (1) above. Perhaps there's a better way to think of this ... When we clone like this for checkpoint/restart/migrate functionality, perhaps we are not really starting up a new, separate, competing job that should have its resources isolated and separated from the original. Perhaps instead we are firing up a convenient alter-ego of the original job, which will be co-operatively using the same resources by default. If that's the normal case, then it seems wrong to force the clone onto disjoint CPUs, or fail for lack thereof. So perhaps we should refine the meaning of 'exclusive', from: - no overlapping siblings to: - no overlapping siblings other than clones of ones self. Then default to cloning right on the same CPU resources as the original, possibly with both original and clone marked exclusive. -- I won't rest till it's th…
creator: Paul Jackson
date: 2007-06-07T22:01:13-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Jackson (pj@sgi.com): > > > The set of people using exclusive cpusets is roughly some subset of > > > those running multiple, cpuset isolated, non-cooperating jobs on big > > > iron, usually with the aid of a batch scheduler. > > > > Unfortunately I would imagine these users to be very intereseted in > > providing checkpoint/restart/migrate functionality. > > Yup - such customers are very interested in checkpoint, restart, and > migrate functionality. > > > Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4 > > later, those cpus can just be taken away from set3? > > Yeah - that works, so far as I know (which isn't all that far ..') > > But both: > 1) that (using whatever cpus are still available) and > 2) my other idea, of not allowing any cloning of cpusets with > exclusive siblings at all, > > looked a little ugly to me. > > For example, such customers as above would not appreciate having their > checkpoint/restart/migrate fail in any case where there weren't spare > non-exclusive cpus, which for users of the exclusive flag, is often the > more common case. > > My rule of thumb when doing ugly stuff is to constrain it as best > I can -- minimize what it allows. This led me to prefer (2) above > over (1) above. > > Perhaps there's a better way to think of this ... When we clone > like this for checkpoint/restart/migrate functionality, perhaps > we are not really starting up a new, separate, competing job that > should have its resources isolated and separated from the original. Depends on whether the cpus are allocated to a customer or to a job. For the most part I would expect any job to be restart either on a different machine, or at a different time, but of course it doesn't have to be that way. > Perhaps instead we are firing up a convenient alter-ego of the > original job, which will be co-operatively using the same resources > by default. If that's the normal case, then it seems wrong to > force the clone onto disjoint C…
creator: serge
date: 2007-06-08T14:32:50-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: On 6/8/07, Serge E. Hallyn <serge@hallyn.com> wrote: > > Anyway the patch I sent is simple enough, and if users end up demanding > the ability to better deal with exclusive cpusets, the patch will be > simple enough to extend by changing cpuset_auto_setup(), so let's > stick with that patch since it's your preference (IIUC). > Sounds good to me, although I think my preference would be to extend the "create()" subsystem callback with a "struct task_struct *clone_task" parameter that indicates that clone_task is cloning its own container; a subsystem like cpusets that needs to do additional setup at that point could inherit settings either from the parent or from clone_task's container (or something else) as desired. (It could also do permission checking based on properties of clone_task, etc). Paul
creator: Paul Menage
date: 2007-06-08T15:55:47-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: Quoting Paul Menage (menage@google.com): > On 6/8/07, Serge E. Hallyn <serge@hallyn.com> wrote: > > > >Anyway the patch I sent is simple enough, and if users end up demanding > >the ability to better deal with exclusive cpusets, the patch will be > >simple enough to extend by changing cpuset_auto_setup(), so let's > >stick with that patch since it's your preference (IIUC). > > > > Sounds good to me, although I think my preference would be to extend > the "create()" subsystem callback with a "struct task_struct > *clone_task" parameter that indicates that clone_task is cloning its > own container; a subsystem like cpusets that needs to do additional > setup at that point could inherit settings either from the parent or > from clone_task's container (or something else) as desired. (It could > also do permission checking based on properties of clone_task, etc). The problem is container_clone() doesn't call ->create explicitly, it does vfs_mkdir. So we have no real way of passing in clone_task. -serge
creator: serge
date: 2007-06-08T16:08:40-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: > Anyway the patch I sent is simple enough, and if users end up demanding > the ability to better deal with exclusive cpusets, the patch will be > simple enough to extend by changing cpuset_auto_setup(), so let's > stick with that patch since it's your preference (IIUC). Yeah - probably so. When someone gets serious about things like checkpoint, restart, and migrate functionality, based on this container cloning, working with cpusets, they will probably have to revisit this interaction with exclusive cpusets. Perhaps a comment could be put in the code, saying something like the above, so whomever does this will realize they are traveling in unchartered territory. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.925.600.0401
creator: Paul Jackson
date: 2007-06-08T17:37:56-00:00

Re: [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: On 5/30/07, William Lee Irwin III <wli@holomorphy.com> wrote: > On Wed, May 30, 2007 at 12:14:55AM -0700, Andrew Morton wrote: > > So how do we do this? > > Is there any sneaky way in which we can modify the kernel so that this new > > code gets exercised more? Obviously, tossing init into some default > > system-wide container would be a start. But I wonder if we can be > > sneakier - for example, create a new container on each setuid(), toss the > > task into that. Or something along those lines? > > How about a container for each thread group, pgrp, session, and user? > I've been thinking about this, and figured that it could be quite useful to be able to mount a container tree that groups tasks by userid or thread group - for doing per-user resource controls, for example, without having to write a controller that specifically handles the per-user case. One option would be to add a mount option, something like mount -t container -ogroupkey=<X> where X could be one of: uid, gid, pgrp, sid, tgid And put hooks in the various places where these ids could change, in order to move tasks between contaners as appropriate. But after some thought it seems to me that this is putting complexity in the kernel that probably doesn't belong there, and additionally is probably not sufficently flexible for some real-life situations. (E.g. the user wants all users in the "student" group to be lumped into the same container, but each user in the "professor" group gets their own container). So maybe this would be better handled in userspace? Have a daemon listing on a process connector socket, and move processes between containers based on notifications from the connector and user-defined rules. We'd probably also want to add some new connector events, such as PROC_EVENT_PGRP, and PROC_EVENT_SID A simple daemon that handles the case where we're classifying based on a single key with no complex rules shouldn't be too hard to write. It also sounds rather like the classification engine …
creator: Paul Menage
date: 2007-06-28T21:27:25-00:00

Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers
type: item
description: On Thu, Jun 28, 2007 at 05:27:25PM -0400, Paul Menage wrote: > So maybe this would be better handled in userspace? Have a daemon > listing on a process connector socket, and move processes between > containers based on notifications from the connector and user-defined > rules. > > We'd probably also want to add some new connector events, such as > PROC_EVENT_PGRP, and PROC_EVENT_SID Yep, this is what I did to test fair-user scheduling on top of my patches. Dhaval has a working program, which listens for UID change events and moves the task to approp. container. I will review that and have it posted soon. -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2007-06-28T22:13:22-00:00