Home » Mailing lists » Devel » [PATCH 00/10] Containers(V10): Generic Process Containers
| [PATCH 00/10] Containers(V10): Generic Process Containers [message #13545] |
Tue, 29 May 2007 13:01  |
Paul Menage
Messages: 642
Registered: September 2006
|
Senior Member |
|
|
This is an update to my multi-hierarchy multi-subsystem generic
process containers patch. Changes since V9 (April 27th) include:
- The patchset has been rebased over 2.6.22-rc2-mm1
- A lattice of lists linking tasks to their css_groups and css_groups
to their containers has been added to support more efficient iteration
across the member tasks of a container.
- Support for the cpusets "release agent" functionality has been added
back in; this is based on a workqueue concept similar to the changes
that Cliff Wickman has been pushing for supporting CPU hot-unplug.
- Several uses of tasklist_lock replaced by reliance on RCU
- Misc cleanups
- Tested with a tweaked version of PaulJ's cpuset_test script
Still TODO:
- decide whether "Containers" is an acceptable name for the system
given its usage by some other development groups, or whether something
else (ProcessSets? ResourceGroups? TaskGroups?) would be better. I'm
inclined to leave this political decision to Andrew/Linus once they're
happy with the technical aspects of the patches.
- add a hash-table based lookup for css_group objects.
- use seq_file properly in container tasks files to avoid having to
allocate a big array for all the container's task pointers.
- lots more testing
- define standards for container file names
--
Generic Process Containers
--------------------------
There have recently been various proposals floating around for
resource management/accounting and other task grouping subsystems in
the kernel, including ResGroups, User BeanCounters, NSProxy
containers, and others. These all need the basic abstraction of being
able to group together multiple processes in an aggregate, in order to
track/limit the resources permitted to those processes, or control
other behaviour of the processes, and all implement this grouping in
different ways.
Already existing in the kernel is the cpuset subsystem; this has a
process grouping mechanism that is mature, tested, and well documented
(particularly with regards to synchronization rules).
This patchset extracts the process grouping code from cpusets into a
generic container system, and makes the cpusets code a client of the
container system, along with a couple of simple example subsystems.
The patch set is structured as follows:
1) Basic container framework - filesystem and tracking structures
2) Simple CPU Accounting example subsystem
3) Support for the "tasks" control file
4) Hooks for fork() and exit()
5) Support for the container_clone() operation
6) Add /proc reporting interface
7) Make cpusets a container subsystem
8) Share container subsystem pointer arrays between tasks with the
same assignments
9) Simple container debugging subsystem
10) Support for a userspace "release agent", similar to the cpusets
release agent functionality
The intention is that the various resource management and
virtualization efforts can also become container clients, with the
result that:
- the userspace APIs are (somewhat) normalised
- it's easier to test out e.g. the ResGroups CPU controller in
conjunction with the BeanCounters memory controller, or use either of
them as the resource-control portion of a virtual server system.
- the additional kernel footprint of any of the competing resource
management systems is substantially reduced, since it doesn't need
to provide process grouping/containment, hence improving their
chances of getting into the kernel
Signed-off-by: Paul Menage <menage@google.com>
|
|
|
|
| [PATCH 09/10] Containers(V10): Simple debug info subsystem [message #13546 is a reply to message #13545] |
Tue, 29 May 2007 13:01  |
Paul Menage
Messages: 642
Registered: September 2006
|
Senior Member |
|
|
This example subsystem exports debugging information as an aid to
diagnosing refcount leaks, etc, in the container framework.
Signed-off-by: Paul Menage <menage@google.com>
---
include/linux/container_subsys.h | 4 +
init/Kconfig | 10 ++++
kernel/Makefile | 1
kernel/container_debug.c | 89 +++++++++++++++++++++++++++++++++++++++
4 files changed, 104 insertions(+)
Index: container-2.6.22-rc2-mm1/include/linux/container_subsys.h
============================================================ =======
--- container-2.6.22-rc2-mm1.orig/include/linux/container_subsys .h
+++ container-2.6.22-rc2-mm1/include/linux/container_subsys.h
@@ -19,4 +19,8 @@ SUBSYS(cpuset)
/* */
+#ifdef CONFIG_CONTAINER_DEBUG
+SUBSYS(debug)
+#endif
+
/* */
Index: container-2.6.22-rc2-mm1/init/Kconfig
============================================================ =======
--- container-2.6.22-rc2-mm1.orig/init/Kconfig
+++ container-2.6.22-rc2-mm1/init/Kconfig
@@ -306,6 +306,16 @@ config LOG_BUF_SHIFT
config CONTAINERS
bool
+config CONTAINER_DEBUG
+ bool "Example debug container subsystem"
+ select CONTAINERS
+ help
+ This option enables a simple container subsystem that
+ exports useful debugging information about the containers
+ framework
+
+ Say N if unsure
+
config CPUSETS
bool "Cpuset support"
depends on SMP
Index: container-2.6.22-rc2-mm1/kernel/container_debug.c
============================================================ =======
--- /dev/null
+++ container-2.6.22-rc2-mm1/kernel/container_debug.c
@@ -0,0 +1,89 @@
+/*
+ * kernel/ccontainer_debug.c - Example container subsystem that
+ * exposes debug info
+ *
+ * Copyright (C) Google Inc, 2007
+ *
+ * Developed by Paul Menage (menage@google.com)
+ *
+ */
+
+#include <linux/container.h>
+#include <linux/fs.h>
+
+static int debug_create(struct container_subsys *ss, struct container *cont)
+{
+ struct container_subsys_state *css = kzalloc(sizeof(*css), GFP_KERNEL);
+ if (!css)
+ return -ENOMEM;
+ cont->subsys[debug_subsys_id] = css;
+ return 0;
+}
+
+static void debug_destroy(struct container_subsys *ss, struct container *cont)
+{
+ kfree(cont->subsys[debug_subsys_id]);
+}
+
+static u64 container_refcount_read(struct container *cont, struct cftype *cft)
+{
+ return atomic_read(&cont->count);
+}
+
+static u64 taskcount_read(struct container *cont, struct cftype *cft)
+{
+ u64 count;
+ container_lock();
+ count = container_task_count(cont);
+ container_unlock();
+ return count;
+}
+
+static u64 current_css_group_read(struct container *cont, struct cftype *cft)
+{
+ return (u64) current->containers;
+}
+
+static u64 current_css_group_refcount_read(struct container *cont,
+ struct cftype *cft)
+{
+ u64 count;
+ rcu_read_lock();
+ count = atomic_read(¤t->containers->ref.refcount);
+ rcu_read_unlock();
+ return count;
+}
+
+static struct cftype files[] = {
+ {
+ .name = "debug.container_refcount",
+ .read_uint = container_refcount_read,
+ },
+ {
+ .name = "debug.taskcount",
+ .read_uint = taskcount_read,
+ },
+
+ {
+ .name = "debug.current_css_group",
+ .read_uint = current_css_group_read,
+ },
+
+ {
+ .name = "debug.current_css_group_refcount",
+ .read_uint = current_css_group_refcount_read,
+ },
+};
+
+static int debug_populate(struct container_subsys *ss, struct container *cont)
+{
+ return container_add_files(cont, files, ARRAY_SIZE(files));
+}
+
+struct container_subsys debug_subsys = {
+ .name = "debug",
+ .create = debug_create,
+ .destroy = debug_destroy,
+ .populate = debug_populate,
+ .subsys_id = debug_subsys_id,
+};
Index: container-2.6.22-rc2-mm1/kernel/Makefile
============================================================ =======
--- container-2.6.22-rc2-mm1.orig/kernel/Makefile
+++ container-2.6.22-rc2-mm1/kernel/Makefile
@@ -37,6 +37,7 @@ obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
obj-$(CONFIG_KEXEC) += kexec.o
obj-$(CONFIG_COMPAT) += compat.o
obj-$(CONFIG_CONTAINERS) += container.o
+obj-$(CONFIG_CONTAINER_DEBUG) += container_debug.o
obj-$(CONFIG_CPUSETS) += cpuset.o
obj-$(CONFIG_CONTAINER_CPUACCT) += cpu_acct.o
obj-$(CONFIG_IKCONFIG) += configs.o
--
|
|
|
|
| [PATCH 07/10] Containers(V10): Make cpusets a client of containers [message #13547 is a reply to message #13545] |
Tue, 29 May 2007 13:01 |
Paul Menage
Messages: 642
Registered: September 2006
|
Senior Member |
|
|
This patch removes the filesystem support logic from the cpusets
system and makes cpusets a container subsystem
Signed-off-by: Paul Menage <menage@google.com>
---
Documentation/cpusets.txt | 91 +--
fs/proc/base.c | 4
include/linux/container_subsys.h | 6
include/linux/cpuset.h | 12
include/linux/mempolicy.h | 12
include/linux/sched.h | 3
init/Kconfig | 6
kernel/cpuset.c | 1151 +++++----------------------------------
kernel/exit.c | 2
kernel/fork.c | 3
mm/mempolicy.c | 2
11 files changed, 241 insertions(+), 1051 deletions(-)
Index: container-2.6.22-rc2-mm1/Documentation/cpusets.txt
============================================================ =======
--- container-2.6.22-rc2-mm1.orig/Documentation/cpusets.txt
+++ container-2.6.22-rc2-mm1/Documentation/cpusets.txt
@@ -7,6 +7,7 @@ Written by Simon.Derr@bull.net
Portions Copyright (c) 2004-2006 Silicon Graphics, Inc.
Modified by Paul Jackson <pj@sgi.com>
Modified by Christoph Lameter <clameter@sgi.com>
+Modified by Paul Menage <menage@google.com>
CONTENTS:
=========
@@ -16,10 +17,9 @@ CONTENTS:
1.2 Why are cpusets needed ?
1.3 How are cpusets implemented ?
1.4 What are exclusive cpusets ?
- 1.5 What does notify_on_release do ?
- 1.6 What is memory_pressure ?
- 1.7 What is memory spread ?
- 1.8 How do I use cpusets ?
+ 1.5 What is memory_pressure ?
+ 1.6 What is memory spread ?
+ 1.7 How do I use cpusets ?
2. Usage Examples and Syntax
2.1 Basic Usage
2.2 Adding/removing cpus
@@ -43,18 +43,19 @@ hierarchy visible in a virtual file syst
hooks, beyond what is already present, required to manage dynamic
job placement on large systems.
-Each task has a pointer to a cpuset. Multiple tasks may reference
-the same cpuset. Requests by a task, using the sched_setaffinity(2)
-system call to include CPUs in its CPU affinity mask, and using the
-mbind(2) and set_mempolicy(2) system calls to include Memory Nodes
-in its memory policy, are both filtered through that tasks cpuset,
-filtering out any CPUs or Memory Nodes not in that cpuset. The
-scheduler will not schedule a task on a CPU that is not allowed in
-its cpus_allowed vector, and the kernel page allocator will not
-allocate a page on a node that is not allowed in the requesting tasks
-mems_allowed vector.
+Cpusets use the generic container subsystem described in
+Documentation/container.txt.
-User level code may create and destroy cpusets by name in the cpuset
+Requests by a task, using the sched_setaffinity(2) system call to
+include CPUs in its CPU affinity mask, and using the mbind(2) and
+set_mempolicy(2) system calls to include Memory Nodes in its memory
+policy, are both filtered through that tasks cpuset, filtering out any
+CPUs or Memory Nodes not in that cpuset. The scheduler will not
+schedule a task on a CPU that is not allowed in its cpus_allowed
+vector, and the kernel page allocator will not allocate a page on a
+node that is not allowed in the requesting tasks mems_allowed vector.
+
+User level code may create and destroy cpusets by name in the container
virtual file system, manage the attributes and permissions of these
cpusets and which CPUs and Memory Nodes are assigned to each cpuset,
specify and query to which cpuset a task is assigned, and list the
@@ -114,7 +115,7 @@ Cpusets extends these two mechanisms as
- Cpusets are sets of allowed CPUs and Memory Nodes, known to the
kernel.
- Each task in the system is attached to a cpuset, via a pointer
- in the task structure to a reference counted cpuset structure.
+ in the task structure to a reference counted container structure.
- Calls to sched_setaffinity are filtered to just those CPUs
allowed in that tasks cpuset.
- Calls to mbind and set_mempolicy are filtered to just
@@ -144,15 +145,10 @@ into the rest of the kernel, none in per
- in page_alloc.c, to restrict memory to allowed nodes.
- in vmscan.c, to restrict page recovery to the current cpuset.
-In addition a new file system, of type "cpuset" may be mounted,
-typically at /dev/cpuset, to enable browsing and modifying the cpusets
-presently known to the kernel. No new system calls are added for
-cpusets - all support for querying and modifying cpusets is via
-this cpuset file system.
-
-Each task under /proc has an added file named 'cpuset', displaying
-the cpuset name, as the path relative to the root of the cpuset file
-system.
+You should mount the "container" filesystem type in order to enable
+browsing and modifying the cpusets presently known to the kernel. No
+new system calls are added for cpusets - all support for querying and
+modifying cpusets is via this cpuset file system.
The /proc/<pid>/status file for each task has two added lines,
displaying the tasks cpus_allowed (on which CPUs it may be scheduled)
@@ -162,16 +158,15 @@ in the format seen in the following exam
Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff
Mems_allowed: ffffffff,ffffffff
-Each cpuset is represented by a directory in the cpuset file system
-containing the following files describing that cpuset:
+Each cpuset is represented by a directory in the container file system
+containing (on top of the standard container files) the following
+files describing that cpuset:
- cpus: list of CPUs in that cpuset
- mems: list of Memory Nodes in that cpuset
- memory_migrate flag: if set, move pages to cpusets nodes
- cpu_exclusive flag: is cpu placement exclusive?
- mem_exclusive flag: is memory placement exclusive?
- - tasks: list of tasks (by pid) attached to that cpuset
- - notify_on_release flag: run /sbin/cpuset_release_agent on exit?
- memory_pressure: measure of how much paging pressure in cpuset
In addition, the root cpuset only has the following file:
@@ -236,21 +231,7 @@ such as requests from interrupt handlers
outside even a mem_exclusive cpuset.
-1.5 What does notify_on_release do ?
-------------------------------------
-
-If the notify_on_release flag is enabled (1) in a cpuset, then whenever
-the last task in the cpuset leaves (exits or attaches to some other
-cpuset) and the last child cpuset of that cpuset is removed, then
-the kernel runs the command /sbin/cpuset_release_agent, supplying the
-pathname (relative to the mount point of the cpuset file system) of the
-abandoned cpuset. This enables automatic removal of abandoned cpusets.
-The default value of notify_on_release in the root cpuset at system
-boot is disabled (0). The default value of other cpusets at creation
-is the current value of their parents notify_on_release setting.
-
-
-1.6 What is memory_pressure ?
+1.5 What is memory_pressure ?
-----------------------------
The memory_pressure of a cpuset provides a simple per-cpuset metric
of the rate that the tasks in a cpuset are attempting to free up in
@@ -307,7 +288,7 @@ the tasks in the cpuset, in units of rec
times 1000.
-1.7 What is memory spread ?
+1.6 What is memory spread ?
---------------------------
There are two boolean flag files per cpuset that control where the
kernel allocates pages for the file system buffers and related in
@@ -378,7 +359,7 @@ data set, the memory allocation across t
can become very uneven.
-1.8 How do I use cpusets ?
+1.7 How do I use cpusets ?
--------------------------
In order to minimize the impact of cpusets on critical kernel
@@ -468,7 +449,7 @@ than stress the kernel.
To start a new job that is to be contained within a cpuset, the steps are:
1) mkdir /dev/cpuset
- 2) mount -t cpuset none /dev/cpuset
+ 2) mount -t container -ocpuset cpuset /dev/cpuset
3) Create the new cpuset by doing mkdir's and write's (or echo's) in
the /dev/cpuset virtual file system.
4) Start a task that will be the "founding father" of the new job.
@@ -480,7 +461,7 @@ For example, the following sequence of c
named "Charlie", containing just CPUs 2 and 3, and Memory Node 1,
and then start a subshell 'sh' in that cpuset:
- mount -t cpuset none /dev/cpuset
+ mount -t container -ocpuset cpuset /dev/cpuset
cd /dev/cpuset
mkdir Charlie
cd Charlie
@@ -512,7 +493,7 @@ Creating, modifying, using the cpusets c
virtual filesystem.
To mount it, type:
-# mount -t cpuset none /dev/cpuset
+# mount -t container -o cpuset cpuset /dev/cpuset
Then under /dev/cpuset you can find a tree that corresponds to the
tree of the cpusets in the system. For instance, /dev/cpuset
@@ -555,6 +536,18 @@ To remove a cpuset, just use rmdir:
This will fail if the cpuset is in use (has cpusets inside, or has
processes attached).
+Note that for legacy reasons, the "cpuset" filesystem exists as a
+wrapper around the container filesystem.
+
+The command
+
+mount -t cpuset X /dev/cpuset
+
+is equivalent to
+
+mount -t container -ocpuset X /dev/cpuset
+echo "/sbin/cpuset_release_agent" > /dev/cpuset/release_agent
+
2.2 Adding/removing cpus
------------------------
Index: container-2.6.22-rc2-mm1/include/linux/cpuset.h
============================================================ =======
--- container-2.6.22-rc2-mm1.orig/include/linux/cpuset.h
+++ container-2.6.22-rc2-mm1/include/linux/cpuset.h
@@ -11,6 +11,7 @@
#include <linux/sched.h>
#include <linux/cpumask.h>
#include <linux/nodemask.h>
+#include <linux/container.h>
#ifdef CONFIG_CPUSETS
@@ -19,8 +20,6 @@ extern int number_of_cpusets; /* How man
extern int cpuset_init_early(void);
extern int cpuset_init(void);
extern void cpuset_init_smp(void);
-extern void cpuset_fork(struct task_struct *p);
-extern void cpuset_exit(struct task_struct *p);
extern cpumask_t cpuset_cpus_allowed(struct task_struct *p);
extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
#defi
...
|
|
|
|
|
|
| Re: [PATCH 00/10] Containers(V10): Generic Process Containers [message #13570 is a reply to message #13564] |
Wed, 30 May 2007 07:39 |
William Lee Irwin III
Messages: 20
Registered: April 2007
|
Junior Member |
|
|
On Wed, May 30, 2007 at 12:14:55AM -0700, Andrew Morton wrote:
> So how do we do this?
> Is there any sneaky way in which we can modify the kernel so that this new
> code gets exercised more? Obviously, tossing init into some default
> system-wide container would be a start. But I wonder if we can be
> sneakier - for example, create a new container on each setuid(), toss the
> task into that. Or something along those lines?
How about a container for each thread group, pgrp, session, and user?
-- wli
|
|
|
|
|
|
|
|
| Re: [PATCH 00/10] Containers(V10): Generic Process Containers [message #13574 is a reply to message #13573] |
Wed, 30 May 2007 09:02 |
Balbir Singh
Messages: 491
Registered: August 2006
|
Senior Member |
|
|
Pavel Emelianov wrote:
>>> Is there any sneaky way in which we can modify the kernel so that this new
>>> code gets exercised more? Obviously, tossing init into some default
>>> system-wide container would be a start. But I wonder if we can be
>>> sneakier - for example, create a new container on each setuid(), toss the
>>> task into that. Or something along those lines?
>> Please, lets get the RSS controller in. It's ready, been tested
>
> It is not 100% ready yet actually :) I am working on it right now
> and hope to get ready till tomorrow.
>
By ready, I meant ready for inclusion as a concept/approach.
>> and commented on widely.
>
> Yup :) Balbir, thanks for testing, your patches are already in.
>
Thanks for including them.
--
Warm Regards,
Balbir Singh
Linux Technology Center
IBM, ISTL
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: [PATCH 00/10] Containers(V10): Generic Process Containers [message #13765 is a reply to message #13762] |
Mon, 04 June 2007 21:11 |
serue
Messages: 750
Registered: February 2006
|
Senior Member |
|
|
Quoting Paul Menage (menage@google.com):
> On 6/4/07, Serge E. Hallyn <serue@us.ibm.com> wrote:
> >root@linuz11 root]# rm -rf /containers/1
>
> Just use "rmdir /containers/1" here.
Hmm. Ok, that works... Odd, I thought rm -rf used to work in the past,
but i'm likely wrong.
thanks,
-serge
> >Ah, I see the second time I typed 'ls /containers/1/tasks' instead of
> >cat. When I then used cat, the file was empty, and I got an oops just
> >like Pavel reported. I bet if I solve the problem he reported, then I
> >solve my problem :)
> >
>
> As far as I could see, Pavel's problem wasn't actually an Oops, it was
> a WARN_ON() when allocating a zero length chunk of memory. There's
> ongoing discussion as to whether this counts as a problem with the
> allocators or the kmalloc() code, since it used to be OK to allocate a
> zero-length chunk.
>
> Paul
|
|
|
|
|
|
|
|
| Re: [PATCH 00/10] Containers(V10): Generic Process Containers [message #13863 is a reply to message #13763] |
Wed, 06 June 2007 22:39 |
serue
Messages: 750
Registered: February 2006
|
Senior Member |
|
|
Quoting Paul Jackson (pj@sgi.com):
> > Would it then make sense to just
> > default to (parent_set - sibling_exclusive_set) for a new sibling's
> > value?
>
> Which could well be empty, which in turn puts one back in the position
> of dealing with a newborn cpuset that is empty (of cpus or of memory),
> or else it introduces a new and odd constraint on when cpusets can be
> created (only when there are non-exclusive cpus and mems available.)
>
> > An option is fine with me, but without such an option at all, cpusets
> > could not be applied to namespaces...
>
> I wasn't paying close enough attention to understand why you couldn't
> do it in two steps - make the container, and then populate it with
> resources.
Sorry, please clarify - are you saying that now you do understand, or
that I should explain?
> But if indeed that's not possible, then I guess we need some sort of
> option specifying whether to create kids empty, or inheriting.
Paul (uh, Menage :) should I do a patch for this or have you got it
already?
thanks,
-serge
|
|
|
|
|
|
| Re: [PATCH 00/10] Containers(V10): Generic Process Containers [message #13865 is a reply to message #13864] |
Thu, 07 June 2007 00:05 |
serue
Messages: 750
Registered: February 2006
|
Senior Member |
|
|
Quoting Paul Jackson (pj@sgi.com):
> > > I wasn't paying close enough attention to understand why you couldn't
> > > do it in two steps - make the container, and then populate it with
> > > resources.
> >
> > Sorry, please clarify - are you saying that now you do understand, or
> > that I should explain?
>
> Could you explain -- I still don't understand why you need this option.
> I still don't understand why you can't do it in two steps - make the
> container, then add cpu/mem separately.
Sure - the key is that the ns subsystem uses container_clone() to
automatically create a new container (on sys_unshare() or clone(2)
with certain flags) and move the current task into it. Let's say
we have done
mount -t container -o ns,cpuset nsproxy /containers
and we, as task 875, happen to be in the topmost container:
/containers/
Now we fork task 999 which does an unshare(CLONE_NEWNS), or we just
clone(CLONE_NEWNS). This will create
/containers/node_999
and move task 999 into that container. Except that when it tries
attach_task() it is refused by cpuset. So the container_clone() fails,
and in turn the sys_unshare() or clone() fails. A login making use
of the pam_namespace.so library would fail this way with the
ns and cpuset subsystems composed.
We could special case this by having
kernel/container.c:container_clone() check whether one of the subsystems
is cpusets and, if so, setting the defaults for mems and cpus, but
that is kind of ugly. I suppose as a cleaner alternative we could
add a container_subsys->inherit_defaults() handler, to be called at
container_clone(), and for cpusets this would set cpus and mems to
the parent values - sibling exclusive values. If that comes to nothing,
then the attach_task() is still refused, and the unshare() or clone()
fails, but this time with good reason.
thanks,
-serge
|
|
|
|
| Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers [message #13866 is a reply to message #13865] |
Thu, 07 June 2007 00:46 |
Paul Jackson
Messages: 157
Registered: February 2006
|
Senior Member |
|
|
> I suppose as a cleaner alternative we could
> add a container_subsys->inherit_defaults() handler, to be called at
> container_clone(), and for cpusets this would set cpus and mems to
> the parent values - sibling exclusive values. If that comes to nothing,
> then the attach_task() is still refused, and the unshare() or clone()
> fails, but this time with good reason.
Unfortunately, I haven't spent the time I should thinking about
container cloning, namespaces and such.
I don't know, for the workloads that matter to me, when, how or
if this container cloning will be used.
I'm tempted to suggest the following.
First, I am assuming that the classic method of creating cpuset
children will still work, such as the following (which can fail
for certain combinations of exclusive cpus or mems):
cd /dev/cpuset/foobar
mkdir foochild
cp cpus foochild
cp mems foochild
echo $$ > foochild/tasks
Second, given that, how about you fail the unshare() or clone()
anytime that the instance to be cloned has any sibling cpusets
with any exclusive flags set.
The exclusive property is not really on friendly terms with cloning.
Now if the above classic code must be encoded using cloning under
the covers, then we've got problems, probably more problems than
just this.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
|
|
|
|
| Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers [message #13912 is a reply to message #13866] |
Thu, 07 June 2007 18:01 |
serue
Messages: 750
Registered: February 2006
|
Senior Member |
|
|
Quoting Paul Jackson (pj@sgi.com):
> > I suppose as a cleaner alternative we could
> > add a container_subsys->inherit_defaults() handler, to be called at
> > container_clone(), and for cpusets this would set cpus and mems to
> > the parent values - sibling exclusive values. If that comes to nothing,
> > then the attach_task() is still refused, and the unshare() or clone()
> > fails, but this time with good reason.
>
> Unfortunately, I haven't spent the time I should thinking about
> container cloning, namespaces and such.
>
> I don't know, for the workloads that matter to me, when, how or
> if this container cloning will be used.
>
> I'm tempted to suggest the following.
>
> First, I am assuming that the classic method of creating cpuset
> children will still work, such as the following (which can fail
> for certain combinations of exclusive cpus or mems):
> cd /dev/cpuset/foobar
> mkdir foochild
> cp cpus foochild
> cp mems foochild
> echo $$ > foochild/tasks
>
> Second, given that, how about you fail the unshare() or clone()
> anytime that the instance to be cloned has any sibling cpusets
> with any exclusive flags set.
The below patch (on top of my previous patch) does basically that. But
I wasn't able to test it, bc i wasn't able to set cpus_exclusive...
For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0
also have to have it exclusively?
If so, then clearly this approach won't work, since if any container has
exclusive cpus, then every container will have siblings with exclusive
cpus, and unshare still isn't possible on the system.
> The exclusive property is not really on friendly terms with cloning.
>
> Now if the above classic code must be encoded using cloning under
> the covers, then we've got problems, probably more problems than
> just this.
>
> --
> I won't rest till it's the best ...
> Programmer, Linux Scalability
> Paul Jackson <pj@sgi.com> 1.925.600.0401
thanks,
-serge
>From 821de58b6ba446e50225606e907baac00130586c Mon Sep 17 00:00:00 2001
From: Serge E. Hallyn <serue@us.ibm.com>
Date: Thu, 7 Jun 2007 13:53:43 -0400
Subject: [PATCH 1/1] containers: implement subsys->auto_setup
container_clone() in one step creates a new container and moves
the current task into it. Since cpusets do not automatically
fill in the allowed cpus and mems, and do not allow a task to
be attached without these filled in, composing the ns subsystem,
which uses container_clone(), and the cpuset subsystem, results
in sys_unshare() (and clone(CLONE_NEWNS)) always being denied.
To allow the two subsystems to be meaningfully composed, implement
subsystem->auto_setup, called from container_clone() after
creating the new container.
Only the cpuset_auto_setup() is currently implemented. If any
sibling containers have exclusive cpus or mems, then the cpus
and mems are not filled in for the new container, meaning that
unshare/clone(CLONE_NEWNS) will be denied. However so long as
no siblings have exclusive cpus or mems, the new container's
cpus and mems are inherited from the parent container.
Signed-off-by: Serge E. Hallyn <serue@us.ibm.com>
---
Documentation/containers.txt | 7 +++++++
include/linux/container.h | 1 +
kernel/container.c | 7 +++++++
kernel/cpuset.c | 21 +++++++++++++++++++++
4 files changed, 36 insertions(+), 0 deletions(-)
diff --git a/Documentation/containers.txt b/Documentation/containers.txt
index ae159b9..28c9e10 100644
--- a/Documentation/containers.txt
+++ b/Documentation/containers.txt
@@ -514,6 +514,13 @@ include/linux/container.h for details). Note that although this
method can return an error code, the error code is currently not
always handled well.
+void auto_setup(struct container_subsys *ss, struct container *cont)
+
+Called at container_clone() to do any paramater initialization
+which might be required before a task could attach. For example
+in cpusets, no task may attach before 'cpus' and 'mems' are
+set up.
+
void bind(struct container_subsys *ss, struct container *root)
LL=callback_mutex
diff --git a/include/linux/container.h b/include/linux/container.h
index 37c0bdf..d809b41 100644
--- a/include/linux/container.h
+++ b/include/linux/container.h
@@ -213,6 +213,7 @@ struct container_subsys {
void (*exit)(struct container_subsys *ss, struct task_struct *task);
int (*populate)(struct container_subsys *ss,
struct container *cont);
+ void (*auto_setup)(struct container_subsys *ss, struct container *cont);
void (*bind)(struct container_subsys *ss, struct container *root);
int subsys_id;
int active;
diff --git a/kernel/container.c b/kernel/container.c
index 988cd8b..e0793f4 100644
--- a/kernel/container.c
+++ b/kernel/container.c
@@ -2316,6 +2316,7 @@ int container_clone(struct task_struct *tsk, struct container_subsys *subsys)
struct inode *inode;
struct css_group *cg;
struct containerfs_root *root;
+ struct container_subsys *ss;
/* We shouldn't be called by an unregistered subsystem */
BUG_ON(!subsys->active);
@@ -2397,6 +2398,12 @@ int container_clone(struct task_struct *tsk, struct container_subsys *subsys)
goto again;
}
+ /* do any required auto-setup */
+ for_each_subsys(root, ss) {
+ if (ss->auto_setup)
+ ss->auto_setup(ss, child);
+ }
+
/* All seems fine. Finish by moving the task into the new container */
ret = attach_task(child, tsk);
mutex_unlock(&container_mutex);
diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index 0f9ce7d..ff01aaa 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -1189,6 +1189,26 @@ int cpuset_populate(struct container_subsys *ss, struct container *cont)
return 0;
}
+void cpuset_auto_setup(struct container_subsys *ss,
+ struct container *container)
+{
+ struct container *parent, *child;
+ struct cpuset *cs, *parent_cs;
+
+ parent = container->parent;
+ list_for_each_entry(child, &parent->children, sibling) {
+ cs = container_cs(child);
+ if (is_mem_exclusive(cs) || is_cpu_exclusive(cs))
+ return;
+ }
+ cs = container_cs(container);
+ parent_cs = container_cs(parent);
+
+ cs->mems_allowed = parent_cs->mems_allowed;
+ cs->cpus_allowed = parent_cs->cpus_allowed;
+ return;
+}
+
/*
* cpuset_create - create a cpuset
* parent: cpuset that will be parent of the new cpuset.
@@ -1249,6 +1269,7 @@ struct container_subsys cpuset_subsys = {
.can_attach = cpuset_can_attach,
.attach = cpuset_attach,
.populate = cpuset_populate,
+ .auto_setup = cpuset_auto_setup,
.subsys_id = cpuset_subsys_id,
.early_init = 1,
};
--
1.5.1.1.GIT
|
|
|
|
| Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers [message #13914 is a reply to message #13912] |
Thu, 07 June 2007 19:21 |
Paul Jackson
Messages: 157
Registered: February 2006
|
Senior Member |
|
|
> For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0
> also have to have it exclusively?
Yes.
> If so, then clearly this approach won't work, since if any container has
> exclusive cpus, then every container will have siblings with exclusive
> cpus, and unshare still isn't possible on the system.
Well, if I'm following you, not exactly.
If we have some exclusive flags set, then every top level container
will have exclusive siblings, but further down the hierarchy, some
subtree might be entirely free of any exclusive settings. Then nodes
below the top of that subtree would not have exclusive set, and would
not have any exclusive siblings.
But, overall, yeah, exclusive is no friend of container cloning.
I just wish I had been thinking harder about how container cloning
will impact my life, and the lives of the customers in my cpuset
intensive corner of the world.
There are certainly a whole bunch of people who will never have any
need for exclusive cpusets.
Perhaps (speculating wildly from great ignorance) there are a whole
bunch of people who will never have need for container cloning.
And perhaps, hoping to get lucky here, the set of people who need both
at the same time on the same system is sufficiently close to empty
that we can just tell them tough toenails - you cannot do both at once.
How wide spread will be the use of container cloning, if it proceeds
as envisioned?
The set of people using exclusive cpusets is roughly some subset of
those running multiple, cpuset isolated, non-cooperating jobs on big
iron, usually with the aid of a batch scheduler. Well, that's what
I am aware of anyway. If there are any other friends of exclusive
cpusets lurking here, you might want to speak up, before I sell your
interests down the river.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
|
|
|
|
| Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers [message #13915 is a reply to message #13914] |
Thu, 07 June 2007 20:17 |
serue
Messages: 750
Registered: February 2006
|
Senior Member |
|
|
Quoting Paul Jackson (pj@sgi.com):
> > For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0
> > also have to have it exclusively?
>
> Yes.
>
> > If so, then clearly this approach won't work, since if any container has
> > exclusive cpus, then every container will have siblings with exclusive
> > cpus, and unshare still isn't possible on the system.
>
> Well, if I'm following you, not exactly.
>
> If we have some exclusive flags set, then every top level container
> will have exclusive siblings, but further down the hierarchy, some
> subtree might be entirely free of any exclusive settings. Then nodes
> below the top of that subtree would not have exclusive set, and would
> not have any exclusive siblings.
>
> But, overall, yeah, exclusive is no friend of container cloning.
>
> I just wish I had been thinking harder about how container cloning
> will impact my life, and the lives of the customers in my cpuset
> intensive corner of the world.
>
> There are certainly a whole bunch of people who will never have any
> need for exclusive cpusets.
>
> Perhaps (speculating wildly from great ignorance) there are a whole
> bunch of people who will never have need for container cloning.
>
> And perhaps, hoping to get lucky here, the set of people who need both
> at the same time on the same system is sufficiently close to empty
> that we can just tell them tough toenails - you cannot do both at once.
>
> How wide spread will be the use of container cloning, if it proceeds
> as envisioned?
It's not just container cloning, but all namespace unsharing. So uses
include (1) providing 'polyinstantiated directory' functionality, i.e.
private per-user /tmp's or per-security-level /tmp and /home's. (2) any
virtual server usage (3) hpc checkpoint/restart users.
> The set of people using exclusive cpusets is roughly some subset of
> those running multiple, cpuset isolated, non-cooperating jobs on big
> iron, usually with the aid of a batch scheduler.
Unfortunately I would imagine these users to be very intereseted in
providing checkpoint/restart/migrate functionality.
> Well, that's what
> I am aware of anyway. If there are any other friends of exclusive
> cpusets lurking here, you might want to speak up, before I sell your
> interests down the river.
>
> --
> I won't rest till it's the best ...
> Programmer, Linux Scalability
> Paul Jackson <pj@sgi.com> 1.925.600.0401
Can you explain to me, though, why it should be that if /cpusets/set0
has access to cpus 0-8, and /cpusets/set0/set1 has exclusive access to
cpus 0-2, and /cpusets/set0/set2 has exclusive access to cpus 3-4,
why i a process in /cpusets/set0 creates /cpusets/set0/set3 through
container_clone, it would be unsafe to have it automatically get cpus 5-8?
Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4
later, those cpus can just be taken away from set3?
thanks,
-serge
|
|
|
|
| Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers [message #13917 is a reply to message #13915] |
Thu, 07 June 2007 22:01 |
Paul Jackson
Messages: 157
Registered: February 2006
|
Senior Member |
|
|
> > The set of people using exclusive cpusets is roughly some subset of
> > those running multiple, cpuset isolated, non-cooperating jobs on big
> > iron, usually with the aid of a batch scheduler.
>
> Unfortunately I would imagine these users to be very intereseted in
> providing checkpoint/restart/migrate functionality.
Yup - such customers are very interested in checkpoint, restart, and
migrate functionality.
> Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4
> later, those cpus can just be taken away from set3?
Yeah - that works, so far as I know (which isn't all that far ..')
But both:
1) that (using whatever cpus are still available) and
2) my other idea, of not allowing any cloning of cpusets with
exclusive siblings at all,
looked a little ugly to me.
For example, such customers as above would not appreciate having their
checkpoint/restart/migrate fail in any case where there weren't spare
non-exclusive cpus, which for users of the exclusive flag, is often the
more common case.
My rule of thumb when doing ugly stuff is to constrain it as best
I can -- minimize what it allows. This led me to prefer (2) above
over (1) above.
Perhaps there's a better way to think of this ... When we clone
like this for checkpoint/restart/migrate functionality, perhaps
we are not really starting up a new, separate, competing job that
should have its resources isolated and separated from the original.
Perhaps instead we are firing up a convenient alter-ego of the
original job, which will be co-operatively using the same resources
by default. If that's the normal case, then it seems wrong to
force the clone onto disjoint CPUs, or fail for lack thereof.
So perhaps we should refine the meaning of 'exclusive', from:
- no overlapping siblings
to:
- no overlapping siblings other than clones of ones self.
Then default to cloning right on the same CPU resources as the
original, possibly with both original and clone marked exclusive.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
|
|
|
|
| Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers [message #13960 is a reply to message #13917] |
Fri, 08 June 2007 14:32 |
serge
Messages: 72
Registered: January 2007
|
Member |
|
|
Quoting Paul Jackson (pj@sgi.com):
> > > The set of people using exclusive cpusets is roughly some subset of
> > > those running multiple, cpuset isolated, non-cooperating jobs on big
> > > iron, usually with the aid of a batch scheduler.
> >
> > Unfortunately I would imagine these users to be very intereseted in
> > providing checkpoint/restart/migrate functionality.
>
> Yup - such customers are very interested in checkpoint, restart, and
> migrate functionality.
>
> > Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4
> > later, those cpus can just be taken away from set3?
>
> Yeah - that works, so far as I know (which isn't all that far ..')
>
> But both:
> 1) that (using whatever cpus are still available) and
> 2) my other idea, of not allowing any cloning of cpusets with
> exclusive siblings at all,
>
> looked a little ugly to me.
>
> For example, such customers as above would not appreciate having their
> checkpoint/restart/migrate fail in any case where there weren't spare
> non-exclusive cpus, which for users of the exclusive flag, is often the
> more common case.
>
> My rule of thumb when doing ugly stuff is to constrain it as best
> I can -- minimize what it allows. This led me to prefer (2) above
> over (1) above.
>
> Perhaps there's a better way to think of this ... When we clone
> like this for checkpoint/restart/migrate functionality, perhaps
> we are not really starting up a new, separate, competing job that
> should have its resources isolated and separated from the original.
Depends on whether the cpus are allocated to a customer or to a job.
For the most part I would expect any job to be restart either on a
different machine, or at a different time, but of course it doesn't have
to be that way.
> Perhaps instead we are firing up a convenient alter-ego of the
> original job, which will be co-operatively using the same resources
> by default. If that's the normal case, then it seems wrong to
> force the clone onto disjoint CPUs, or fail for lack thereof.
>
> So perhaps we should refine the meaning of 'exclusive', from:
> - no overlapping siblings
> to:
> - no overlapping siblings other than clones of ones self.
I'm not sure that clones of self will happen often enough to make a
special case for them :)
Anyway the patch I sent is simple enough, and if users end up demanding
the ability to better deal with exclusive cpusets, the patch will be
simple enough to extend by changing cpuset_auto_setup(), so let's
stick with that patch since it's your preference (IIUC).
> Then default to cloning right on the same CPU resources as the
> original, possibly with both original and clone marked exclusive.
Thanks,
-serge
|
|
|
|
|
|
|
|
| Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers [message #13983 is a reply to message #13960] |
Fri, 08 June 2007 17:37 |
Paul Jackson
Messages: 157
Registered: February 2006
|
Senior Member |
|
|
> Anyway the patch I sent is simple enough, and if users end up demanding
> the ability to better deal with exclusive cpusets, the patch will be
> simple enough to extend by changing cpuset_auto_setup(), so let's
> stick with that patch since it's your preference (IIUC).
Yeah - probably so.
When someone gets serious about things like checkpoint, restart, and
migrate functionality, based on this container cloning, working with
cpusets, they will probably have to revisit this interaction with
exclusive cpusets.
Perhaps a comment could be put in the code, saying something like the
above, so whomever does this will realize they are traveling in
unchartered territory.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@sgi.com> 1.925.600.0401
|
|
|
|
| Re: [PATCH 00/10] Containers(V10): Generic Process Containers [message #14471 is a reply to message #13570] |
Thu, 28 June 2007 21:27 |
Paul Menage
Messages: 642
Registered: September 2006
|
Senior Member |
|
|
On 5/30/07, William Lee Irwin III <wli@holomorphy.com> wrote:
> On Wed, May 30, 2007 at 12:14:55AM -0700, Andrew Morton wrote:
> > So how do we do this?
> > Is there any sneaky way in which we can modify the kernel so that this new
> > code gets exercised more? Obviously, tossing init into some default
> > system-wide container would be a start. But I wonder if we can be
> > sneakier - for example, create a new container on each setuid(), toss the
> > task into that. Or something along those lines?
>
> How about a container for each thread group, pgrp, session, and user?
>
I've been thinking about this, and figured that it could be quite
useful to be able to mount a container tree that groups tasks by
userid or thread group - for doing per-user resource controls, for
example, without having to write a controller that specifically
handles the per-user case.
One option would be to add a mount option, something like
mount -t container -ogroupkey=<X>
where X could be one of: uid, gid, pgrp, sid, tgid
And put hooks in the various places where these ids could change, in
order to move tasks between contaners as appropriate. But after some
thought it seems to me that this is putting complexity in the kernel
that probably doesn't belong there, and additionally is probably not
sufficently flexible for some real-life situations. (E.g. the user
wants all users in the "student" group to be lumped into the same
container, but each user in the "professor" group gets their own
container).
So maybe this would be better handled in userspace? Have a daemon
listing on a process connector socket, and move processes between
containers based on notifications from the connector and user-defined
rules.
We'd probably also want to add some new connector events, such as
PROC_EVENT_PGRP, and PROC_EVENT_SID
A simple daemon that handles the case where we're classifying based on
a single key with no complex rules shouldn't be too hard to write.
It also sounds rather like the classification engine that
ResourceGroups had originally in the kernel and moved to userspace, so
I'll take a look at that and see if it's adaptable for this.
Paul
|
|
|
|
| Re: [ckrm-tech] [PATCH 00/10] Containers(V10): Generic Process Containers [message #14472 is a reply to message #14471] |
Thu, 28 June 2007 22:13 |
Srivatsa Vaddagiri
Messages: 241
Registered: August 2006
|
Senior Member |
|
|
On Thu, Jun 28, 2007 at 05:27:25PM -0400, Paul Menage wrote:
> So maybe this would be better handled in userspace? Have a daemon
> listing on a process connector socket, and move processes between
> containers based on notifications from the connector and user-defined
> rules.
>
> We'd probably also want to add some new connector events, such as
> PROC_EVENT_PGRP, and PROC_EVENT_SID
Yep, this is what I did to test fair-user scheduling on top of my
patches.
Dhaval has a working program, which listens for UID change events and
moves the task to approp. container. I will review that and have it
posted soon.
--
Regards,
vatsa
|
|
|
|
Goto Forum:
Current Time: Tue Jul 16 22:55:11 GMT 2024
Total time taken to generate the page: 0.02984 seconds
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
