OpenVZ Forum - RDF feed
type: channel
items: [PATCH 0/7] Generic Process Containers (+ ResGroups/BeanCounters)
items: [PATCH 1/7] Generic container system abstracted from cpusets code
items: [PATCH 2/7] Cpusets hooked into containers
items: [PATCH 3/7] Add generic multi-subsystem API to containers
items: [PATCH 5/7] Resource Groups over generic containers
items: [PATCH 6/7] Split Cpusets into Cpusets and Memsets
items: [PATCH 7/7] BeanCounters over generic process containers
items: Re: [PATCH 0/7] Generic Process Containers (+ ResGroups/BeanCounters)
items: [Patch 1/3] Miscellaneous container fixes
items: [Patch 2/3] cpu controller (v3) - based on RTLIMIT_RT_CPU patch

[PATCH 0/7] Generic Process Containers (+ ResGroups/BeanCounters)
type: item
description: This is an update to my multi-hierarchy generic containers patch (against 2.6.19-rc6). Changes include: - an example patch implementing the BeanCounters core and numfiles counters over generic containers. The addition of the BeanCounters code unifies the three main process grouping abstractions (Cpusets, ResGroups and BeanCounters). - a patch splitting Cpusets into two independently groupable subsystems, Cpusets and Memsets. - support for a subsystem to keep a container alive via refcounts (e.g. the BeanCounters numfiles counter has a reference to the beancounter object from each file charged to that beancounter, so needs to be able to keep the beancounter alive until the file is destroyed) ------------------------------------- There have recently been various proposals floating around for resource management/accounting subsystems in the kernel, including Res Groups, User BeanCounters and others. These all need the basic abstraction of being able to group together multiple processes in an aggregate, in order to track/limit the resources permitted to those processes, and all implement this grouping in different ways. Already existing in the kernel is the cpuset subsystem; this has a process grouping mechanism that is mature, tested, and well documented (particularly with regards to synchronization rules). This patchset extracts the process grouping code from cpusets into a generic container system, and makes the cpusets code a client of the container system. It also provides several example clients of the container system, including ResGroups and BeanCounters The change is implemented in five stages plus two additional example patches: 1) extract the process grouping code from cpusets into a standalone system 2) remove the process grouping code from cpusets and hook into the container system 3) convert the container system to present a generic multi-hierarchy API, and make cpusets a client of that API 4) add a simple CPU accounting container subsystem as an example 5…
creator: Paul Menage
date: 2006-11-23T12:08:48-00:00

[PATCH 1/7] Generic container system abstracted from cpusets code
type: item
description: This patch creates a generic process container system based on (and parallel top) the cpusets code. At a coarse level it was created by copying kernel/cpuset.c, doing s/cpuset/container/g, and stripping out any code that was cpuset-specific rather than applicable to any process container subsystem. Signed-off-by: Paul Menage <menage@google.com> --- Documentation/containers.txt | 229 +++++++ fs/proc/base.c | 11 include/linux/container.h | 96 +++ include/linux/sched.h | 5 init/Kconfig | 9 init/main.c | 3 kernel/Makefile | 1 kernel/container.c | 1343 +++++++++++++++++++++++++++++++++++++++++++ kernel/exit.c | 2 kernel/fork.c | 3 10 files changed, 1699 insertions(+), 3 deletions(-) Index: container-2.6.19-rc5/fs/proc/base.c ============================================================ ======= --- container-2.6.19-rc5.orig/fs/proc/base.c +++ container-2.6.19-rc5/fs/proc/base.c @@ -68,6 +68,7 @@ #include <linux/security.h> #include <linux/ptrace.h> #include <linux/seccomp.h> +#include <linux/container.h> #include <linux/cpuset.h> #include <linux/audit.h> #include <linux/poll.h> @@ -1782,7 +1783,10 @@ static struct pid_entry tgid_base_stuff[ #ifdef CONFIG_SCHEDSTATS INF("schedstat", S_IRUGO, pid_schedstat), #endif -#ifdef CONFIG_CPUSETS +#ifdef CONFIG_CONTAINERS + REG("container", S_IRUGO, container), +#endif +#ifdef CONFIG_PROC_PID_CPUSET REG("cpuset", S_IRUGO, cpuset), #endif INF("oom_score", S_IRUGO, oom_score), @@ -2056,7 +2060,10 @@ static struct pid_entry tid_base_stuff[] #ifdef CONFIG_SCHEDSTATS INF("schedstat", S_IRUGO, pid_schedstat), #endif -#ifdef CONFIG_CPUSETS +#ifdef CONFIG_CONTAINERS + REG("container", S_IRUGO, container), +#endif +#ifdef CONFIG_PROC_PID_CPUSET REG("cpuset", S_IRUGO, cpuset), #endif INF("oom_score", S_IRUGO, oom_score), Index: container-2.6.19-rc5/include/linux/container.h ============================================================ ======= --- /dev/null +++ container-2.6.19-rc5/include/linux/container.h @@ -0,0 +1,96 @@ +#ifndef _LINUX_CONTA…
creator: Paul Menage
date: 2006-11-23T12:08:49-00:00

[PATCH 2/7] Cpusets hooked into containers
type: item
description: This patch removes the process grouping code from the cpusets code, instead hooking it into the generic container system. This temporarily adds cpuset-specific code in kernel/container.c, which is removed by the next patch in the series. Signed-off-by: Paul Menage <menage@google.com> --- Documentation/cpusets.txt | 81 +- fs/super.c | 5 include/linux/container.h | 7 include/linux/cpuset.h | 25 include/linux/fs.h | 2 include/linux/mempolicy.h | 2 include/linux/sched.h | 4 init/Kconfig | 23 kernel/container.c | 107 +++ kernel/cpuset.c | 1273 +++++----------------------------------------- kernel/exit.c | 2 kernel/fork.c | 7 mm/oom_kill.c | 6 13 files changed, 330 insertions(+), 1214 deletions(-) Index: container-2.6.19-rc5/include/linux/container.h ============================================================ ======= --- container-2.6.19-rc5.orig/include/linux/container.h +++ container-2.6.19-rc5/include/linux/container.h @@ -47,6 +47,10 @@ struct container { struct container *parent; /* my parent */ struct dentry *dentry; /* container fs entry */ + +#ifdef CONFIG_CPUSETS + struct cpuset *cpuset; +#endif }; /* struct cftype: @@ -79,6 +83,9 @@ struct cftype { int container_add_file(struct container *cont, const struct cftype *cft); int container_is_removed(const struct container *cont); +void container_set_release_agent_path(const char *path); + +int container_path(const struct container *cont, char *buf, int buflen); #else /* !CONFIG_CONTAINERS */ Index: container-2.6.19-rc5/include/linux/cpuset.h ============================================================ ======= --- container-2.6.19-rc5.orig/include/linux/cpuset.h +++ container-2.6.19-rc5/include/linux/cpuset.h @@ -11,16 +11,15 @@ #include <linux/sched.h> #include <linux/cpumask.h> #include <linux/nodemask.h> +#include <linux/container.h> #ifdef CONFIG_CPUSETS -extern int number_of_cpusets; /* How many cpusets are defined in system? */ +extern int number_of_cpusets; /* How many cpusets are defined in system? */ extern…
creator: Paul Menage
date: 2006-11-23T12:08:50-00:00

[PATCH 3/7] Add generic multi-subsystem API to containers
type: item
description: This patch removes all cpuset-specific knowlege from the container system, replacing it with a generic API that can be used by multiple subsystems. Cpusets is adapted to be a container subsystem. Signed-off-by: Paul Menage <menage@google.com> --- Documentation/containers.txt | 233 ++++++++++++- include/linux/container.h | 132 +++++++ include/linux/cpuset.h | 16 include/linux/mempolicy.h | 12 include/linux/sched.h | 4 kernel/container.c | 716 +++++++++++++++++++++++++++++++++++-------- kernel/cpuset.c | 165 ++++++--- mm/mempolicy.c | 2 8 files changed, 1048 insertions(+), 232 deletions(-) Index: container-2.6.19-rc6/include/linux/container.h ============================================================ ======= --- container-2.6.19-rc6.orig/include/linux/container.h +++ container-2.6.19-rc6/include/linux/container.h @@ -14,8 +14,6 @@ #ifdef CONFIG_CONTAINERS -extern int number_of_containers; /* How many containers are defined in system? */ - extern int container_init_early(void); extern int container_init(void); extern void container_init_smp(void); @@ -30,6 +28,68 @@ extern void container_unlock(void); extern void container_manage_lock(void); extern void container_manage_unlock(void); +struct containerfs_root; + +/* Per-subsystem/per-container state maintained by the system. */ +struct container_subsys_state { + /* The container that this subsystem is attached to. Useful + * for subsystems that want to know about the container + * hierarchy structure */ + struct container *container; + + /* State maintained by the container system to allow + * subsystems to be "busy". Should be accessed via css_get() + * and css_put() */ + spinlock_t refcnt_lock; + atomic_t refcnt; +}; + +/* + * Call css_get() to hold a reference on the container; following a + * return of 0, this container subsystem state object is guaranteed + * not to be destroyed until css_put() is called on it. A non-zero + * return code indicates that a reference could not be taken. + * + */ + +static inline int…
creator: Paul Menage
date: 2006-11-23T12:08:51-00:00

[PATCH 5/7] Resource Groups over generic containers
type: item
description: This patch provides the RG core and numtasks controller as container subsystems, intended as an example of how to implement a more complex resource control system over generic process containers. The changes to the core involve primarily removing the group management, task membership and configfs support and adding interface layers to talk to the generic container layer instead. Each resource controller becomes an independent container subsystem; the RG core is essentially a library that the resource controllers can use to provide the RG API to userspace. Rather than a single shares and stats file in each group, there's a <controller>_shares and a <controller>_stats file, each linked to the appropriate resource controller. include/linux/moduleparam.h | 12 - include/linux/numtasks.h | 28 ++ include/linux/res_group.h | 87 ++++++++ include/linux/res_group_rc.h | 97 ++++++++ init/Kconfig | 22 ++ kernel/Makefile | 1 kernel/fork.c | 7 kernel/res_group/Makefile | 2 kernel/res_group/local.h | 38 +++ kernel/res_group/numtasks.c | 467 +++++++++++++++++++++++++++++++++++++++++++ kernel/res_group/res_group.c | 160 ++++++++++++++ kernel/res_group/rgcs.c | 302 +++++++++++++++++++++++++++ kernel/res_group/shares.c | 228 ++++++++++++++++++++ 13 files changed, 1447 insertions(+), 4 deletions(-) Index: container-2.6.19-rc6/include/linux/moduleparam.h ============================================================ ======= --- container-2.6.19-rc6.orig/include/linux/moduleparam.h +++ container-2.6.19-rc6/include/linux/moduleparam.h @@ -75,11 +75,17 @@ struct kparam_array /* Helper functions: type is byte, short, ushort, int, uint, long, ulong, charp, bool or invbool, or XXX if you define param_get_XXX, param_set_XXX and param_check_XXX. */ -#define module_param_named(name, value, type, perm) \ - param_check_##type(name, &(value)); \ - module_param_call(name, param_set_##type, param_get_##type, &value, perm); \ +#define module_param_named_call(name, value, type, set, perm) \ + param_check_…
creator: Paul Menage
date: 2006-11-23T12:08:53-00:00

[PATCH 6/7] Split Cpusets into Cpusets and Memsets
type: item
description: This patch splits the Cpusets container subsystem into two independent subsystems; currently CPUsets are the cpu and memory node control functionality in Cpusets are pretty much disjoint and unrelated; now that the common process container abstraction has been moved out, there's no particular reason to keep them together in the same subsystem. Signed-off-by: Paul Menage <menage@google.com> --- fs/proc/array.c | 2 include/linux/cpuset.h | 75 -- include/linux/mempolicy.h | 2 include/linux/memset.h | 125 ++++ include/linux/sched.h | 10 init/Kconfig | 14 init/main.c | 3 kernel/Makefile | 1 kernel/cpuset.c | 994 +-------------------------------- kernel/memset.c | 1352 ++++++++++++++++++++++++++++++++++++++++++++++ mm/filemap.c | 6 mm/hugetlb.c | 4 mm/memory_hotplug.c | 4 mm/mempolicy.c | 36 - mm/migrate.c | 4 mm/oom_kill.c | 14 mm/page_alloc.c | 26 mm/slab.c | 12 mm/vmscan.c | 10 19 files changed, 1593 insertions(+), 1101 deletions(-) Index: container-2.6.19-rc6/include/linux/cpuset.h =================================================================== --- container-2.6.19-rc6.orig/include/linux/cpuset.h +++ container-2.6.19-rc6/include/linux/cpuset.h @@ -10,58 +10,22 @@ #include <linux/sched.h> #include <linux/cpumask.h> -#include <linux/nodemask.h> #include <linux/container.h> #ifdef CONFIG_CPUSETS -extern int number_of_cpusets; /* How many cpusets are defined in system? */ - extern int cpuset_init_early(void); extern int cpuset_init(void); extern void cpuset_init_smp(void); extern cpumask_t cpuset_cpus_allowed(struct task_struct *p); -extern nodemask_t cpuset_mems_allowed(struct task_struct *p); -void cpuset_init_current_mems_allowed(void); -void cpuset_update_task_memory_state(void); -#define cpuset_nodes_subset_current_mems_allowed(nodes) \ - nodes_subset((nodes), current->mems_allowed) -int cpuset_zonelist_valid_mems_allowed(struct zonelist *zl); - -extern int __cpuset_zone_allowed(struct zone *z, gfp_t gfp_mask); -static int inline cpuset_zone_allowed(struct zone *…
creator: Paul Menage
date: 2006-11-23T12:08:54-00:00

[PATCH 7/7] BeanCounters over generic process containers
type: item
description: This patch implements the BeanCounter resource control abstraction over generic process containers. It contains the beancounter core code, plus the numfiles resource counter. It doesn't currently contain any of the memory tracking code or the code for switching beancounter context in interrupts. Currently all the beancounters resource counters are lumped into a single hierarchy; ideally it would be possible for each resource counter to be a separate container subsystem, allowing them to be connected to different hierarchies. --- fs/file_table.c | 11 + include/bc/beancounter.h | 192 ++++++++++++++++++++++++ include/bc/misc.h | 27 +++ include/linux/fs.h | 3 init/Kconfig | 4 init/main.c | 3 kernel/Makefile | 1 kernel/bc/Kconfig | 17 ++ kernel/bc/Makefile | 7 kernel/bc/beancounter.c | 371 +++++++++++++++++++++++++++++++++++++++++++++++ kernel/bc/misc.c | 56 +++++++ 11 files changed, 691 insertions(+), 1 deletion(-) Index: container-2.6.19-rc6/init/Kconfig ============================================================ ======= --- container-2.6.19-rc6.orig/init/Kconfig +++ container-2.6.19-rc6/init/Kconfig @@ -601,6 +601,10 @@ config STOP_MACHINE Need stop_machine() primitive. endmenu +menu "Beancounters" +source "kernel/bc/Kconfig" +endmenu + menu "Block layer" source "block/Kconfig" endmenu Index: container-2.6.19-rc6/kernel/Makefile ============================================================ ======= --- container-2.6.19-rc6.orig/kernel/Makefile +++ container-2.6.19-rc6/kernel/Makefile @@ -12,6 +12,7 @@ obj-y = sched.o fork.o exec_domain.o obj-$(CONFIG_STACKTRACE) += stacktrace.o obj-y += time/ +obj-$(CONFIG_BEANCOUNTERS) += bc/ obj-$(CONFIG_DEBUG_MUTEXES) += mutex-debug.o obj-$(CONFIG_LOCKDEP) += lockdep.o ifeq ($(CONFIG_PROC_FS),y) Index: container-2.6.19-rc6/kernel/bc/Kconfig ============================================================ ======= --- /dev/null +++ container-2.6.19-rc6/kernel/bc/Kconfig @@ -0,0 +1,17 @@ +config BEANCOUNTERS + bool "Enable resource account…
creator: Paul Menage
date: 2006-11-23T12:08:55-00:00

Re: [PATCH 0/7] Generic Process Containers (+ ResGroups/BeanCounters)
type: item
description: I got a chance to build and test this patch set, to see if it behaved like I expected cpusets to behave, on an ia64 SN2 Altix system. Two details - otherwise looked good. I continue to like this approach. The two details are (1) /proc/<pid>/cpuset not configured by default if CPUSETS configured, and (2) a locking bug wedging tasks trying to rmdir a cpuset off the notify_on_release hook. 1) I had to enable CONFIG_PROC_PID_CPUSET. I used the following one line change to do this. I am willing to consider, in due time, phasing out such legacy cpuset support. But so long as it is small stuff that is not getting in anyone's way, I think we should take our sweet time about doing so -- as in a year or two after marking it deprecated or some such. No sense deciding that matter now; keep the current cpuset API working throughout any transitition to container based cpusets, then revisit the question of whether to deprecate and eventually remove these kernel API details, later on, after the major reconstruction dust settles. In general, we try to avoid removing kernel API's, especially if they are happily being used and working and not causing anyone grief. ============================ begin ============================ --- 2.6.19-rc5.orig/init/Kconfig 2006-11-29 21:14:48.071114833 -0800 +++ 2.6.19-rc5/init/Kconfig 2006-11-29 22:19:02.015166048 -0800 @@ -268,6 +268,7 @@ config CPUSETS config PROC_PID_CPUSET bool "Include legacy /proc/<pid>/cpuset file" depends on CPUSETS + default y if CPUSETS config CONTAINER_CPUACCT bool "Simple CPU accounting container subsystem" ============================= end ============================= 2) I wedged the kernel on the container_lock, doing a removal of a cpuset using notify_on_release. Right now, that test system has the following two tasks, wedged: ============================ begin ============================ F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 0 S root 4992 34 0 71 -5 - 380 wait 22:51 ? 00:00:00 /bin/sh /sbin/c…
creator: Paul Jackson
date: 2006-11-30T07:32:29-00:00

Re: [PATCH 0/7] Generic Process Containers (+ ResGroups/BeanCounters)
type: item
description: On 11/29/06, Paul Jackson <pj@sgi.com> wrote: > config PROC_PID_CPUSET > bool "Include legacy /proc/<pid>/cpuset file" > depends on CPUSETS > + default y if CPUSETS > Sounds very reasonable. > 2) I wedged the kernel on the container_lock, doing a removal of a cpuset > using notify_on_release. > I guess I've not really tested doing interesting things from the notify_on_release code, just checked that it successfully executed a simple command. I'll look into it. Thanks for the feedback. Paul
creator: Paul Menage
date: 2006-11-30T08:01:06-00:00

Re: [PATCH 0/7] Generic Process Containers (+ ResGroups/BeanCounters)
type: item
description: On 11/29/06, Paul Jackson <pj@sgi.com> wrote: > > 2) I wedged the kernel on the container_lock, doing a removal of a cpuset > using notify_on_release. I couldn't reproduce this, with a /sbin/cpuset_release_agent that does: #!/bin/bash logger cpuset_release_agent $1 rmdir /dev/cpuset/$1 and running the commands: while true; do mkdir -p /dev/cpuset/bar/foo echo 1 > /dev/cpuset/bar/notify_on_release rmdir /dev/cpuset/bar/foo usleep 1000 done Is it actually reproducible for you? If so, could you get a fuller backtrace? Thanks, Paul
creator: Paul Menage
date: 2006-11-30T08:51:26-00:00

[Patch 1/3] Miscellaneous container fixes
type: item
description: This patches fixes various bugs I hit in the recently posted container patches. 1. If a subsystem registers with fork/exit hook during bootup (much before rcu is initialized), then the resulting synchronize_rcu() in container_register_subsys() hangs. Avoid this by not calling synchronize_rcu() if we arent fully booted yet. 2. If cpuset_create fails() for some reason, then the resulting call to cpuset_destroy can trip. Avoid this by initializing container->...->cpuset pointer to NULL in cpuset_create(). 3. container_rmdir->cpuset_destroy->update_flag can deadlock on container_lock(). Avoid this by introducing __update_flag, which doesnt take container_lock(). (I have also hit some lockdep warnings. Will post them after some review, to make sure that they are not introduced by my patches). Signed-off-by : Srivatsa Vaddagiri <vatsa@in.ibm.com> --- linux-2.6.19-rc6-vatsa/kernel/container.c | 4 ++- linux-2.6.19-rc6-vatsa/kernel/cpuset.c | 35 ++++++++++++++++++++++++------ 2 files changed, 31 insertions(+), 8 deletions(-) diff -puN kernel/container.c~container_fixes kernel/container.c --- linux-2.6.19-rc6/kernel/container.c~container_fixes 2006-12-01 16:19:41.000000000 +0530 +++ linux-2.6.19-rc6-vatsa/kernel/container.c 2006-12-01 16:20:20.000000000 +0530 @@ -1344,6 +1344,7 @@ static long container_create(struct cont cont->parent = parent; cont->root = parent->root; cont->hierarchy = parent->hierarchy; + cont->top_container = parent->top_container; for_each_subsys(cont->hierarchy, ss) { err = ss->create(ss, cont); @@ -1580,7 +1581,8 @@ int container_register_subsys(struct con if (!need_forkexit_callback && (new_subsys->fork || new_subsys->exit)) { need_forkexit_callback = 1; - synchronize_rcu(); + if (system_state == SYSTEM_RUNNING) + synchronize_rcu(); } /* If this subsystem requested that it be notified with fork diff -puN kernel/cpuset.c~container_fixes kernel/cpuset.c --- linux-2.6.19-rc6/kernel/cpuset.c~container_fixes 2006-12-01 19:02:35.000000000 +0530 +++ linux-2.…
creator: Srivatsa Vaddagiri
date: 2006-12-01T16:46:32-00:00

[Patch 2/3] cpu controller (v3) - based on RTLIMIT_RT_CPU patch
type: item
description: Nick/Ingo, Here's another approach for a minimal cpu controller. Would greatly appreciate any feedback as before. This version is inspired by Ingo's RTLIMIT_RT_CPU patches found here: http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2 .6.11-rc2/2.6.11-rc2-mm2/broken-out/rlimit_rt_cpu.patch This patch is also about 80% smaller than the patches I had posted earlier: http://lkml.org/lkml/2006/9/28/236 Primary differences between Ingo's RT_LIMIT_CPU patch and this one: - This patch handles starvation of lower priority tasks in a group. - This patch uses tokens for accounting cpu consumption. I didnt get good results with decaying avg approach used in the rtlimit patches. - Task grouping based on cpuset/containers and not on rtlimit. Other features: - Retains one-runqueue-per-cpu, as is prevalent today - scheduling no longer of O(1) complexity, similar to rtlimit patches. This can be avoided if we use separate runqueues for different groups (as was done in http://lkml.org/lkml/2006/9/28/236) - Only limit supported (no guarantee) Unsupported feature: - SMP load balance aware of group limits. This can be handled using smpnice later if required (http://lkml.org/lkml/2006/9/28/244) Signed-off-by : Srivatsa Vaddagiri <vatsa@in.ibm.com> --- linux-2.6.19-rc6-vatsa/include/linux/sched.h | 3 linux-2.6.19-rc6-vatsa/kernel/sched.c | 195 ++++++++++++++++++++++++++- 2 files changed, 195 insertions(+), 3 deletions(-) diff -puN include/linux/sched.h~cpu_ctlr include/linux/sched.h --- linux-2.6.19-rc6/include/linux/sched.h~cpu_ctlr 2006-12-01 20:45:08.000000000 +0530 +++ linux-2.6.19-rc6-vatsa/include/linux/sched.h 2006-12-01 20:45:08.000000000 +0530 @@ -1095,6 +1095,9 @@ static inline void put_task_struct(struc /* Not implemented yet, only for 486*/ #define PF_STARTING 0x00000002 /* being created */ #define PF_EXITING 0x00000004 /* getting shut down */ +#ifdef CONFIG_CPUMETER +#define PF_STARVING 0x00000010 /* Task starving for CPU */ +#endif #define PF_FORKNOEXEC 0x00000040 …
creator: Srivatsa Vaddagiri
date: 2006-12-01T16:51:25-00:00