OpenVZ Forum - RDF feed
type: channel
items: Re: [PATCH] Hookup group-scheduler with task container infrastructure
items: Re: Re: [PATCH] Hookup group-scheduler with task container infrastructure

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Mon, 10 Sep 2007 22:53:34 +0530 Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> wrote: > On Mon, Sep 10, 2007 at 07:05:00PM +0200, Jan Engelhardt wrote: > > On Sep 10 2007 22:40, Srivatsa Vaddagiri wrote: > > >+#ifdef CONFIG_FAIR_GROUP_SCHED > > >+SUBSYS(cpuctlr) > > >+#endif > > > > cpuctl, cpuctrl, cpu_controller? > > *shrug* .. I used "cpuctlr" to mean "CPU Controller". Any other short names > would do. From your list, cpuctl or cpuctrl both qualifies IMO! > > Unless folks have strong objection to it, I prefer "cptctlr", the way it is. > objection ;) "cpuctlr" isn't memorable. Kernel code is write-rarely, read-often. "cpu_controller", please. The extra typing is worth it ;) _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: akpm
date: 2007-09-10T17:22:59-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Sep 10 2007 10:22, Andrew Morton wrote: >> Unless folks have strong objection to it, I prefer "cptctlr", the way it is. > >Kernel code is write-rarely, read-often. I think you mean __read_mostly. :-) Jan -- _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Jan Engelhardt
date: 2007-09-10T17:41:37-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Mon, Sep 10, 2007 at 10:22:59AM -0700, Andrew Morton wrote: > objection ;) "cpuctlr" isn't memorable. Kernel code is write-rarely, > read-often. "cpu_controller", please. The extra typing is worth it ;) Ok! Here's the modified patch (against 2.6.23-rc4-mm1). Signed-off-by : Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> Signed-off-by : Dhaval Giani <dhaval@linux.vnet.ibm.com> --- include/linux/container_subsys.h | 5 init/Kconfig | 9 + kernel/sched.c | 311 +++++++++++++++++++++++++++++++++++++-- kernel/sched_fair.c | 3 4 files changed, 313 insertions(+), 15 deletions(-) Index: current/include/linux/container_subsys.h =================================================================== --- current.orig/include/linux/container_subsys.h +++ current/include/linux/container_subsys.h @@ -36,3 +36,8 @@ SUBSYS(mem_container) #endif /* */ +#ifdef CONFIG_FAIR_GROUP_SCHED +SUBSYS(cpu_controller) +#endif + +/* */ Index: current/init/Kconfig =================================================================== --- current.orig/init/Kconfig +++ current/init/Kconfig @@ -326,6 +326,15 @@ config RESOURCE_COUNTERS infrastructure that works with containers depends on CONTAINERS +config FAIR_GROUP_SCHED + bool "Fair group scheduler" + depends on EXPERIMENTAL && CONTAINERS + help + This option enables you to group tasks and control CPU resource + allocation to such groups. + + Say N if unsure. + config SYSFS_DEPRECATED bool "Create deprecated sysfs files" default y Index: current/kernel/sched.c =================================================================== --- current.orig/kernel/sched.c +++ current/kernel/sched.c @@ -179,6 +179,58 @@ struct load_stat { unsigned long delta_fair, delta_exec, delta_stat; }; +#ifdef CONFIG_FAIR_GROUP_SCHED + +#include <linux/container.h> + +struct cfs_rq; + +/* task group related information */ +struct task_grp { + struct container_subsys_state css; + /* schedulable entities of this group on each cpu */ + struct sched_entity **se; + /* runqueue "owned" …
creator: Srivatsa Vaddagiri
date: 2007-09-10T17:46:49-00:00

Re: Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On 9/10/07, Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> wrote: > > Unless folks have strong objection to it, I prefer "cptctlr", the way it is. > By definition any container (about to be renamed control group) subsystem is some kind of "controller" so that bit seems a bit redundant. Any reason not to just call it "cpu" or "cpu_sched" Paul _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Paul Menage
date: 2007-09-10T18:38:10-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On 10/09/2007, Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> wrote: > On Mon, Sep 10, 2007 at 10:22:59AM -0700, Andrew Morton wrote: > > objection ;) "cpuctlr" isn't memorable. Kernel code is write-rarely, > > read-often. "cpu_controller", please. The extra typing is worth it ;) > > Ok! Here's the modified patch (against 2.6.23-rc4-mm1). > > [ ... ] > > +/* change task's runqueue when it moves between groups */ > +static void sched_move_task(struct container_subsys *ss, struct container *cont, > + struct container *old_cont, struct task_struct *tsk) > +{ > + int on_rq; > + unsigned long flags; > + struct rq *rq; > + > + rq = task_rq_lock(tsk, &flags); > + I guess, update_rq_clock(rq) should be placed here. humm... do you really need deactivate/activate_task() here? 'rq' and p->se.load.weight stay unchanged so enqueue/dequeue_task() would do a job, no? Although, I might be missing (definitely, as this part is not completely clear to me yet) something important at this late hour :) > + on_rq = tsk->se.on_rq; > + if (on_rq) > + deactivate_task(rq, tsk, 0); > + > + if (unlikely(rq->curr == tsk) && tsk->sched_class == &fair_sched_class) > + tsk->sched_class->put_prev_task(rq, tsk); > + > + set_task_cfs_rq(tsk); > + > + if (on_rq) > + activate_task(rq, tsk, 0); > + > + if (unlikely(rq->curr == tsk) && tsk->sched_class == &fair_sched_class) > + tsk->sched_class->set_curr_task(rq); > + > + task_rq_unlock(rq, &flags); > +} > + -- Best regards, Dmitry Adamushko _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dmitry Adamushko
date: 2007-09-10T22:28:51-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On 10/09/2007, Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> wrote: > On Mon, Sep 10, 2007 at 10:22:59AM -0700, Andrew Morton wrote: > > objection ;) "cpuctlr" isn't memorable. Kernel code is write-rarely, > > read-often. "cpu_controller", please. The extra typing is worth it ;) > > Ok! Here's the modified patch (against 2.6.23-rc4-mm1). as everyone seems to be in a quest for a better name... I think, the obvious one would be just 'group_sched'. -- Best regards, Dmitry Adamushko _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dmitry Adamushko
date: 2007-09-10T22:37:43-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Tue, Sep 11, 2007 at 12:28:51AM +0200, Dmitry Adamushko wrote: > > + rq = task_rq_lock(tsk, &flags); > > + > > I guess, update_rq_clock(rq) should be placed here. > > humm... do you really need deactivate/activate_task() here? 'rq' and > p->se.load.weight stay unchanged so enqueue/dequeue_task() would do a > job, no? Good catch. Here's the updated patch. Signed-off-by : Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> Signed-off-by : Dhaval Giani <dhaval@linux.vnet.ibm.com> --- include/linux/container_subsys.h | 6 init/Kconfig | 9 + kernel/sched.c | 312 +++++++++++++++++++++++++++++++++++++-- kernel/sched_fair.c | 3 4 files changed, 315 insertions(+), 15 deletions(-) Index: current/include/linux/container_subsys.h =================================================================== --- current.orig/include/linux/container_subsys.h +++ current/include/linux/container_subsys.h @@ -36,3 +36,9 @@ SUBSYS(mem_container) #endif /* */ + +#ifdef CONFIG_FAIR_GROUP_SCHED +SUBSYS(cpu) +#endif + +/* */ Index: current/init/Kconfig =================================================================== --- current.orig/init/Kconfig +++ current/init/Kconfig @@ -326,6 +326,15 @@ config RESOURCE_COUNTERS infrastructure that works with containers depends on CONTAINERS +config FAIR_GROUP_SCHED + bool "Fair group scheduler" + depends on EXPERIMENTAL && CONTAINERS + help + This option enables you to group tasks and control CPU resource + allocation to such groups. + + Say N if unsure. + config SYSFS_DEPRECATED bool "Create deprecated sysfs files" default y Index: current/kernel/sched.c =================================================================== --- current.orig/kernel/sched.c +++ current/kernel/sched.c @@ -179,6 +179,58 @@ struct load_stat { unsigned long delta_fair, delta_exec, delta_stat; }; +#ifdef CONFIG_FAIR_GROUP_SCHED + +#include <linux/container.h> + +struct cfs_rq; + +/* task group related information */ +struct task_grp { + struct container_subsys_state css; + /* schedulable…
creator: Srivatsa Vaddagiri
date: 2007-09-11T04:41:45-00:00

Re: Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Mon, Sep 10, 2007 at 11:38:10AM -0700, Paul Menage wrote: > By definition any container (about to be renamed control group) > subsystem is some kind of "controller" so that bit seems a bit > redundant. > > Any reason not to just call it "cpu" or "cpu_sched" Done (in the latest patch I sent a while back)! -- Regards, vatsa _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Srivatsa Vaddagiri
date: 2007-09-11T04:44:43-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On 11/09/2007, Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> wrote: > > [ ... ] I guess, 'rq->curr == tsk' implies a task was on the 'rq' (before dequeueing) in this particular case. What's about a minor optimization like below (plus, let's make use of task_running()): [ btw., real-time task can be also added to a container, right? I guess, it can be used at least for group-aware load-balancing of rt_tasks... otherwise, I guess, cpu-resource controlling is not that applicable to, say, SCHED_FIFO :-) Anyway, to this goal rt_task should become aware of group-related bits of the 'struct sched_entity'... and at the moment, the code below is effectively just a 'nop' for them.. right? ] /* change task's runqueue when it moves between groups */ static void sched_move_task(struct container_subsys *ss, struct container *cont, struct container *old_cont, struct task_struct *tsk) { int on_rq, running; unsigned long flags; struct rq *rq; rq = task_rq_lock(tsk, &flags); update_rq_clock(rq); running = task_running(rq, tsk); on_rq = tsk->se.on_rq; if (on_rq) { dequeue_task(rq, tsk, 0); if (unlikely(running) && tsk->sched_class == &fair_sched_class) tsk->sched_class->put_prev_task(rq, tsk); } set_task_cfs_rq(tsk); if (on_rq) { enqueue_task(rq, tsk, 0); if (unlikely(running) && tsk->sched_class == &fair_sched_class) tsk->sched_class->set_curr_task(rq); } task_rq_unlock(rq, &flags); } > -- > Regards, > vatsa > -- Best regards, Dmitry Adamushko _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Dmitry Adamushko
date: 2007-09-11T09:53:51-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Tue, Sep 11, 2007 at 11:53:51AM +0200, Dmitry Adamushko wrote: > I guess, 'rq->curr == tsk' implies a task was on the 'rq' (before > dequeueing) in this particular case. What's about a minor optimization > like below (plus, let's make use of task_running()): Hi Dmitry, The modified code looks very good to me (and neater too!). Updated patch below. > [ btw., real-time task can be also added to a container, right? I am not sure whether we want to allow that. I have been assuming that only SCHED_NORMAL tasks will be in containers. Imagine two SCHED_FIFO tasks being put in two different containers -and- controlling cpu cycle allocation to both containers. Do we actually timeslice the two SCHED_FIFO tasks now that are across containers? Also we need a mechanism to choose a container first to schedule and then the real-time task in it (similar to how its being done in SCHED_NORMAL case). Which means we need one rt_rq per container per cpu. Kirill, do you require real-time tasks to be present in containers? > I guess, it can be used at least for group-aware load-balancing of > rt_tasks... otherwise, I guess, cpu-resource controlling is not that > applicable to, say, SCHED_FIFO :-) Yes, if we were to allow SCHED_FIFO/RR tasks in containers, then the load balancer would need to spread rt-tasks of a container across CPUs to ensure low scheduling latencies for them. > Anyway, to this goal rt_task should become aware of group-related bits > of the 'struct sched_entity'... and at the moment, the code below is > effectively just a 'nop' for them.. right? ] Yes. To emphasise this code doesn't support real-time tasks in a container, I am returning -ENOTSUP when trying to move a rt-task into a container. Thanks for all your review! Add interface to control cpu bandwidth allocation to task-groups. Signed-off-by : Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> Signed-off-by : Dhaval Giani <dhaval@linux.vnet.ibm.com> --- include/linux/container_subsys.h | 6 init/Kconfig | 9 + kernel/…
creator: Srivatsa Vaddagiri
date: 2007-09-12T11:33:47-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Wed, Sep 12, 2007 at 05:12:02PM +0530, Srivatsa Vaddagiri wrote: > Yes. To emphasise this code doesn't support real-time tasks in a container, I > am returning -ENOTSUP when trying to move a rt-task into a container. s/ENOTSUP/EINVAL -- Regards, vatsa _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Srivatsa Vaddagiri
date: 2007-09-12T12:05:06-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: Hi Srivatsa, please find a few more minor comments below. > [ ... ] > + > +/* destroy runqueue etc associated with a task group */ > +static void sched_destroy_group(struct container_subsys *ss, > + struct container *cont) > +{ > + struct task_grp *tg = container_tg(cont); > + struct cfs_rq *cfs_rq; > + struct sched_entity *se; > + int i; > + > + for_each_possible_cpu(i) { > + cfs_rq = tg->cfs_rq[i]; > + list_del_rcu(&cfs_rq->leaf_cfs_rq_list); > + } > + > + /* wait for possible concurrent references to cfs_rqs complete */ > + synchronize_sched(); > + > + /* now it should be safe to free those cfs_rqs */ > + for_each_possible_cpu(i) { > + cfs_rq = tg->cfs_rq[i]; > + kfree(cfs_rq); > + > + se = tg->se[i]; > + kfree(se); > + } > + > + kfree(tg); > +} kfree(tg->cfs_rq) && kfree(tg->se) ? > + > +/* change task's runqueue when it moves between groups */ > +static void sched_move_task(struct container_subsys *ss, struct container *cont, > + struct container *old_cont, struct task_struct *tsk) > +{ > + int on_rq, running; > + unsigned long flags; > + struct rq *rq; > + > + rq = task_rq_lock(tsk, &flags); > + > + if (tsk->sched_class != &fair_sched_class) > + goto done; this check should be redundant now with sched_can_attach() in place. > +static void set_se_shares(struct sched_entity *se, unsigned long shares) > +{ > + struct cfs_rq *cfs_rq = se->cfs_rq; > + struct rq *rq = cfs_rq->rq; > + int on_rq; > + > + spin_lock_irq(&rq->lock); > + > + on_rq = se->on_rq; > + if (on_rq) > + __dequeue_entity(cfs_rq, se); > + > + se->load.weight = shares; > + se->load.inv_weight = div64_64((1ULL<<32), shares); A bit of nit-picking... are you sure, there is no need in non '__' versions of dequeue/enqueu() here (at least, for the sake of update_curr())? Although, I don't have -mm at hand at this very moment and original -rc4 (that I have at hand) doesn't already have 'se->load' at all... so will look later. > > -- > Regards, > vatsa > -- Best regards, Dmitry Adamushko ___________________…
creator: Dmitry Adamushko
date: 2007-09-12T16:25:37-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Wed, Sep 12, 2007 at 06:25:37PM +0200, Dmitry Adamushko wrote: > > + kfree(tg); > > +} > > kfree(tg->cfs_rq) && kfree(tg->se) ? oops, yes! > > + if (tsk->sched_class != &fair_sched_class) > > + goto done; > > this check should be redundant now with sched_can_attach() in place. well, there is remote possibility that task could have changed policies between sched_can_attach() and sched_move_task()! In the updated patch below, I have put some hooks to deal with that scenario elsewhere also (in rt_mutex_setprio and __setscheduler). Pls let me know if you see any issues with that. > > +static void set_se_shares(struct sched_entity *se, unsigned long shares) > > +{ > > + struct cfs_rq *cfs_rq = se->cfs_rq; > > + struct rq *rq = cfs_rq->rq; > > + int on_rq; > > + > > + spin_lock_irq(&rq->lock); > > + > > + on_rq = se->on_rq; > > + if (on_rq) > > + __dequeue_entity(cfs_rq, se); > > + > > + se->load.weight = shares; > > + se->load.inv_weight = div64_64((1ULL<<32), shares); > > A bit of nit-picking... are you sure, there is no need in non '__' > versions of dequeue/enqueu() here (at least, for the sake of You are right, we need the dequeue/enqueue_entity here, as the sched entity load is changing in between the two function calls. How does the updated patch look? (Thanks again for all your review so far!) -- Add interface to control cpu bandwidth allocation to task-groups. Signed-off-by : Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> Signed-off-by : Dhaval Giani <dhaval@linux.vnet.ibm.com> --- include/linux/container_subsys.h | 6 init/Kconfig | 9 + kernel/sched.c | 344 +++++++++++++++++++++++++++++++++++++-- kernel/sched_fair.c | 3 kernel/sched_idletask.c | 5 kernel/sched_rt.c | 5 6 files changed, 355 insertions(+), 17 deletions(-) Index: current/include/linux/container_subsys.h =================================================================== --- current.orig/include/linux/container_subsys.h +++ current/include/linux/container_subsys.h @@ -36,3 +36,9 @@ SUBSYS(mem_conta…
creator: Srivatsa Vaddagiri
date: 2007-09-13T12:27:21-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: * Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> wrote: > Add interface to control cpu bandwidth allocation to task-groups. btw., just in case it was not obvious, i'll repeat my older assessment of your patch: the general picture approach looks good to me and the code is upstream-worthy. ( suggestion: if you want more people to test it then you might want to do some add-on "put all users into separate groups" patch and .config option - which could be tried without people having to know anything about container setup. ) Ingo _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Ingo Molnar
date: 2007-09-14T09:41:58-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Fri, Sep 14, 2007 at 11:41:58AM +0200, Ingo Molnar wrote: > * Srivatsa Vaddagiri <vatsa@linux.vnet.ibm.com> wrote: > > Add interface to control cpu bandwidth allocation to task-groups. > > btw., just in case it was not obvious, i'll repeat my older assessment > of your patch: the general picture approach looks good to me and the > code is upstream-worthy. Thanks for the feedback! > ( suggestion: if you want more people to test it then you might want to > do some add-on "put all users into separate groups" patch and .config > option - which could be tried without people having to know anything > about container setup. ) I do want more people to test it and I agree that hooking onto user-id based groups is the best way to get that done. How do we implement that? I have two choices: 1. Do a kernel patch, as you suggest above, which defines task-groups based on user-id and hook that group definition with group scheduler. We need to provide some means for the admin to tune relative nice-value of each user (perhaps thr' sysctl?). This user-id based grouping will have to be mutually exclusive with task-container based grouping. Hence we need to ensure that only one form of grouping is selected and not both at compile time. 2. Enable only one form of grouping, which is task-container based. Provide a user-space daemon (attached) which can automatically put tasks of different users in different task-containers. The daemon will need to be started at early boot-time. It can also be extended to support a configuration file (ex: inittab) where cpu allocation for different users are specified. The fact that daemon is managing to provide fair allocation to users should be transparent. I hope that task-containers (aka cgroups) will go into 2.6.24, in which case the second option seems to be more attractive to me. I will neverthless try to work out Option 1, just to see how it looks. -- Regards, vatsa _______________________________________________ Containers mailing list Contain…
creator: Srivatsa Vaddagiri
date: 2007-09-14T15:51:13-00:00

Re: [PATCH] Hookup group-scheduler with task container infrastructure
type: item
description: On Fri, Sep 14, 2007 at 09:29:13PM +0530, Srivatsa Vaddagiri wrote: > 2. Enable only one form of grouping, which is task-container based. Provide a > user-space daemon (attached) which can automatically put tasks of different Oops ..forgot to attach. But I realize that I had already sent the source for the daemon here: http://article.gmane.org/gmane.linux.kernel/553267 -- Regards, vatsa _______________________________________________ Containers mailing list Containers@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/containers
creator: Srivatsa Vaddagiri
date: 2007-09-14T16:13:36-00:00