OpenVZ Forum - RDF feed
type: channel
items: [PATCH] BC: resource beancounters (v4) (added user memory)
items: [PATCH 1/13] BC: introduce atomic_dec_and_lock_irqsave()
items: [PATCH 2/13] BC: kconfig
items: [PATCH 3/17] BC: beancounters core (API)
items: [PATCH 4/13] BC: context inheriting and changing
items: [PATCH 5/13] BC: user interface (syscalls)
items: [PATCH 6/13] BC: kernel memory (core)
items: [PATCH 7/13] BC: kernel memory (marks)
items: [PATCH 8/13] BC: locked pages (core)
items: [PATCH 9/13] BC: locked pages (charge hooks)
items: [PATCH 10/13] BC: privvm pages
items: [PATCH 11/13] BC: vmrss (preparations)
items: [PATCH 12/13] BC: vmrss (core)
items: [PATCH 13/13] BC: vmrss (charges)
items: Re: [ckrm-tech] [PATCH 5/13] BC: user interface (syscalls)
items: Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
items: Re: [PATCH 11/13] BC: vmrss (preparations)
items: Re: [PATCH 9/13] BC: locked pages (charge hooks)
items: Re: [ckrm-tech] [PATCH 9/13] BC: locked pages (charge hooks)
items: Re: [PATCH 7/13] BC: kernel memory (marks)
items: Re: [ckrm-tech] [PATCH 11/13] BC: vmrss (preparations)
items: Re: Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)

[PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Core Resource Beancounters (BC) + kernel/user memory control. BC allows to account and control consumption of kernel resources used by group of processes. Draft UBC description on OpenVZ wiki can be found at http://wiki.openvz.org/UBC_parameters The full BC patch set allows to control: - kernel memory. All the kernel objects allocatable on user demand should be accounted and limited for DoS protection. E.g. page tables, task structs, vmas etc. - virtual memory pages. BCs allow to limit a container to some amount of memory and introduces 2-level OOM killer taking into account container's consumption. pages shared between containers are correctly charged as fractions (tunable). - network buffers. These includes TCP/IP rcv/snd buffers, dgram snd buffers, unix, netlinks and other buffers. - minor resources accounted/limited by number: tasks, files, flocks, ptys, siginfo, pinned dcache mem, sockets, iptentries (for containers with virtualized networking) As the first step we want to propose for discussion the most complicated parts of resource management: kernel memory and virtual memory. The patch set to be sent provides core for BC and management of kernel memory only. Virtual memory management will be sent in a couple of days. The patches in these series are: diff-atomic-dec-and-lock-irqsave.patch introduce atomic_dec_and_lock_irqsave() diff-bc-kconfig.patch: Adds kernel/bc/Kconfig file with UBC options and includes it into arch Kconfigs diff-bc-core.patch: Contains core functionality and interfaces of BC: find/create beancounter, initialization, charge/uncharge of resource, core objects' declarations. diff-bc-task.patch: Contains code responsible for setting BC on task, it's inheriting and setting host context in interrupts. Task contains three beancounters: 1. exec_bc - current context. all resources are charged to this beancounter. 2. fork_bc - beancounter which is inherited by task's children on fork diff-bc-syscalls.patch: Patch adds system calls for BC managemen…
creator: dev
date: 2006-09-05T14:59:30-00:00

[PATCH 1/13] BC: introduce atomic_dec_and_lock_irqsave()
type: item
description: Oleg Nesterov noticed to me that the construction like (used in beancounter patches and free_uid()): local_irq_save(flags); if (atomic_dec_and_lock(&refcnt, &lock)) ... is not that good for preemtible kernels, since with preemption spin_lock() can schedule() to reduce latency. However, it won't schedule if interrupts are disabled. So this patch introduces atomic_dec_and_lock_irqsave() as a logical counterpart to atomic_dec_and_lock(). Signed-Off-By: Pavel Emelianov <xemul@sw.ru> Signed-Off-By: Kirill Korotaev <dev@sw.ru> --- include/linux/spinlock.h | 6 ++++++ kernel/user.c | 5 +---- lib/dec_and_lock.c | 19 +++++++++++++++++++ 3 files changed, 26 insertions(+), 4 deletions(-) --- ./include/linux/spinlock.h.dlirq 2006-08-28 10:17:35.000000000 +0400 +++ ./include/linux/spinlock.h 2006-08-28 11:22:37.000000000 +0400 @@ -266,6 +266,12 @@ extern int _atomic_dec_and_lock(atomic_t #define atomic_dec_and_lock(atomic, lock) \ __cond_lock(lock, _atomic_dec_and_lock(atomic, lock)) +extern int _atomic_dec_and_lock_irqsave(atomic_t *atomic, spinlock_t *lock, + unsigned long *flagsp); +#define atomic_dec_and_lock_irqsave(atomic, lock, flags) \ + __cond_lock(lock, \ + _atomic_dec_and_lock_irqsave(atomic, lock, &flags)) + /** * spin_can_lock - would spin_trylock() succeed? * @lock: the spinlock in question. --- ./kernel/user.c.dlirq 2006-07-10 12:39:20.000000000 +0400 +++ ./kernel/user.c 2006-08-28 11:08:56.000000000 +0400 @@ -108,15 +108,12 @@ void free_uid(struct user_struct *up) if (!up) return; - local_irq_save(flags); - if (atomic_dec_and_lock(&up->__count, &uidhash_lock)) { + if (atomic_dec_and_lock_irqsave(&up->__count, &uidhash_lock, flags)) { uid_hash_remove(up); spin_unlock_irqrestore(&uidhash_lock, flags); key_put(up->uid_keyring); key_put(up->session_keyring); kmem_cache_free(uid_cachep, up); - } else { - local_irq_restore(flags); } } --- ./lib/dec_and_lock.c.dlirq 2006-04-21 11:59:36.000000000 +0400 +++ ./lib/dec_and_lock.c 2006-08-28 11:22:08.000000000 +0400 @@ -33,3 …
creator: dev
date: 2006-09-05T15:16:08-00:00

[PATCH 2/13] BC: kconfig
type: item
description: Add kernel/bc/Kconfig file with BC options and include it into arch Kconfigs Signed-off-by: Pavel Emelianov <xemul@sw.ru> Signed-off-by: Kirill Korotaev <dev@sw.ru> --- init/Kconfig | 2 ++ kernel/bc/Kconfig | 25 +++++++++++++++++++++++++ 2 files changed, 27 insertions(+) --- ./init/Kconfig.bckm 2006-07-10 12:39:10.000000000 +0400 +++ ./init/Kconfig 2006-07-28 14:10:41.000000000 +0400 @@ -222,6 +222,8 @@ source "crypto/Kconfig" Say N if unsure. +source "kernel/bc/Kconfig" + config SYSCTL bool --- ./kernel/bc/Kconfig.bckconf 2006-09-05 12:21:09.000000000 +0400 +++ ./kernel/bc/Kconfig 2006-09-05 12:19:54.000000000 +0400 @@ -0,0 +1,25 @@ +# +# Resource beancounters (BC) +# +# Copyright (C) 2006 OpenVZ. SWsoft Inc + +menu "User resources" + +config BEANCOUNTERS + bool "Enable resource accounting/control" + default n + help + When Y this option provides accounting and allows configuring + limits for user's consumption of exhaustible system resources. + The most important resource controlled by this patch is unswappable + memory (either mlock'ed or used by internal kernel structures and + buffers). The main goal of this patch is to protect processes + from running short of important resources because of accidental + misbehavior of processes or malicious activity aiming to ``kill'' + the system. It's worth mentioning that resource limits configured + by setrlimit(2) do not give an acceptable level of protection + because they cover only a small fraction of resources and work on a + per-process basis. Per-process accounting doesn't prevent malicious + users from spawning a lot of resource-consuming processes. + +endmenu
creator: dev
date: 2006-09-05T15:16:34-00:00

[PATCH 3/17] BC: beancounters core (API)
type: item
description: Core functionality and interfaces of BC: find/create beancounter, initialization, charge/uncharge of resource, core objects' declarations. Basic structures: bc_resource_parm - resource description beancounter - set of resources, id, lock Signed-off-by: Pavel Emelianov <xemul@sw.ru> Signed-off-by: Kirill Korotaev <dev@sw.ru> --- include/bc/beancounter.h | 155 +++++++++++++++++++++++++++ include/linux/types.h | 16 ++ init/main.c | 4 kernel/Makefile | 1 kernel/bc/Makefile | 7 + kernel/bc/beancounter.c | 263 +++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 446 insertions(+) --- ./include/bc/beancounter.h.bccore 2006-09-05 12:06:35.000000000 +0400 +++ ./include/bc/beancounter.h 2006-09-05 12:15:57.000000000 +0400 @@ -0,0 +1,155 @@ +/* + * include/bc/beancounter.h + * + * Copyright (C) 2006 OpenVZ. SWsoft Inc + * + */ + +#ifndef _LINUX_BEANCOUNTER_H +#define _LINUX_BEANCOUNTER_H + +/* + * Resource list. + */ + +#define BC_RESOURCES 0 + +struct bc_resource_parm { + unsigned long barrier; /* A barrier over which resource allocations + * are failed gracefully. e.g. if the amount + * of consumed memory is over the barrier + * further sbrk() or mmap() calls fail, the + * existing processes are not killed. + */ + unsigned long limit; /* hard resource limit */ + unsigned long held; /* consumed resources */ + unsigned long maxheld; /* maximum amount of consumed resources */ + unsigned long minheld; /* minumum amount of consumed resources */ + unsigned long failcnt; /* count of failed charges */ +}; + +/* + * Kernel internal part. + */ + +#ifdef __KERNEL__ + +#include <linux/spinlock.h> +#include <linux/list.h> +#include <asm/atomic.h> + +#define BC_MAXVALUE LONG_MAX + +/* + * Resource management structures + * Serialization issues: + * beancounter list management is protected via bc_hash_lock + * task pointers are set only for current task and only once + * refcount is managed atomically + * value and limit comparison and change are protected by per-bc spinlock + …
creator: dev
date: 2006-09-05T15:17:35-00:00

[PATCH 4/13] BC: context inheriting and changing
type: item
description: Contains code responsible for setting BC on task, it's inheriting and setting host context in interrupts. Task references 2 beancounters: 1. exec_bc: current context. all resources are charged to this beancounter. 3. fork_bc: beancounter which is inherited by task's children on fork Signed-off-by: Pavel Emelianov <xemul@sw.ru> Signed-off-by: Kirill Korotaev <dev@sw.ru> --- include/bc/task.h | 57 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/sched.h | 5 ++++ kernel/bc/Makefile | 1 kernel/bc/beancounter.c | 3 ++ kernel/bc/misc.c | 31 ++++++++++++++++++++++++++ kernel/fork.c | 5 ++++ kernel/irq/handle.c | 9 +++++++ kernel/softirq.c | 8 ++++++ 8 files changed, 119 insertions(+) --- ./include/bc/task.h.bctask 2006-09-05 12:24:07.000000000 +0400 +++ ./include/bc/task.h 2006-09-05 12:38:53.000000000 +0400 @@ -0,0 +1,57 @@ +/* + * include/bc/task.h + * + * Copyright (C) 2006 OpenVZ. SWsoft Inc + * + */ + +#ifndef __BC_TASK_H_ +#define __BC_TASK_H_ + +struct beancounter; + +struct task_beancounter { + struct beancounter *exec_bc; + struct beancounter *fork_bc; +}; + +#ifdef CONFIG_BEANCOUNTERS + +#define get_exec_bc() (current->task_bc.exec_bc) + +#define set_exec_bc(new) ({ \ + struct task_beancounter *tbc; \ + struct beancounter *old; \ + tbc = &current->task_bc; \ + old = tbc->exec_bc; \ + tbc->exec_bc = new; \ + old; \ + }) + +#define reset_exec_bc(old, expected) do { \ + struct task_beancounter *tbc; \ + tbc = &current->task_bc; \ + BUG_ON(tbc->exec_bc != expected); \ + tbc->exec_bc = old; \ + } while (0) + +void bc_task_charge(struct task_struct *parent, struct task_struct *new); +void bc_task_uncharge(struct task_struct *tsk); + +#else + +#define get_exec_bc() (NULL) +#define set_exec_bc(new) (NULL) +#define reset_exec_bc(new, expected) do { } while (0) + +static inline void bc_task_charge(struct task_struct *parent, + struct task_struct *new) +{ +} + +static inline void bc_task_uncharge(struct task_struct *tsk) +{ +} + +#endif +#endif --- ./include/l…
creator: dev
date: 2006-09-05T15:19:40-00:00

[PATCH 5/13] BC: user interface (syscalls)
type: item
description: Add the following system calls for BC management: 1. sys_get_bcid - get current BC id 2. sys_set_bcid - change exec_ and fork_ BCs on current 3. sys_set_bclimit - set limits for resources consumtions 4. sys_get_bcstat - return br_resource_parm on resource Signed-off-by: Pavel Emelianov <xemul@sw.ru> Signed-off-by: Kirill Korotaev <dev@sw.ru> --- arch/i386/kernel/syscall_table.S | 4 + arch/ia64/kernel/entry.S | 4 + arch/sparc/kernel/entry.S | 2 arch/sparc/kernel/systbls.S | 6 + arch/sparc64/kernel/entry.S | 2 arch/sparc64/kernel/systbls.S | 10 ++- include/asm-i386/unistd.h | 6 + include/asm-ia64/unistd.h | 6 + include/asm-powerpc/systbl.h | 4 + include/asm-powerpc/unistd.h | 6 + include/asm-sparc/unistd.h | 4 + include/asm-sparc64/unistd.h | 4 + include/asm-x86_64/unistd.h | 10 ++- kernel/bc/Makefile | 1 kernel/bc/sys.c | 120 +++++++++++++++++++++++++++++++++++++++ kernel/sys_ni.c | 6 + 16 files changed, 186 insertions(+), 9 deletions(-) --- ./arch/i386/kernel/syscall_table.S.bcsys 2006-09-05 11:47:31.000000000 +0400 +++ ./arch/i386/kernel/syscall_table.S 2006-09-05 12:47:21.000000000 +0400 @@ -318,3 +318,7 @@ ENTRY(sys_call_table) .long sys_vmsplice .long sys_move_pages .long sys_getcpu + .long sys_get_bcid + .long sys_set_bcid /* 320 */ + .long sys_set_bclimit + .long sys_get_bcstat --- ./arch/ia64/kernel/entry.S.bcsys 2006-09-05 11:47:31.000000000 +0400 +++ ./arch/ia64/kernel/entry.S 2006-09-05 12:47:21.000000000 +0400 @@ -1610,5 +1610,9 @@ sys_call_table: data8 sys_sync_file_range // 1300 data8 sys_tee data8 sys_vmsplice + data8 sys_get_bcid + data8 sys_set_bcid + data8 sys_set_bclimit // 1305 + data8 sys_get_bcstat .org sys_call_table + 8*NR_syscalls // guard against failures to increase NR_syscalls --- ./arch/sparc/kernel/entry.S.bcsys 2006-07-10 12:39:10.000000000 +0400 +++ ./arch/sparc/kernel/entry.S 2006-09-05 12:47:21.000000000 +0400 @@ -37,7 +37,7 @@ #define curptr g6 -#define NR_SYSCALLS 300 /* Each OS is different... */ +#define NR_SYSCALLS 304 /* Each O…
creator: dev
date: 2006-09-05T15:21:07-00:00

[PATCH 6/13] BC: kernel memory (core)
type: item
description: Introduce BC_KMEMSIZE resource which accounts kernel objects allocated by task's request. Reference to BC is kept on struct page or slab object. For slabs each struct slab contains a set of pointers corresponding objects are charged to. Allocation charge rules: 1. Pages - if allocation is performed with __GFP_BC flag - page is charged to current's exec_bc. 2. Slabs - kmem_cache may be created with SLAB_BC flag - in this case each allocation is charged. Caches used by kmalloc are created with SLAB_BC | SLAB_BC_NOCHARGE flags. In this case only __GFP_BC allocations are charged. Signed-off-by: Pavel Emelianov <xemul@sw.ru> Signed-off-by: Kirill Korotaev <dev@sw.ru> --- include/bc/beancounter.h | 4 + include/bc/kmem.h | 46 +++++++++++++++++ include/linux/gfp.h | 8 ++- include/linux/mm.h | 4 + include/linux/slab.h | 4 + include/linux/vmalloc.h | 1 kernel/bc/Makefile | 1 kernel/bc/beancounter.c | 3 + kernel/bc/kmem.c | 85 +++++++++++++++++++++++++++++++++ mm/mempool.c | 2 mm/page_alloc.c | 11 ++++ mm/slab.c | 121 ++++++++++++++++++++++++++++++++++++++--------- mm/vmalloc.c | 6 ++ 13 files changed, 271 insertions(+), 25 deletions(-) --- ./include/bc/beancounter.h.bckmemcore 2006-09-05 12:54:17.000000000 +0400 +++ ./include/bc/beancounter.h 2006-09-05 12:54:40.000000000 +0400 @@ -12,7 +12,9 @@ * Resource list. */ -#define BC_RESOURCES 0 +#define BC_KMEMSIZE 0 + +#define BC_RESOURCES 1 struct bc_resource_parm { unsigned long barrier; /* A barrier over which resource allocations --- /dev/null 2006-07-18 14:52:43.075228448 +0400 +++ ./include/bc/kmem.h 2006-09-05 12:54:40.000000000 +0400 @@ -0,0 +1,46 @@ +/* + * include/bc/kmem.h + * + * Copyright (C) 2006 OpenVZ. SWsoft Inc + * + */ + +#ifndef __BC_KMEM_H_ +#define __BC_KMEM_H_ + +/* + * BC_KMEMSIZE accounting + */ + +struct mm_struct; +struct page; +struct beancounter; + +#ifdef CONFIG_BEANCOUNTERS +int __must_check bc_page_charge(struct page *page, int order, gfp_t flags); +void bc_page_uncharge(struct page *page, int order…
creator: dev
date: 2006-09-05T15:21:47-00:00

[PATCH 7/13] BC: kernel memory (marks)
type: item
description: Mark some kmem caches with SLAB_BC and some allocations with __GFP_BC to cause charging/limiting of appropriate kernel resources. Signed-off-by: Pavel Emelianov <xemul@sw.ru> Signed-off-by: Kirill Korotaev <dev@sw.ru> --- arch/i386/kernel/ldt.c | 4 ++-- arch/i386/mm/init.c | 4 ++-- arch/i386/mm/pgtable.c | 6 ++++-- drivers/char/tty_io.c | 10 +++++----- fs/file.c | 8 ++++---- fs/locks.c | 2 +- fs/namespace.c | 3 ++- fs/select.c | 7 ++++--- include/asm-i386/thread_info.h | 4 ++-- include/asm-ia64/pgalloc.h | 24 +++++++++++++++++------- include/asm-x86_64/pgalloc.h | 12 ++++++++---- include/asm-x86_64/thread_info.h | 5 +++-- ipc/msgutil.c | 4 ++-- ipc/sem.c | 7 ++++--- ipc/util.c | 8 ++++---- kernel/fork.c | 15 ++++++++------- kernel/posix-timers.c | 3 ++- kernel/signal.c | 2 +- kernel/user.c | 2 +- mm/rmap.c | 3 ++- mm/shmem.c | 3 ++- 21 files changed, 80 insertions(+), 56 deletions(-) --- ./arch/i386/kernel/ldt.c.bckmemch 2006-09-05 12:53:51.000000000 +0400 +++ ./arch/i386/kernel/ldt.c 2006-09-05 12:58:17.000000000 +0400 @@ -39,9 +39,9 @@ static int alloc_ldt(mm_context_t *pc, i oldsize = pc->size; mincount = (mincount+511)&(~511); if (mincount*LDT_ENTRY_SIZE > PAGE_SIZE) - newldt = vmalloc(mincount*LDT_ENTRY_SIZE); + newldt = vmalloc_bc(mincount*LDT_ENTRY_SIZE); else - newldt = kmalloc(mincount*LDT_ENTRY_SIZE, GFP_KERNEL); + newldt = kmalloc(mincount*LDT_ENTRY_SIZE, GFP_KERNEL_BC); if (!newldt) return -ENOMEM; --- ./arch/i386/mm/init.c.bckmemch 2006-09-05 12:53:51.000000000 +0400 +++ ./arch/i386/mm/init.c 2006-09-05 12:58:17.000000000 +0400 @@ -709,7 +709,7 @@ void __init pgtable_cache_init(void) pmd_cache = kmem_cache_create("pmd", PTRS_PER_PMD*sizeof(pmd_t), PTRS_PER_PMD*sizeof(pmd_t), - 0, + SLAB_BC, pmd_ctor, NULL); if (!pmd_cache) @@ -718,7 +718,7 @@ void __init pgtable_cache_init(void) pgd_cache = kmem_cache_create("pgd", PTRS_PER_PGD*sizeof(pgd_t), PTRS_PER_PGD*sizeof(pgd_t), - 0, + SLAB_BC, pgd_ctor, PTRS_PER_PMD == 1 ? pgd_dtor : NULL); if (!pgd_cache) --- …
creator: dev
date: 2006-09-05T15:23:00-00:00

[PATCH 8/13] BC: locked pages (core)
type: item
description: Introduce new resource BC_LOCKEDPAGES which stands for accounting of mlock-ed user pages. Locked pages are important to be accounted separately as they are unreclaimable. Pages are charged to mm_struct BC. Signed-Off-By: Pavel Emelianov <xemul@sw.ru> Signed-Off-By: Kirill Korotaev <dev@sw.ru> --- include/bc/beancounter.h | 3 - include/bc/vmpages.h | 95 +++++++++++++++++++++++++++++++++++++++++++++++ include/linux/sched.h | 3 + include/linux/shmem_fs.h | 5 ++ kernel/bc/Makefile | 1 kernel/bc/beancounter.c | 2 kernel/bc/vmpages.c | 75 +++++++++++++++++++++++++++++++++++++ kernel/fork.c | 11 +++-- mm/shmem.c | 4 + 9 files changed, 195 insertions(+), 4 deletions(-) --- ./include/bc/beancounter.h.bclockcore 2006-09-05 12:54:40.000000000 +0400 +++ ./include/bc/beancounter.h 2006-09-05 12:59:27.000000000 +0400 @@ -13,8 +13,9 @@ */ #define BC_KMEMSIZE 0 +#define BC_LOCKEDPAGES 1 -#define BC_RESOURCES 1 +#define BC_RESOURCES 2 struct bc_resource_parm { unsigned long barrier; /* A barrier over which resource allocations --- /dev/null 2006-07-18 14:52:43.075228448 +0400 +++ ./include/bc/vmpages.h 2006-09-05 13:04:03.000000000 +0400 @@ -0,0 +1,95 @@ +/* + * include/bc/vmpages.h + * + * Copyright (C) 2006 OpenVZ. SWsoft Inc + * + */ + +#ifndef __BC_VMPAGES_H_ +#define __BC_VMPAGES_H_ + +#include <bc/beancounter.h> +#include <bc/task.h> + +struct mm_struct; +struct file; +struct shmem_inode_info; + +#ifdef CONFIG_BEANCOUNTERS +int __must_check bc_memory_charge(struct mm_struct *mm, unsigned long size, + unsigned long vm_flags, struct file *vm_file, int strict); +void bc_memory_uncharge(struct mm_struct *mm, unsigned long size, + unsigned long vm_flags, struct file *vm_file); + +int __must_check bc_locked_charge(struct mm_struct *mm, unsigned long size); +void bc_locked_uncharge(struct mm_struct *mm, unsigned long size); + +int __must_check bc_locked_shm_charge(struct shmem_inode_info *info, + unsigned long size); +void bc_locked_shm_uncharge(struct shmem_inode_info *info, + unsig…
creator: dev
date: 2006-09-05T15:24:18-00:00

[PATCH 9/13] BC: locked pages (charge hooks)
type: item
description: Introduce calls to BC core over the kernel to charge locked memory. Normaly new locked piece of memory may appear in insert_vm_struct, but there are places (do_mmap_pgoff, dup_mmap etc) when new vma is not inserted by insert_vm_struct(), but either link_vma-ed or merged with some other - these places call BC code explicitly. Plus sys_mlock[all] itself has to be patched to charge/uncharge needed amount of pages. Signed-Off-By: Pavel Emelianov <xemul@sw.ru> Signed-Off-By: Kirill Korotaev <dev@sw.ru> --- fs/binfmt_elf.c | 5 ++- include/asm-alpha/mman.h | 1 include/asm-generic/mman.h | 1 include/asm-mips/mman.h | 1 include/asm-parisc/mman.h | 1 include/linux/mm.h | 1 mm/mlock.c | 21 +++++++++++++--- mm/mmap.c | 59 ++++++++++++++++++++++++++++++++++++++------- mm/mremap.c | 18 ++++++++++++- mm/shmem.c | 12 ++++++++- 10 files changed, 104 insertions(+), 16 deletions(-) --- ./fs/binfmt_elf.c.bclockcharge 2006-09-05 12:53:54.000000000 +0400 +++ ./fs/binfmt_elf.c 2006-09-05 13:08:26.000000000 +0400 @@ -360,7 +360,7 @@ static unsigned long load_elf_interp(str eppnt = elf_phdata; for (i = 0; i < interp_elf_ex->e_phnum; i++, eppnt++) { if (eppnt->p_type == PT_LOAD) { - int elf_type = MAP_PRIVATE | MAP_DENYWRITE; + int elf_type = MAP_PRIVATE|MAP_DENYWRITE|MAP_EXECPRIO; int elf_prot = 0; unsigned long vaddr = 0; unsigned long k, map_addr; @@ -846,7 +846,8 @@ static int load_elf_binary(struct linux_ if (elf_ppnt->p_flags & PF_X) elf_prot |= PROT_EXEC; - elf_flags = MAP_PRIVATE | MAP_DENYWRITE | MAP_EXECUTABLE; + elf_flags = MAP_PRIVATE | MAP_DENYWRITE | + MAP_EXECUTABLE | MAP_EXECPRIO; vaddr = elf_ppnt->p_vaddr; if (loc->elf_ex.e_type == ET_EXEC || load_addr_set) { --- ./include/asm-alpha/mman.h.mapfx 2006-04-21 11:59:35.000000000 +0400 +++ ./include/asm-alpha/mman.h 2006-09-05 18:13:12.000000000 +0400 @@ -28,6 +28,7 @@ #define MAP_NORESERVE 0x10000 /* don't check for reservations */ #define MAP_POPULATE 0x20000 /* populate (prefault) pagetables */ #define MAP_NONBLOCK 0x40000 /* …
creator: dev
date: 2006-09-05T15:25:53-00:00

[PATCH 10/13] BC: privvm pages
type: item
description: This patch instroduces new resource - BC_PRIVVMPAGES. It is an upper estimation of currently used physical memory. There are different approaches to user pages control: a) account all the mappings on mmap/brk and reject as soon as the sum of VMA's lengths reaches the barrier. This approach is very bad as applications always map more than they really use, very often MUCH more. b) account only the really used memory and reject as soon as RSS reaches the limit. This approach is not good either as user space pages are allocated in page fault handler and the only way to reject allocation is to kill the task. Comparing to previous scenarion this is much worse as application won't even be able to terminate gracefully. c) account a part of memory on mmap/brk and reject there, and account the rest of the memory in page fault handlers without any rejects. This type of accounting is used in UBC. d) account physical memory and behave like a standalone kernel - reclaim user memory when run out of it. This type of memory control is to be introduced later as an addition to c). UBC provides all the needed statistics for this (physical memory, swap pages etc.) Privvmpages accounting is described in details in http://wiki.openvz.org/User_pages_accounting A note about sys_mprotect: as it can change mapping state from BC_VM_PRIVATE to !BC_VM_PRIVATE and vice-versa appropriate amount of pages is (un)charged in mprotect_fixup. Signed-Off-By: Pavel Emelianov <xemul@sw.ru> Signed-Off-By: Kirill Korotaev <dev@sw.ru> --- include/bc/beancounter.h | 3 +- include/bc/vmpages.h | 44 +++++++++++++++++++++++++++++++++++++++ kernel/bc/beancounter.c | 2 + kernel/bc/vmpages.c | 53 ++++++++++++++++++++++++++++++++++++++++++++--- kernel/fork.c | 9 +++++++ mm/mprotect.c | 17 ++++++++++++++- mm/shmem.c | 7 ++++++ 7 files changed, 129 insertions(+), 6 deletions(-) --- ./include/bc/beancounter.h.bcprivvm 2006-09-05 12:59:27.000000000 +0400 +++ ./include/bc/beancounter.h 2006-09-05 13:17:50.000000000 +0400 @…
creator: dev
date: 2006-09-05T15:26:58-00:00

[PATCH 11/13] BC: vmrss (preparations)
type: item
description: This patch does simple things: - intruduces an bc_magic field on beancunter to make sure union on struct page is correctly used in next patches - adds nr_beancounters - adds unused_privvmpages variable (counter of privvm pages which are not mapped into VM address space and thus potentially can be allocated later) This is needed by vmrss accounting and is done to make patch reviewing simpler. Signed-Off-By: Pavel Emelianov <xemul@sw.ru> Signed-Off-By: Kirill Korotaev <dev@sw.ru> --- include/bc/beancounter.h | 13 +++++++++++++ include/bc/vmpages.h | 2 ++ kernel/bc/beancounter.c | 5 +++++ kernel/bc/kmem.c | 1 + kernel/bc/vmpages.c | 44 ++++++++++++++++++++++++++++++++++++++++---- 5 files changed, 61 insertions(+), 4 deletions(-) --- ./include/bc/beancounter.h.bcvmrssprep 2006-09-05 13:17:50.000000000 +0400 +++ ./include/bc/beancounter.h 2006-09-05 13:44:33.000000000 +0400 @@ -45,6 +45,13 @@ struct bc_resource_parm { #define BC_MAXVALUE LONG_MAX /* + * This magic is used to distinuish user beancounter and pages beancounter + * in struct page. page_ub and page_bc are placed in union and MAGIC + * ensures us that we don't use pbc as ubc in bc_page_uncharge(). + */ +#define BC_MAGIC 0x62756275UL + +/* * Resource management structures * Serialization issues: * beancounter list management is protected via bc_hash_lock @@ -54,11 +61,13 @@ struct bc_resource_parm { */ struct beancounter { + unsigned long bc_magic; atomic_t bc_refcount; spinlock_t bc_lock; bcid_t bc_id; struct hlist_node hash; + unsigned long unused_privvmpages; /* resources statistics and settings */ struct bc_resource_parm bc_parms[BC_RESOURCES]; }; @@ -74,6 +83,8 @@ enum bc_severity { BC_BARRIER, BC_LIMIT, #ifdef CONFIG_BEANCOUNTERS +extern unsigned int nr_beancounters = 1; + /* * These functions tune minheld and maxheld values for a given * resource when held value changes @@ -137,6 +137,8 @@ extern const char *bc_rnames[]; #else /* CONFIG_BEANCOUNTERS */ +#define nr_beancounters 0 + #define beancounter_fin…
creator: dev
date: 2006-09-05T15:28:05-00:00

[PATCH 12/13] BC: vmrss (core)
type: item
description: This is the core of vmrss accounting. The main introduced object is page_beancounter. It ties together page and BCs which use the page. This allows correctly account fractions of memory shared between BCs (http://wiki.openvz.org/RSS_fractions_accounting) Accounting API: 1. bc_alloc_rss_counter() allocates a tie between page and BC 2. bc_free_rss_counter frees it. (1) and (2) must be done each time a page is about to be added to someone's rss. 3. When page is touched by BC (i.e. by any task which mm belongs to BC) page is bc_vmrss_page_add()-ed to that BC. Touching page leads to subtracting it from unused_prvvmpages and adding to held_pages. 4. When page is unmapped from BC it is bc_vmrss_page_del()-ed from it. 5. When task forks all it's mapped pages must be bc_vmrss_page_dup()-ed. i.e. page beancounter reference counter must be increased. 6. Some pages (former PGReserved) must be added to rss, but without having a reference on it. These pages are bc_vmrss_page_add_noref()-ed. Signed-Off-By: Pavel Emelianov <xemul@sw.ru> Signed-Off-By: Kirill Korotaev <dev@sw.ru> --- include/bc/beancounter.h | 3 include/bc/vmpages.h | 4 include/bc/vmrss.h | 72 ++++++ include/linux/mm.h | 6 include/linux/shmem_fs.h | 2 init/main.c | 2 kernel/bc/Kconfig | 9 kernel/bc/Makefile | 1 kernel/bc/beancounter.c | 9 kernel/bc/vmpages.c | 7 kernel/bc/vmrss.c | 508 +++++++++++++++++++++++++++++++++++++++++++++++ mm/shmem.c | 6 12 files changed, 627 insertions(+), 2 deletions(-) --- ./include/bc/beancounter.h.bcrsscore 2006-09-05 13:44:33.000000000 +0400 +++ ./include/bc/beancounter.h 2006-09-05 13:50:29.000000000 +0400 @@ -68,6 +68,9 @@ struct beancounter { struct hlist_node hash; unsigned long unused_privvmpages; +#ifdef CONFIG_BEANCOUNTERS_RSS + unsigned long long rss_pages; +#endif /* resources statistics and settings */ struct bc_resource_parm bc_parms[BC_RESOURCES]; }; --- ./include/bc/vmpages.h.bcrsscore 2006-09-05 13:40:21.000000000 +0400 +++ ./include/bc/vmpages.h 2006-09-05 13:46:35.000…
creator: dev
date: 2006-09-05T15:28:58-00:00

[PATCH 13/13] BC: vmrss (charges)
type: item
description: Introduce calls to BC code over the kernel to add accounting of physical pages/privvmpages. Signed-Off-By: Pavel Emelianov <xemul@sw.ru> Signed-Off-By: Kirill Korotaev <dev@sw.ru> --- fs/exec.c | 11 ++++ include/linux/mm.h | 3 - kernel/fork.c | 2 mm/filemap_xip.c | 2 mm/fremap.c | 11 ++++ mm/memory.c | 141 +++++++++++++++++++++++++++++++++++++++++------------ mm/migrate.c | 3 + mm/mprotect.c | 12 +++- mm/rmap.c | 4 + mm/swapfile.c | 47 ++++++++++++----- 10 files changed, 186 insertions(+), 50 deletions(-) --- ./fs/exec.c.bcrssch 2006-09-05 12:53:55.000000000 +0400 +++ ./fs/exec.c 2006-09-05 13:51:55.000000000 +0400 @@ -50,6 +50,8 @@ #include <linux/cn_proc.h> #include <linux/audit.h> +#include <bc/vmrss.h> + #include <asm/uaccess.h> #include <asm/mmu_context.h> @@ -308,6 +310,11 @@ void install_arg_page(struct vm_area_str struct mm_struct *mm = vma->vm_mm; pte_t * pte; spinlock_t *ptl; + struct page_beancounter *pb; + + pb = bc_alloc_rss_counter(); + if (IS_ERR(pb)) + goto out_nopb; if (unlikely(anon_vma_prepare(vma))) goto out; @@ -325,11 +332,15 @@ void install_arg_page(struct vm_area_str set_pte_at(mm, address, pte, pte_mkdirty(pte_mkwrite(mk_pte( page, vma->vm_page_prot)))); page_add_new_anon_rmap(page, vma, address); + bc_vmrss_page_add(page, mm, vma, &pb); pte_unmap_unlock(pte, ptl); /* no need for flush_tlb */ + bc_free_rss_counter(pb); return; out: + bc_free_rss_counter(pb); +out_nopb: __free_page(page); force_sig(SIGKILL, current); } --- ./include/linux/mm.h.bcrssch 2006-09-05 13:47:12.000000000 +0400 +++ ./include/linux/mm.h 2006-09-05 13:51:55.000000000 +0400 @@ -753,7 +753,8 @@ void free_pgd_range(struct mmu_gather ** void free_pgtables(struct mmu_gather **tlb, struct vm_area_struct *start_vma, unsigned long floor, unsigned long ceiling); int copy_page_range(struct mm_struct *dst, struct mm_struct *src, - struct vm_area_struct *vma); + struct vm_area_struct *vma, + struct vm_area_struct *dst_vma); int zeromap_page_range(struct vm_area_struct *vma, unsign…
creator: dev
date: 2006-09-05T15:29:59-00:00

Re: [ckrm-tech] [PATCH 5/13] BC: user interface (syscalls)
type: item
description: > + > +asmlinkage long sys_set_bcid(bcid_t id) > +{ > + int error; > + struct beancounter *bc; > + struct task_beancounter *task_bc; > + > + task_bc = &current->task_bc; I was playing around with the bc patches and found that to make use of bc's, I had to actually call set_bcid() and then exec() a task/shell so that the id would stick around. Would you consider changing sys_set_bcid to sys_set_task_bcid() or adding a new system call sys_set_task_bcid()? We could pass the pid that we intend to associate with the new id. This also means we'll need locking around to protect task->task_bc. -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-05T16:04:52-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Kirill Korotaev wrote: > Core Resource Beancounters (BC) + kernel/user memory control. > > BC allows to account and control consumption > of kernel resources used by group of processes. > > Draft UBC description on OpenVZ wiki can be found at > http://wiki.openvz.org/UBC_parameters > > The full BC patch set allows to control: > - kernel memory. All the kernel objects allocatable > on user demand should be accounted and limited > for DoS protection. > E.g. page tables, task structs, vmas etc. One of the key requirements of resource management for us is to be able to migrate tasks across resource groups. Since bean counters do not associate a list of tasks associated with them, I do not see how this can be done with the existing bean counters. -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-05T16:53:21-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-05 at 19:02 +0400, Kirill Korotaev wrote: > Core Resource Beancounters (BC) + kernel/user memory control. > > BC allows to account and control consumption > of kernel resources used by group of processes. Hi Kirill, I've honestly lost track of these discussions along the way, so I hope you don't mind summarizing a bit. Do these patches help with accounting for anything other than memory? Will we need new user/kernel interfaces for cpu, i/o bandwidth, etc...? Have you given any thought to the possibility that a task might need to move between accounting contexts? That has certainly been a "requirement" pushed on to CKRM for a long time, and the need goes something like this: 1. A system runs a web server, which services several virtual domains 2. that web server receives a request for foo.com 3. the web server switches into foo.com's accounting context 4. the web server reads things from disk, allocates some memory, and makes a database request. 5. the database receives the request, and switches into foo.com's accounting context, and charges foo.com for its resource use etc... So, the goal is to run _one_ copy of an application on a system, but account for its resources in a much more fine-grained way than at the application level. I think we can probably use beancounters for this, if we do not worry about migrating _existing_ charges when we change accounting context. Does that make sense? -- Dave
creator: Dave Hansen
date: 2006-09-05T17:46:32-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Dave Hansen wrote: > On Tue, 2006-09-05 at 19:02 +0400, Kirill Korotaev wrote: >> Core Resource Beancounters (BC) + kernel/user memory control. >> >> BC allows to account and control consumption >> of kernel resources used by group of processes. > > Hi Kirill, > > I've honestly lost track of these discussions along the way, so I hope > you don't mind summarizing a bit. > > Do these patches help with accounting for anything other than memory? > Will we need new user/kernel interfaces for cpu, i/o bandwidth, etc...? > > Have you given any thought to the possibility that a task might need to > move between accounting contexts? That has certainly been a > "requirement" pushed on to CKRM for a long time, and the need goes > something like this: > > 1. A system runs a web server, which services several virtual domains > 2. that web server receives a request for foo.com > 3. the web server switches into foo.com's accounting context > 4. the web server reads things from disk, allocates some memory, and > makes a database request. > 5. the database receives the request, and switches into foo.com's > accounting context, and charges foo.com for its resource use > etc... > > So, the goal is to run _one_ copy of an application on a system, but > account for its resources in a much more fine-grained way than at the > application level. > > I think we can probably use beancounters for this, if we do not worry > about migrating _existing_ charges when we change accounting context. > Does that make sense? > > -- Dave This is much better stated than I did. Thanks! -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-05T18:28:49-00:00

Re: [PATCH 11/13] BC: vmrss (preparations)
type: item
description: Kirill Korotaev wrote: <snip> > --- ./include/bc/beancounter.h.bcvmrssprep 2006-09-05 > 13:17:50.000000000 +0400 > +++ ./include/bc/beancounter.h 2006-09-05 13:44:33.000000000 +0400 > @@ -45,6 +45,13 @@ struct bc_resource_parm { > #define BC_MAXVALUE LONG_MAX > > /* > + * This magic is used to distinuish user beancounter and pages beancounter > + * in struct page. page_ub and page_bc are placed in union and MAGIC > + * ensures us that we don't use pbc as ubc in bc_page_uncharge(). > + */ > +#define BC_MAGIC 0x62756275UL > + > +/* > * Resource management structures > * Serialization issues: > * beancounter list management is protected via bc_hash_lock > @@ -54,11 +61,13 @@ struct bc_resource_parm { > */ > > struct beancounter { > + unsigned long bc_magic; > atomic_t bc_refcount; > spinlock_t bc_lock; > bcid_t bc_id; > struct hlist_node hash; > > + unsigned long unused_privvmpages; > /* resources statistics and settings */ > struct bc_resource_parm bc_parms[BC_RESOURCES]; > }; > @@ -74,6 +83,8 @@ enum bc_severity { BC_BARRIER, BC_LIMIT, > > #ifdef CONFIG_BEANCOUNTERS > > +extern unsigned int nr_beancounters = 1; > + my gcc doesn't like this one ... regards, C. Signed-off-by: Cedric Le Goater <clg@fr.ibm.com> --- include/bc/beancounter.h | 2 +- kernel/bc/beancounter.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) Index: 2.6.18-rc5-mm1/include/bc/beancounter.h ============================================================ ======= --- 2.6.18-rc5-mm1.orig/include/bc/beancounter.h +++ 2.6.18-rc5-mm1/include/bc/beancounter.h @@ -86,7 +86,7 @@ enum bc_severity { BC_BARRIER, BC_LIMIT, #ifdef CONFIG_BEANCOUNTERS -extern unsigned int nr_beancounters = 1; +extern unsigned int nr_beancounters; /* * These functions tune minheld and maxheld values for a given Index: 2.6.18-rc5-mm1/kernel/bc/beancounter.c ============================================================ ======= --- 2.6.18-rc5-mm1.orig/kernel/bc/beancounter.c +++ 2.6.18-rc5-mm1/kernel/bc/beancounter.c @@ -20,7 +20…
creator: Cedric Le Goater
date: 2006-09-05T22:09:41-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-05 at 10:46 -0700, Dave Hansen wrote: > On Tue, 2006-09-05 at 19:02 +0400, Kirill Korotaev wrote: > > Core Resource Beancounters (BC) + kernel/user memory control. > > > > BC allows to account and control consumption > > of kernel resources used by group of processes. > > Hi Kirill, > > I've honestly lost track of these discussions along the way, so I hope > you don't mind summarizing a bit. > > Do these patches help with accounting for anything other than memory? > Will we need new user/kernel interfaces for cpu, i/o bandwidth, etc...? > > Have you given any thought to the possibility that a task might need to > move between accounting contexts? That has certainly been a > "requirement" pushed on to CKRM for a long time, and the need goes > something like this: > > 1. A system runs a web server, which services several virtual domains > 2. that web server receives a request for foo.com > 3. the web server switches into foo.com's accounting context > 4. the web server reads things from disk, allocates some memory, and > makes a database request. > 5. the database receives the request, and switches into foo.com's > accounting context, and charges foo.com for its resource use > etc... > I'm wondering why not have different processes to serve different domains on the same physical server...particularly when they have different database to work on. Is the amount of memory that you save by having a single copy that much useful that you are even okay to serialize the whole operation (What would happen, while the request for foo.com is getting worked on, there is another request for foo_bar.com...does it need to wait for foo.com request to get done before it can be served). > So, the goal is to run _one_ copy of an application on a system, but > account for its resources in a much more fine-grained way than at the > application level. > What is that fine grained way. If not process based then can it be associated with file system location? -rohit
creator: Rohit Seth
date: 2006-09-06T00:17:58-00:00

Re: [PATCH 9/13] BC: locked pages (charge hooks)
type: item
description: Kirill Korotaev wrote: > Introduce calls to BC core over the kernel to charge locked memory. > > Normaly new locked piece of memory may appear in insert_vm_struct, > but there are places (do_mmap_pgoff, dup_mmap etc) when new vma > is not inserted by insert_vm_struct(), but either link_vma-ed or > merged with some other - these places call BC code explicitly. > > Plus sys_mlock[all] itself has to be patched to charge/uncharge > needed amount of pages. I still haven't heard your good reasons why such a complex scheme is required when my really simple proposal of unconditionally charging the page to the container it was allocated by. That has the benefit of not being full of user explotable holes and also not putting such a huge burden on mm/ and the wider kernel in general. -- Send instant messages to your online friends http://au.messenger.yahoo.com
creator: Nick Piggin
date: 2006-09-06T03:43:35-00:00

Re: [ckrm-tech] [PATCH 5/13] BC: user interface (syscalls)
type: item
description: Balbir Singh wrote: >> + >> +asmlinkage long sys_set_bcid(bcid_t id) >> +{ >> + int error; >> + struct beancounter *bc; >> + struct task_beancounter *task_bc; >> + >> + task_bc = &current->task_bc; > > I was playing around with the bc patches and found that to make > use of bc's, I had to actually call set_bcid() and then exec() a > task/shell so that the id would stick around. Would you consider That sounds very strange as sys_set_bcid() actually changes current's exec_bc. One note is about mm's bc - mm obtains new bc only after fork or exec - that's true. But kmemsize starts charging right after the sys_set_bcid. > changing sys_set_bcid to sys_set_task_bcid() or adding a new > system call sys_set_task_bcid()? We could pass the pid that we > intend to associate with the new id. This also means we'll need > locking around to protect task->task_bc.
creator: Pavel Emelianov
date: 2006-09-06T08:29:35-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Balbir Singh wrote: > Kirill Korotaev wrote: >> Core Resource Beancounters (BC) + kernel/user memory control. >> >> BC allows to account and control consumption >> of kernel resources used by group of processes. >> >> Draft UBC description on OpenVZ wiki can be found at >> http://wiki.openvz.org/UBC_parameters >> >> The full BC patch set allows to control: >> - kernel memory. All the kernel objects allocatable >> on user demand should be accounted and limited >> for DoS protection. >> E.g. page tables, task structs, vmas etc. > > One of the key requirements of resource management for us is to be > able to > migrate tasks across resource groups. Since bean counters do not > associate Then could you tell me please what to do with all the resources allocated by the task you are moving to another group? > a list of tasks associated with them, I do not see how this can be done > with the existing bean counters. > Associating a list of tasks with beancounter is not so hard actually. The question is wether this is usefull (regarding my previous comment).
creator: Pavel Emelianov
date: 2006-09-06T08:34:33-00:00

Re: [PATCH 9/13] BC: locked pages (charge hooks)
type: item
description: Nick Piggin wrote: > Kirill Korotaev wrote: > >> Introduce calls to BC core over the kernel to charge locked memory. >> >> Normaly new locked piece of memory may appear in insert_vm_struct, >> but there are places (do_mmap_pgoff, dup_mmap etc) when new vma >> is not inserted by insert_vm_struct(), but either link_vma-ed or >> merged with some other - these places call BC code explicitly. >> >> Plus sys_mlock[all] itself has to be patched to charge/uncharge >> needed amount of pages. > > > I still haven't heard your good reasons why such a complex scheme is > required when my really simple proposal of unconditionally charging > the page to the container it was allocated by. Charging the page to the container it was allocated in is a possible and correct way, we agree, but how does this comment refer to locked pages accounting? > > That has the benefit of not being full of user explotable holes and > also not putting such a huge burden on mm/ and the wider kernel in > general.
creator: Pavel Emelianov
date: 2006-09-06T08:45:01-00:00

Re: [ckrm-tech] [PATCH 5/13] BC: user interface (syscalls)
type: item
description: Pavel Emelianov wrote: > Balbir Singh wrote: >>> + >>> +asmlinkage long sys_set_bcid(bcid_t id) >>> +{ >>> + int error; >>> + struct beancounter *bc; >>> + struct task_beancounter *task_bc; >>> + >>> + task_bc = &current->task_bc; >> I was playing around with the bc patches and found that to make >> use of bc's, I had to actually call set_bcid() and then exec() a >> task/shell so that the id would stick around. Would you consider > That sounds very strange as sys_set_bcid() actually changes current's > exec_bc. > One note is about mm's bc - mm obtains new bc only after fork or exec - > that's > true. But kmemsize starts charging right after the sys_set_bcid. I was playing around only with kmemsize. I think the reason for my observation is this bash --> (my utility) --> set_bcid() Since bash spawns my utility in a separate process, it creates and assigns a bean counter to it and then my utility exits. Unless it spawns/exec()'s a new shell, the beancounter is freed when the task exits (my utility). >> changing sys_set_bcid to sys_set_task_bcid() or adding a new >> system call sys_set_task_bcid()? We could pass the pid that we >> intend to associate with the new id. This also means we'll need >> locking around to protect task->task_bc. > -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-06T08:57:38-00:00

Re: [PATCH 9/13] BC: locked pages (charge hooks)
type: item
description: Pavel Emelianov wrote: >Nick Piggin wrote: > >>Kirill Korotaev wrote: >> >> >>>Introduce calls to BC core over the kernel to charge locked memory. >>> >>>Normaly new locked piece of memory may appear in insert_vm_struct, >>>but there are places (do_mmap_pgoff, dup_mmap etc) when new vma >>>is not inserted by insert_vm_struct(), but either link_vma-ed or >>>merged with some other - these places call BC code explicitly. >>> >>>Plus sys_mlock[all] itself has to be patched to charge/uncharge >>>needed amount of pages. >>> >> >>I still haven't heard your good reasons why such a complex scheme is >>required when my really simple proposal of unconditionally charging >>the page to the container it was allocated by. >> >Charging the page to the container it was allocated in is a possible and >correct way, we agree, but how does this comment refer to locked pages > If it is a possible and correct way, I'd must rather see *that* way get tried first, and then made more complex or discarded if it is found to be insufficient. >accounting? > That's where I'd looked at enough mm/ stuff to decide that it wasn't just my usual unjustified whining. Complexity of this approach is quite... high. Sorry if that wasn't clear. -- Send instant messages to your online friends http://au.messenger.yahoo.com
creator: Nick Piggin
date: 2006-09-06T09:41:05-00:00

Re: [ckrm-tech] [PATCH 5/13] BC: user interface (syscalls)
type: item
description: Balbir Singh wrote: > Pavel Emelianov wrote: >> Balbir Singh wrote: >>>> + >>>> +asmlinkage long sys_set_bcid(bcid_t id) >>>> +{ >>>> + int error; >>>> + struct beancounter *bc; >>>> + struct task_beancounter *task_bc; >>>> + >>>> + task_bc = &current->task_bc; >>> I was playing around with the bc patches and found that to make >>> use of bc's, I had to actually call set_bcid() and then exec() a >>> task/shell so that the id would stick around. Would you consider >> That sounds very strange as sys_set_bcid() actually changes current's >> exec_bc. >> One note is about mm's bc - mm obtains new bc only after fork or exec - >> that's >> true. But kmemsize starts charging right after the sys_set_bcid. > > I was playing around only with kmemsize. I think the reason for my > observation > is this > > bash --> (my utility) --> set_bcid() > > Since bash spawns my utility in a separate process, it creates and > assigns > a bean counter to it and then my utility exits. Unless it > spawns/exec()'s a > new shell, the beancounter is freed when the task exits (my utility). Well, beancounter is not "inherited" by parent task :) After setting bcid you need to spawn/exec a new shell. But seeting limits and getting stats is possible from the old shell as well as from the new one. > >>> changing sys_set_bcid to sys_set_task_bcid() or adding a new >>> system call sys_set_task_bcid()? We could pass the pid that we >>> intend to associate with the new id. This also means we'll need >>> locking around to protect task->task_bc. >> > >
creator: Pavel Emelianov
date: 2006-09-06T10:42:48-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Balbir Singh wrote: > Kirill Korotaev wrote: > >> Core Resource Beancounters (BC) + kernel/user memory control. >> >> BC allows to account and control consumption >> of kernel resources used by group of processes. >> >> Draft UBC description on OpenVZ wiki can be found at >> http://wiki.openvz.org/UBC_parameters >> >> The full BC patch set allows to control: >> - kernel memory. All the kernel objects allocatable >> on user demand should be accounted and limited >> for DoS protection. >> E.g. page tables, task structs, vmas etc. > > > One of the key requirements of resource management for us is to be able to > migrate tasks across resource groups. Since bean counters do not associate > a list of tasks associated with them, I do not see how this can be done > with the existing bean counters. It was discussed multiple times already. The key problem here is the objects which do not _belong_ to tasks. e.g. IPC objects. They exist in global namespace and can't be reaccounted. At least no one proposed the policy to reaccount. And please note, IPCs are not the only such objects. But I guess your comment mostly concerns user pages, yeah? In this case reaccounting can be easily done using page beancounters which are introduced in this patch set. So if it is a requirement, then lets cooperate and create such functionality. So for now I see 2 main requirements from people: - memory reclamation - tasks moving across beancounters I agree with these requirements and lets move into this direction. But moving so far can't be done without accepting: 1. core functionality 2. accounting Thanks, Kirill
creator: dev
date: 2006-09-06T13:04:20-00:00

Re: [ckrm-tech] [PATCH 5/13] BC: user interface (syscalls)
type: item
description: Pavel Emelianov wrote: > Balbir Singh wrote: >> Pavel Emelianov wrote: >>> Balbir Singh wrote: >>>>> + >>>>> +asmlinkage long sys_set_bcid(bcid_t id) >>>>> +{ >>>>> + int error; >>>>> + struct beancounter *bc; >>>>> + struct task_beancounter *task_bc; >>>>> + >>>>> + task_bc = &current->task_bc; >>>> I was playing around with the bc patches and found that to make >>>> use of bc's, I had to actually call set_bcid() and then exec() a >>>> task/shell so that the id would stick around. Would you consider >>> That sounds very strange as sys_set_bcid() actually changes current's >>> exec_bc. >>> One note is about mm's bc - mm obtains new bc only after fork or exec - >>> that's >>> true. But kmemsize starts charging right after the sys_set_bcid. >> I was playing around only with kmemsize. I think the reason for my >> observation >> is this >> >> bash --> (my utility) --> set_bcid() >> >> Since bash spawns my utility in a separate process, it creates and >> assigns >> a bean counter to it and then my utility exits. Unless it >> spawns/exec()'s a >> new shell, the beancounter is freed when the task exits (my utility). > Well, beancounter is not "inherited" by parent task :) > After setting bcid you need to spawn/exec a new shell. > But seeting limits and getting stats is possible from the old shell > as well as from the new one. That's what I suspected. I suggest changing the system call to allow adding any task to a particular id (not necessarily only the current one). It would help us group tasks to a particular id. It would also solve my problem of spawning a shell each time I decide to use a task with a beancounter and limits. >>>> changing sys_set_bcid to sys_set_task_bcid() or adding a new >>>> system call sys_set_task_bcid()? We could pass the pid that we >>>> intend to associate with the new id. This also means we'll need >>>> locking around to protect task->task_bc. >> -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-06T13:23:27-00:00

Re: [ckrm-tech] [PATCH 5/13] BC: user interface (syscalls)
type: item
description: Kirill Korotaev wrote: > Add the following system calls for BC management: > 1. sys_get_bcid - get current BC id > 2. sys_set_bcid - change exec_ and fork_ BCs on current > 3. sys_set_bclimit - set limits for resources consumtions > 4. sys_get_bcstat - return br_resource_parm on resource > > Signed-off-by: Pavel Emelianov <xemul@sw.ru> > Signed-off-by: Kirill Korotaev <dev@sw.ru> > > --- ./include/asm-powerpc/systbl.h.bcsys 2006-07-10 12:39:19.000000000 +0400 > +++ ./include/asm-powerpc/systbl.h 2006-09-05 12:47:21.000000000 +0400 > @@ -304,3 +304,7 @@ SYSCALL_SPU(fchmodat) > SYSCALL_SPU(faccessat) > COMPAT_SYS_SPU(get_robust_list) > COMPAT_SYS_SPU(set_robust_list) > +SYSCALL(sys_get_bcid) > +SYSCALL(sys_set_bcid) > +SYSCALL(sys_set_bclimit) > +SYSCALL(sys_get_bcstat) Fix a build error for powerpc boxes. While compiling on powerpc, Vaidyanathan Srinivasan caught this error. System calls on powerpc do not need sys_ prefix. Signed-off-by: Balbir Singh <balbir@in.ibm.com> Signed-off-by: Vaidyanathan Srinivasan <svaidy@in.ibm.com> --- include/asm-powerpc/systbl.h | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff -puN include/asm-powerpc/systbl.h~fix-powerpc-build include/asm-powerpc/systbl.h --- linux-2.6.18-rc5/include/asm-powerpc/systbl.h~fix-powerpc-bu ild 2006-09-06 19:03:18.000000000 +0530 +++ linux-2.6.18-rc5-balbir/include/asm-powerpc/systbl.h 2006-09-06 19:03:38.000000000 +0530 @@ -304,7 +304,7 @@ SYSCALL_SPU(fchmodat) SYSCALL_SPU(faccessat) COMPAT_SYS_SPU(get_robust_list) COMPAT_SYS_SPU(set_robust_list) -SYSCALL(sys_get_bcid) -SYSCALL(sys_set_bcid) -SYSCALL(sys_set_bclimit) -SYSCALL(sys_get_bcstat) +SYSCALL(get_bcid) +SYSCALL(set_bcid) +SYSCALL(set_bclimit) +SYSCALL(get_bcstat) _ -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-06T13:45:44-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: > On Tue, 2006-09-05 at 19:02 +0400, Kirill Korotaev wrote: > >>Core Resource Beancounters (BC) + kernel/user memory control. >> >>BC allows to account and control consumption >>of kernel resources used by group of processes. > > > Hi Kirill, > > I've honestly lost track of these discussions along the way, so I hope > you don't mind summarizing a bit. I think we need to create wiki to summarize it once and forever. http://wiki.openvz.org/UBC_discussion > Do these patches help with accounting for anything other than memory? this patch set - no, but the complete one - does: * numfile * numptys * numsocks (TCP, other, etc.) * numtasks * numflocks ... this list of resources was chosen to make sure that no DoS from the container is possible. This list is extensible easily and if resource is out of interest than its limits can be set to unlimited. > Will we need new user/kernel interfaces for cpu, i/o bandwidth, etc...? no. no new interfaces are required. BUT: I remind you the talks at OKS/OLS and in previous UBC discussions. It was noted that having a separate interfaces for CPU, I/O bandwidth and memory maybe worthwhile. BTW, I/O bandwidth already has a separate interface :/ > Have you given any thought to the possibility that a task might need to > move between accounting contexts? That has certainly been a > "requirement" pushed on to CKRM for a long time, and the need goes > something like this: Yes we thought about this and this is no more problematic for BC than for CKRM. See my explanation below. > 1. A system runs a web server, which services several virtual domains > 2. that web server receives a request for foo.com > 3. the web server switches into foo.com's accounting context > 4. the web server reads things from disk, allocates some memory, and > makes a database request. > 5. the database receives the request, and switches into foo.com's > accounting context, and charges foo.com for its resource use > etc... The question is - whether web server is multithrea…
creator: dev
date: 2006-09-06T13:54:13-00:00

Re: [PATCH 11/13] BC: vmrss (preparations)
type: item
description: Thanks a lot!!! > Kirill Korotaev wrote: > > <snip> > >>--- ./include/bc/beancounter.h.bcvmrssprep 2006-09-05 >>13:17:50.000000000 +0400 >>+++ ./include/bc/beancounter.h 2006-09-05 13:44:33.000000000 +0400 >>@@ -45,6 +45,13 @@ struct bc_resource_parm { >>#define BC_MAXVALUE LONG_MAX >> >>/* >>+ * This magic is used to distinuish user beancounter and pages beancounter >>+ * in struct page. page_ub and page_bc are placed in union and MAGIC >>+ * ensures us that we don't use pbc as ubc in bc_page_uncharge(). >>+ */ >>+#define BC_MAGIC 0x62756275UL >>+ >>+/* >> * Resource management structures >> * Serialization issues: >> * beancounter list management is protected via bc_hash_lock >>@@ -54,11 +61,13 @@ struct bc_resource_parm { >> */ >> >>struct beancounter { >>+ unsigned long bc_magic; >> atomic_t bc_refcount; >> spinlock_t bc_lock; >> bcid_t bc_id; >> struct hlist_node hash; >> >>+ unsigned long unused_privvmpages; >> /* resources statistics and settings */ >> struct bc_resource_parm bc_parms[BC_RESOURCES]; >>}; >>@@ -74,6 +83,8 @@ enum bc_severity { BC_BARRIER, BC_LIMIT, >> >>#ifdef CONFIG_BEANCOUNTERS >> >>+extern unsigned int nr_beancounters = 1; >>+ > > > my gcc doesn't like this one ... > > regards, > > C. > > Signed-off-by: Cedric Le Goater <clg@fr.ibm.com> > > --- > include/bc/beancounter.h | 2 +- > kernel/bc/beancounter.c | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > Index: 2.6.18-rc5-mm1/include/bc/beancounter.h > ============================================================ ======= > --- 2.6.18-rc5-mm1.orig/include/bc/beancounter.h > +++ 2.6.18-rc5-mm1/include/bc/beancounter.h > @@ -86,7 +86,7 @@ enum bc_severity { BC_BARRIER, BC_LIMIT, > > #ifdef CONFIG_BEANCOUNTERS > > -extern unsigned int nr_beancounters = 1; > +extern unsigned int nr_beancounters; > > /* > * These functions tune minheld and maxheld values for a given > Index: 2.6.18-rc5-mm1/kernel/bc/beancounter.c > ============================================================ ======= > --…
creator: dev
date: 2006-09-06T13:56:03-00:00

Re: [ckrm-tech] [PATCH 9/13] BC: locked pages (charge hooks)
type: item
description: Nick, > Kirill Korotaev wrote: > > >>Introduce calls to BC core over the kernel to charge locked memory. >> >>Normaly new locked piece of memory may appear in insert_vm_struct, >>but there are places (do_mmap_pgoff, dup_mmap etc) when new vma >>is not inserted by insert_vm_struct(), but either link_vma-ed or >>merged with some other - these places call BC code explicitly. >> >>Plus sys_mlock[all] itself has to be patched to charge/uncharge >>needed amount of pages. > > > > I still haven't heard your good reasons why such a complex scheme is > required when my really simple proposal of unconditionally charging > the page to the container it was allocated by. Nick can you elaborate what your proposal is? Probably I missed it somewhere... > That has the benefit of not being full of user explotable holes and > also not putting such a huge burden on mm/ and the wider kernel in > general. I guess you will have to account locked pages still and thus complexity won't be reduced much in this regard... Thanks, Kirill
creator: dev
date: 2006-09-06T14:16:42-00:00

Re: [PATCH 7/13] BC: kernel memory (marks)
type: item
description: Minor issue bellow in arch/ia64/mm/init.c. I'm not sure what the charge argument should be. Please check. Regards, C. Signed-off-by: Cedric Le Goater <clg@fr.ibm.com> --- arch/ia64/mm/init.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Index: 2.6.18-rc5-mm1/arch/ia64/mm/init.c ============================================================ ======= --- 2.6.18-rc5-mm1.orig/arch/ia64/mm/init.c +++ 2.6.18-rc5-mm1/arch/ia64/mm/init.c @@ -95,7 +95,7 @@ check_pgt_cache(void) preempt_disable(); while (unlikely((pages_to_free = min_pages_to_free()) > 0)) { while (pages_to_free--) { - free_page((unsigned long)pgtable_quicklist_alloc()); + free_page((unsigned long)pgtable_quicklist_alloc(0)); } preempt_enable(); preempt_disable();
creator: Cedric Le Goater
date: 2006-09-06T14:19:15-00:00

Re: [ckrm-tech] [PATCH 5/13] BC: user interface (syscalls)
type: item
description: thanks a lot! > Kirill Korotaev wrote: > >>Add the following system calls for BC management: >> 1. sys_get_bcid - get current BC id >> 2. sys_set_bcid - change exec_ and fork_ BCs on current >> 3. sys_set_bclimit - set limits for resources consumtions >> 4. sys_get_bcstat - return br_resource_parm on resource >> >>Signed-off-by: Pavel Emelianov <xemul@sw.ru> >>Signed-off-by: Kirill Korotaev <dev@sw.ru> >> >>--- ./include/asm-powerpc/systbl.h.bcsys 2006-07-10 12:39:19.000000000 +0400 >>+++ ./include/asm-powerpc/systbl.h 2006-09-05 12:47:21.000000000 +0400 >>@@ -304,3 +304,7 @@ SYSCALL_SPU(fchmodat) >> SYSCALL_SPU(faccessat) >> COMPAT_SYS_SPU(get_robust_list) >> COMPAT_SYS_SPU(set_robust_list) >>+SYSCALL(sys_get_bcid) >>+SYSCALL(sys_set_bcid) >>+SYSCALL(sys_set_bclimit) >>+SYSCALL(sys_get_bcstat) > > > > Fix a build error for powerpc boxes. While compiling on powerpc, Vaidyanathan > Srinivasan caught this error. System calls on powerpc do not need sys_ prefix. > > Signed-off-by: Balbir Singh <balbir@in.ibm.com> > Signed-off-by: Vaidyanathan Srinivasan <svaidy@in.ibm.com> > --- > > include/asm-powerpc/systbl.h | 8 ++++---- > 1 files changed, 4 insertions(+), 4 deletions(-) > > diff -puN include/asm-powerpc/systbl.h~fix-powerpc-build > include/asm-powerpc/systbl.h > --- linux-2.6.18-rc5/include/asm-powerpc/systbl.h~fix-powerpc-bu ild 2006-09-06 > 19:03:18.000000000 +0530 > +++ linux-2.6.18-rc5-balbir/include/asm-powerpc/systbl.h 2006-09-06 > 19:03:38.000000000 +0530 > @@ -304,7 +304,7 @@ SYSCALL_SPU(fchmodat) > SYSCALL_SPU(faccessat) > COMPAT_SYS_SPU(get_robust_list) > COMPAT_SYS_SPU(set_robust_list) > -SYSCALL(sys_get_bcid) > -SYSCALL(sys_set_bcid) > -SYSCALL(sys_set_bclimit) > -SYSCALL(sys_get_bcstat) > +SYSCALL(get_bcid) > +SYSCALL(set_bcid) > +SYSCALL(set_bclimit) > +SYSCALL(get_bcstat) > _ >
creator: dev
date: 2006-09-06T14:20:34-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Kirill Korotaev wrote: > Balbir Singh wrote: >> Kirill Korotaev wrote: >> >>> Core Resource Beancounters (BC) + kernel/user memory control. >>> >>> BC allows to account and control consumption >>> of kernel resources used by group of processes. >>> >>> Draft UBC description on OpenVZ wiki can be found at >>> http://wiki.openvz.org/UBC_parameters >>> >>> The full BC patch set allows to control: >>> - kernel memory. All the kernel objects allocatable >>> on user demand should be accounted and limited >>> for DoS protection. >>> E.g. page tables, task structs, vmas etc. >> >> One of the key requirements of resource management for us is to be able to >> migrate tasks across resource groups. Since bean counters do not associate >> a list of tasks associated with them, I do not see how this can be done >> with the existing bean counters. > It was discussed multiple times already. > The key problem here is the objects which do not _belong_ to tasks. > e.g. IPC objects. They exist in global namespace and can't be reaccounted. > At least no one proposed the policy to reaccount. > And please note, IPCs are not the only such objects. > > But I guess your comment mostly concerns user pages, yeah? Yes. > In this case reaccounting can be easily done using page beancounters > which are introduced in this patch set. > So if it is a requirement, then lets cooperate and create such functionality. > Sure, let's cooperate and talk. > So for now I see 2 main requirements from people: > - memory reclamation > - tasks moving across beancounters > Some not quite so urgent ones - like support for guarantees. I think this can be worked out as we make progress. > I agree with these requirements and lets move into this direction. > But moving so far can't be done without accepting: > 1. core functionality > 2. accounting > Some of the core functionality might be a limiting factor for the requirements. Lets agree on the requirements, I think its a great step forward and then build the core func…
creator: Balbir Singh
date: 2006-09-06T19:17:56-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Wed, 2006-09-06 at 17:06 +0400, Kirill Korotaev wrote: > Balbir Singh wrote: > > Kirill Korotaev wrote: > > > >> Core Resource Beancounters (BC) + kernel/user memory control. > >> > >> BC allows to account and control consumption > >> of kernel resources used by group of processes. > >> > >> Draft UBC description on OpenVZ wiki can be found at > >> http://wiki.openvz.org/UBC_parameters > >> > >> The full BC patch set allows to control: > >> - kernel memory. All the kernel objects allocatable > >> on user demand should be accounted and limited > >> for DoS protection. > >> E.g. page tables, task structs, vmas etc. > > > > > > One of the key requirements of resource management for us is to be able to > > migrate tasks across resource groups. Since bean counters do not associate > > a list of tasks associated with them, I do not see how this can be done > > with the existing bean counters. > It was discussed multiple times already. > The key problem here is the objects which do not _belong_ to tasks. > e.g. IPC objects. They exist in global namespace and can't be reaccounted. > At least no one proposed the policy to reaccount. > And please note, IPCs are not the only such objects. >From implementation point of view I do not see it to be any different than how it can be done under UBC. AFAICS, beancounters are associated with tasks not those "objects". Those "objects" get their bc through some association with a task. The same can be done in the other case also. If my understanding is wrong, please tell me how one can associate such "object" to a bc. > > But I guess your comment mostly concerns user pages, yeah? > In this case reaccounting can be easily done using page beancounters > which are introduced in this patch set. > So if it is a requirement, then lets cooperate and create such functionality. hmm... that is what I thought I was doing when I was replying on these threads. May be I should have waited for this "call for co-operation" before jumping on it :) > >…
creator: Chandra Seetharaman
date: 2006-09-06T21:47:34-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Wed, 2006-09-06 at 17:57 +0400, Kirill Korotaev wrote: > > On Tue, 2006-09-05 at 19:02 +0400, Kirill Korotaev wrote: > > > >>Core Resource Beancounters (BC) + kernel/user memory control. > >> > >>BC allows to account and control consumption > >>of kernel resources used by group of processes. > > > > > > Hi Kirill, > > > > I've honestly lost track of these discussions along the way, so I hope > > you don't mind summarizing a bit. > I think we need to create wiki to summarize it once and forever. > http://wiki.openvz.org/UBC_discussion > > > Do these patches help with accounting for anything other than memory? > this patch set - no, but the complete one - does: > * numfile > * numptys > * numsocks (TCP, other, etc.) > * numtasks > * numflocks > ... > this list of resources was chosen to make sure that no DoS from the container > is possible. > This list is extensible easily and if resource is out of interest than > its limits can be set to unlimited. > > > Will we need new user/kernel interfaces for cpu, i/o bandwidth, etc...? > no. no new interfaces are required. Good to know that. Your CPU controller supports guarantee ? Do you have a i/o controller ? > > BUT: I remind you the talks at OKS/OLS and in previous UBC discussions. > It was noted that having a separate interfaces for CPU, I/O bandwidth But, it will be lot simpler for the user to configure/use if they are together. We should discuss this also. > and memory maybe worthwhile. BTW, I/O bandwidth already has a separate > interface :/ > > > Have you given any thought to the possibility that a task might need to > > move between accounting contexts? That has certainly been a > > "requirement" pushed on to CKRM for a long time, and the need goes > > something like this: > Yes we thought about this and this is no more problematic for BC > than for CKRM. See my explanation below. > > > 1. A system runs a web server, which services several virtual domains > > 2. that web server receives a request for foo.com > > …
creator: Chandra Seetharaman
date: 2006-09-06T21:54:01-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Thu, 2006-09-07 at 00:47 +0530, Balbir Singh wrote: <snip> > > Some not quite so urgent ones - like support for guarantees. I think this can IMO, guarantee support should be considered to be part of the infrastructure. Controller functionalities/implementation will be different with/without guarantee support. In other words, adding guarantee feature later will cause re-implementations. > be worked out as we make progress. > > > I agree with these requirements and lets move into this direction. > > But moving so far can't be done without accepting: > > 1. core functionality > > 2. accounting > > > > Some of the core functionality might be a limiting factor for the requirements. > Lets agree on the requirements, I think its a great step forward and then > build the core functionality with these requirements in mind. > > > Thanks, > > Kirill > > -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-06T22:06:11-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Chandra Seetharaman wrote: > On Thu, 2006-09-07 at 00:47 +0530, Balbir Singh wrote: > > <snip> >> Some not quite so urgent ones - like support for guarantees. I think this can > > IMO, guarantee support should be considered to be part of the > infrastructure. Controller functionalities/implementation will be > different with/without guarantee support. In other words, adding > guarantee feature later will cause re-implementations. Thanks for pointing this out. Thats what I implied in the comment below. > >> be worked out as we make progress. >> >>> I agree with these requirements and lets move into this direction. >>> But moving so far can't be done without accepting: >>> 1. core functionality >>> 2. accounting >>> >> Some of the core functionality might be a limiting factor for the requirements. >> Lets agree on the requirements, I think its a great step forward and then >> build the core functionality with these requirements in mind. >> >>> Thanks, >>> Kirill >>> -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-07T03:08:48-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Chandra Seetharaman wrote: [snip] >>> Will we need new user/kernel interfaces for cpu, i/o bandwidth, etc...? >>> >> no. no new interfaces are required. >> > > Good to know that. > > Your CPU controller supports guarantee ? > It does, but CPU controller is not so simple as memory one. > Do you have a i/o controller ? > > >> BUT: I remind you the talks at OKS/OLS and in previous UBC discussions. >> It was noted that having a separate interfaces for CPU, I/O bandwidth >> > > But, it will be lot simpler for the user to configure/use if they are > together. We should discuss this also. > IMHO such unification may only imply that one syscall is used to pass configuration info into kernel. Each controller has specific configurating parameters different from the other ones. E.g. CPU controller must assign a "weight" to each group to share CPU time accordingly, but what is a "weight" for memory controller? IO may operate on "bandwidth" and it's not clear what is a "bandwidth" in Kb/sec for CPU controller and so on. [snip] >> The question is - whether web server is multithreaded or not... >> If it is not - then no problem here, you can change current >> context and new resources will be charged accordingly. >> >> And current BC code is _able_ to handle it with _minor_ changes. >> (One just need to save bc not on mm struct, but rather on vma struct >> and change mm->bc on set_bc_id()). >> >> However, no one (can some one from CKRM team please?) explained so far >> what to do with threads. Consider the following example. >> >> 1. Threaded web server spawns a child to serve a client. >> 2. child thread touches some pages and they are charged to child BC >> (which differs from parent's one) >> 3. child exits, but since its mm is shared with parent, these pages >> stay mapped and charged to child BC. >> >> So the question is: what to do with these pages? >> - should we recharge them to another BC? >> - leave them charged? >> > > Leave them charged. It will be charged to the appro…
creator: Pavel Emelianov
date: 2006-09-07T07:29:17-00:00

Re: [ckrm-tech] [PATCH 11/13] BC: vmrss (preparations)
type: item
description: Kirill Korotaev wrote: > This patch does simple things: > - intruduces an bc_magic field on beancunter to make sure > union on struct page is correctly used in next patches > - adds nr_beancounters > - adds unused_privvmpages variable (counter of privvm pages > which are not mapped into VM address space and thus potentially > can be allocated later) > > +static inline void privvm_uncharge(struct beancounter *bc, unsigned long sz) > +{ > + if (unlikely(bc->unused_privvmpages < sz)) { > + printk("BC: overuncharging %d unused pages: val %lu held %lu\n", > + bc->bc_id, sz, bc->unused_privvmpages); I hit this path, when I do not enable CONFIG_BEANCOUNTERS_RSS. I suspect it has something to do with the code in mod_rss_pages(). I suspect the that CONFIG_BEANCOUNTERS_RSS needs to be enabled to get the accounting right. In addition, Could you please make this a warning with KERN_WARNING. > + sz = bc->unused_privvmpages; > + } > + bc->unused_privvmpages -= sz; > + bc_update_privvmpages(bc); > +} > + -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-07T16:28:45-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Thu, 2006-09-07 at 11:29 +0400, Pavel Emelianov wrote: > Chandra Seetharaman wrote: > > [snip] > >>> Will we need new user/kernel interfaces for cpu, i/o bandwidth, etc...? > >>> > >> no. no new interfaces are required. > >> > > > > Good to know that. > > > > Your CPU controller supports guarantee ? > > > It does, but CPU controller is not so simple as memory one. Hmm... the reason I asked is that the UBC infrastructure doesn't provide guarantee support and Kirill mentioned there is no changes required to UBC if you have to move your CPU controller to be under UBC. >From your reply it does look like you need to make some changes (add guarantee support) to UBC, if you want to move the CPU controller to be under UBC. > > Do you have a i/o controller ? > > > > > >> BUT: I remind you the talks at OKS/OLS and in previous UBC discussions. > >> It was noted that having a separate interfaces for CPU, I/O bandwidth > >> > > > > But, it will be lot simpler for the user to configure/use if they are > > together. We should discuss this also. > > > IMHO such unification may only imply that one syscall is used to pass > configuration info into kernel. > Each controller has specific configurating parameters different from the > other ones. E.g. CPU controller must assign a "weight" to each group to > share CPU time accordingly, but what is a "weight" for memory controller? > IO may operate on "bandwidth" and it's not clear what is a "bandwidth" in > Kb/sec for CPU controller and so on. > > [snip] > >> The question is - whether web server is multithreaded or not... > >> If it is not - then no problem here, you can change current > >> context and new resources will be charged accordingly. > >> > >> And current BC code is _able_ to handle it with _minor_ changes. > >> (One just need to save bc not on mm struct, but rather on vma struct > >> and change mm->bc on set_bc_id()). > >> > >> However, no one (can some one from CKRM team please?) explained so far > >> what to do with threa…
creator: Chandra Seetharaman
date: 2006-09-07T19:16:56-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Thu, 2006-09-07 at 11:29 +0400, Pavel Emelianov wrote: <snip> > >> BUT: I remind you the talks at OKS/OLS and in previous UBC discussions. > >> It was noted that having a separate interfaces for CPU, I/O bandwidth > >> > > > > But, it will be lot simpler for the user to configure/use if they are > > together. We should discuss this also. > > > IMHO such unification may only imply that one syscall is used to pass > configuration info into kernel. > Each controller has specific configurating parameters different from the > other ones. E.g. CPU controller must assign a "weight" to each group to > share CPU time accordingly, but what is a "weight" for memory controller? > IO may operate on "bandwidth" and it's not clear what is a "bandwidth" in > Kb/sec for CPU controller and so on. CKRM/RG handles this by eliminating the units from the interface and abstracting them to be "shares". Each resource controller converts the shares to its own units and handles properly. User can specify the quantities simply as a percentage. CPU controller would treat it as cycles/ticks (within a time), memory controller would treat it as number of pages, and I/O controller would treat it as bandwidth, and so on... <snip> -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-07T19:29:15-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Chandra Seetharaman wrote: [snip] >>>> The question is - whether web server is multithreaded or not... >>>> If it is not - then no problem here, you can change current >>>> context and new resources will be charged accordingly. >>>> >>>> And current BC code is _able_ to handle it with _minor_ changes. >>>> (One just need to save bc not on mm struct, but rather on vma struct >>>> and change mm->bc on set_bc_id()). >>>> >>>> However, no one (can some one from CKRM team please?) explained so far >>>> what to do with threads. Consider the following example. >>>> >>>> 1. Threaded web server spawns a child to serve a client. >>>> 2. child thread touches some pages and they are charged to child BC >>>> (which differs from parent's one) >>>> 3. child exits, but since its mm is shared with parent, these pages >>>> stay mapped and charged to child BC. >>>> >>>> So the question is: what to do with these pages? >>>> - should we recharge them to another BC? >>>> - leave them charged? >>>> >>>> >>> Leave them charged. It will be charged to the appropriate UBC when they >>> touch it again. >>> >>> >> Do you mean that page must be re-charged each time someone touches it? >> > > What I meant is that to leave them charged, and if when they are > ummapped and mapped later, charge it to the appropriate BC. > In this case multithreaded apache that tries to serve each domain in separate BC will fill the memory with BC-s, held by pages allocated and mapped in threads.
creator: Pavel Emelianov
date: 2006-09-08T07:22:47-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Chandra Seetharaman wrote: > On Thu, 2006-09-07 at 11:29 +0400, Pavel Emelianov wrote: > <snip> > > >>>> BUT: I remind you the talks at OKS/OLS and in previous UBC discussions. >>>> It was noted that having a separate interfaces for CPU, I/O bandwidth >>>> >>>> >>> But, it will be lot simpler for the user to configure/use if they are >>> together. We should discuss this also. >>> >>> >> IMHO such unification may only imply that one syscall is used to pass >> configuration info into kernel. >> Each controller has specific configurating parameters different from the >> other ones. E.g. CPU controller must assign a "weight" to each group to >> share CPU time accordingly, but what is a "weight" for memory controller? >> IO may operate on "bandwidth" and it's not clear what is a "bandwidth" in >> Kb/sec for CPU controller and so on. >> > > CKRM/RG handles this by eliminating the units from the interface and > abstracting them to be "shares". Each resource controller converts the > shares to its own units and handles properly. > That's what I'm talking about - common syscall/ioct/etc and each controller parses its input itself. That's OK for us. [snip]
creator: Pavel Emelianov
date: 2006-09-08T07:26:34-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Chandra Seetharaman wrote: > On Thu, 2006-09-07 at 00:47 +0530, Balbir Singh wrote: > > <snip> >> Some not quite so urgent ones - like support for guarantees. I think >> this can > > IMO, guarantee support should be considered to be part of the > infrastructure. Controller functionalities/implementation will be > different with/without guarantee support. In other words, adding > guarantee feature later will cause re-implementations. I'm afraid we have different understandings of what a "guarantee" is. Don't we? Guarantee may be one of 1. container will be able to touch that number of pages 2. container will be able to sys_mmap() that number of pages 3. container will not be killed unless it touches that number of pages 4. anything else Let's decide what kind of a guarantee we want. >> be worked out as we make progress. >> >>> I agree with these requirements and lets move into this direction. >>> But moving so far can't be done without accepting: >>> 1. core functionality >>> 2. accounting >>> >> Some of the core functionality might be a limiting factor for the requirements. >> Lets agree on the requirements, I think its a great step forward and then >> build the core functionality with these requirements in mind. >> >>> Thanks, >>> Kirill >>>
creator: Pavel Emelianov
date: 2006-09-08T07:33:00-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-05 at 17:17 -0700, Rohit Seth wrote: > I'm wondering why not have different processes to serve different > domains on the same physical server...particularly when they have > different database to work on. This is largely because this is I think how it is done today, and it has a lot of disadvantages. They also want to be able to account for traffic on the same database. Think of a large web hosting environment where you charged everyone (hundreds or thousands of users) by CPU and I/O bandwidth used at all levels of a given transaction. > Is the amount of memory that you save by > having a single copy that much useful that you are even okay to > serialize the whole operation (What would happen, while the request for > foo.com is getting worked on, there is another request for > foo_bar.com...does it need to wait for foo.com request to get done > before it can be served). Let's put it this way. Enterprise databases can be memory pigs. It isn't feasible to run hundreds or thousands of copies on each machine. -- Dave
creator: Dave Hansen
date: 2006-09-08T15:30:50-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Wed, 2006-09-06 at 17:06 +0400, Kirill Korotaev wrote: > It was discussed multiple times already. > The key problem here is the objects which do not _belong_ to tasks. Heh. The original CKRM patches didn't have a strong binding to tasks. They took it away to make them more mergeable. ;) -- Dave
creator: Dave Hansen
date: 2006-09-08T15:33:53-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: > I'm afraid we have different understandings of what a "guarantee" is. It appears so. > Don't we? > Guarantee may be one of > > 1. container will be able to touch that number of pages > 2. container will be able to sys_mmap() that number of pages > 3. container will not be killed unless it touches that number of pages A "death sentence" guarantee? I like it. :) > 4. anything else > > Let's decide what kind of a guarantee we want. I think of it as: "I will be allowed to use this many total pages, and they are guaranteed not to fail." (1), I think. The sum of all of the system's guarantees must be less than or equal to the amount of free memory on the machine. If we knew to which NUMA node the memory was going to go, we might as well take the pages out of the allocator. -- Dave
creator: Dave Hansen
date: 2006-09-08T15:43:41-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Fri, 2006-09-08 at 08:30 -0700, Dave Hansen wrote: > On Tue, 2006-09-05 at 17:17 -0700, Rohit Seth wrote: > > I'm wondering why not have different processes to serve different > > domains on the same physical server...particularly when they have > > different database to work on. > > This is largely because this is I think how it is done today, and it has > a lot of disadvantages. If it has lot of disadvantages then we should try to avoid that mechanism. Though I think it is okay to allow processes to be moved around with the clear expectation that it is a very heavy operation (as I think at least all the anon pages should be moved too along with task) and should not be generally done. > They also want to be able to account for > traffic on the same database. Think of a large web hosting environment > where you charged everyone (hundreds or thousands of users) by CPU and > I/O bandwidth used at all levels of a given transaction. > > > Is the amount of memory that you save by > > having a single copy that much useful that you are even okay to > > serialize the whole operation (What would happen, while the request for > > foo.com is getting worked on, there is another request for > > foo_bar.com...does it need to wait for foo.com request to get done > > before it can be served). > > Let's put it this way. Enterprise databases can be memory pigs. It > isn't feasible to run hundreds or thousands of copies on each machine. > The extra cost is probably the stack and private data segment...yes there could be trade offs there depending on how big these segments are. Though if there are big shared segments then that can be charged to a single container. Thanks, -rohit
creator: Rohit Seth
date: 2006-09-08T17:10:37-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Rohit Seth wrote: > On Fri, 2006-09-08 at 08:30 -0700, Dave Hansen wrote: >> On Tue, 2006-09-05 at 17:17 -0700, Rohit Seth wrote: >>> I'm wondering why not have different processes to serve different >>> domains on the same physical server...particularly when they have >>> different database to work on. >> This is largely because this is I think how it is done today, and it has >> a lot of disadvantages. > > If it has lot of disadvantages then we should try to avoid that > mechanism. Though I think it is okay to allow processes to be moved > around with the clear expectation that it is a very heavy operation (as > I think at least all the anon pages should be moved too along with task) > and should not be generally done. > >> They also want to be able to account for >> traffic on the same database. Think of a large web hosting environment >> where you charged everyone (hundreds or thousands of users) by CPU and >> I/O bandwidth used at all levels of a given transaction. >> >>> Is the amount of memory that you save by >>> having a single copy that much useful that you are even okay to >>> serialize the whole operation (What would happen, while the request for >>> foo.com is getting worked on, there is another request for >>> foo_bar.com...does it need to wait for foo.com request to get done >>> before it can be served). >> Let's put it this way. Enterprise databases can be memory pigs. It >> isn't feasible to run hundreds or thousands of copies on each machine. >> > > > The extra cost is probably the stack and private data segment... Also maintenability, licensing, blah, blah. Replicating the software stack for each service level one wishes to provide, if avoidable as it seems to be, isn't such a good idea. Same sort of reasoning for why containers make sense compared to Xen/VMWare instances. Memory resources, by their very nature, will be tougher to account when a single database/app server services multiple clients and we can essentially give up on that (taking the…
creator: Shailabh Nagar
date: 2006-09-08T17:26:53-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Dave Hansen wrote: > On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: >> I'm afraid we have different understandings of what a "guarantee" is. > > It appears so. > >> Don't we? >> Guarantee may be one of >> >> 1. container will be able to touch that number of pages >> 2. container will be able to sys_mmap() that number of pages >> 3. container will not be killed unless it touches that number of pages > > A "death sentence" guarantee? I like it. :) > >> 4. anything else >> >> Let's decide what kind of a guarantee we want. I think of guarantees w.r.t resources as the lower limit on the resource. Guarantees and limits can be thought of as the range (guarantee, limit] for the usage of the resource. > > I think of it as: "I will be allowed to use this many total pages, and > they are guaranteed not to fail." (1), I think. The sum of all of the > system's guarantees must be less than or equal to the amount of free > memory on the machine. > Yes, totally agree. > If we knew to which NUMA node the memory was going to go, we might as > well take the pages out of the allocator. > > -- Dave > -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-08T18:26:56-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Fri, 2006-09-08 at 11:22 +0400, Pavel Emelianov wrote: > Chandra Seetharaman wrote: > > [snip] > >>>> The question is - whether web server is multithreaded or not... > >>>> If it is not - then no problem here, you can change current > >>>> context and new resources will be charged accordingly. > >>>> > >>>> And current BC code is _able_ to handle it with _minor_ changes. > >>>> (One just need to save bc not on mm struct, but rather on vma struct > >>>> and change mm->bc on set_bc_id()). > >>>> > >>>> However, no one (can some one from CKRM team please?) explained so far > >>>> what to do with threads. Consider the following example. > >>>> > >>>> 1. Threaded web server spawns a child to serve a client. > >>>> 2. child thread touches some pages and they are charged to child BC > >>>> (which differs from parent's one) > >>>> 3. child exits, but since its mm is shared with parent, these pages > >>>> stay mapped and charged to child BC. > >>>> > >>>> So the question is: what to do with these pages? > >>>> - should we recharge them to another BC? > >>>> - leave them charged? > >>>> > >>>> > >>> Leave them charged. It will be charged to the appropriate UBC when they > >>> touch it again. > >>> > >>> > >> Do you mean that page must be re-charged each time someone touches it? > >> > > > > What I meant is that to leave them charged, and if when they are > > ummapped and mapped later, charge it to the appropriate BC. > > > In this case multithreaded apache that tries to serve each domain in > separate BC will fill the memory with BC-s, held by pages allocated > and mapped in threads. I do not understand how the memory will be filled with BCs. Can you explain, please. -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-08T19:07:22-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Fri, 2006-09-08 at 11:26 +0400, Pavel Emelianov wrote: > Chandra Seetharaman wrote: > > On Thu, 2006-09-07 at 11:29 +0400, Pavel Emelianov wrote: > > <snip> > > > > > >>>> BUT: I remind you the talks at OKS/OLS and in previous UBC discussions. > >>>> It was noted that having a separate interfaces for CPU, I/O bandwidth > >>>> > >>>> > >>> But, it will be lot simpler for the user to configure/use if they are > >>> together. We should discuss this also. > >>> > >>> > >> IMHO such unification may only imply that one syscall is used to pass > >> configuration info into kernel. > >> Each controller has specific configurating parameters different from the > >> other ones. E.g. CPU controller must assign a "weight" to each group to > >> share CPU time accordingly, but what is a "weight" for memory controller? > >> IO may operate on "bandwidth" and it's not clear what is a "bandwidth" in > >> Kb/sec for CPU controller and so on. > >> > > > > CKRM/RG handles this by eliminating the units from the interface and > > abstracting them to be "shares". Each resource controller converts the > > shares to its own units and handles properly. > > > That's what I'm talking about - common syscall/ioct/etc and each controller > parses its input itself. That's OK for us. Yes, we can eliminate the "units"(KBs, cycles/ticks, pages etc.,) from the interface and use a (unitless) number to specify the amount of resource a resource group/container uses. > > [snip] > > ------------------------------------------------------------ ------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&b id=263057&dat=121642 > _______________________________________________ > ckrm-tech mailing list > https://lists.sourceforge.net/lists/listinfo/ckrm-tech -- --------------…
creator: Chandra Seetharaman
date: 2006-09-08T19:10:41-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: > Chandra Seetharaman wrote: > > On Thu, 2006-09-07 at 00:47 +0530, Balbir Singh wrote: > > > > <snip> > >> Some not quite so urgent ones - like support for guarantees. I think > >> this can > > > > IMO, guarantee support should be considered to be part of the > > infrastructure. Controller functionalities/implementation will be > > different with/without guarantee support. In other words, adding > > guarantee feature later will cause re-implementations. > I'm afraid we have different understandings of what a "guarantee" is. > Don't we? may be (I am not sure :), lets get it clarified. > Guarantee may be one of > > 1. container will be able to touch that number of pages > 2. container will be able to sys_mmap() that number of pages > 3. container will not be killed unless it touches that number of pages > 4. anything else I would say (1) with slight modification "container will be able to touch _at least_ that number of pages" Note that it is not only in the context of memory alone, it is generic across resources. For CPU it will be, "container will get _at least_ X ticks in Y seconds" For number of tasks it will be, "container will get _at least_ X active tasks at any point of time" and so on. And as Dave pointed, sum of guarantees of all containers _can not_ exceed the total amount of that resource available at the system level. > > Let's decide what kind of a guarantee we want. > >> be worked out as we make progress. > >> > >>> I agree with these requirements and lets move into this direction. > >>> But moving so far can't be done without accepting: > >>> 1. core functionality > >>> 2. accounting > >>> > >> Some of the core functionality might be a limiting factor for the requirements. > >> Lets agree on the requirements, I think its a great step forward and then > >> build the core functionality with these requirements in mind. > >> > >>> Thanks, > >>> Kirill > >>> > > > -----------------------------------…
creator: Chandra Seetharaman
date: 2006-09-08T19:23:44-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Fri, 2006-09-08 at 12:23 -0700, Chandra Seetharaman wrote: > On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: > > Chandra Seetharaman wrote: > > > On Thu, 2006-09-07 at 00:47 +0530, Balbir Singh wrote: > > > > > > <snip> > > >> Some not quite so urgent ones - like support for guarantees. I think > > >> this can > > > > > > IMO, guarantee support should be considered to be part of the > > > infrastructure. Controller functionalities/implementation will be > > > different with/without guarantee support. In other words, adding > > > guarantee feature later will cause re-implementations. > > I'm afraid we have different understandings of what a "guarantee" is. > > Don't we? > > may be (I am not sure :), lets get it clarified. > > > Guarantee may be one of > > > > 1. container will be able to touch that number of pages > > 2. container will be able to sys_mmap() that number of pages > > 3. container will not be killed unless it touches that number of pages > > 4. anything else > > I would say (1) with slight modification > "container will be able to touch _at least_ that number of pages" > Does this scheme support running of tasks outside of containers on the same platform where you have tasks running inside containers. If so then how will you ensure processes running out side any container will not leave less than the total guaranteed memory to different containers. -rohit
creator: Rohit Seth
date: 2006-09-08T21:43:54-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Balbir Singh wrote: > Dave Hansen wrote: >> On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: >>> I'm afraid we have different understandings of what a "guarantee" is. >> >> It appears so. >> >>> Don't we? >>> Guarantee may be one of >>> >>> 1. container will be able to touch that number of pages >>> 2. container will be able to sys_mmap() that number of pages >>> 3. container will not be killed unless it touches that number of >>> pages >> >> A "death sentence" guarantee? I like it. :) >> >>> 4. anything else >>> >>> Let's decide what kind of a guarantee we want. > > I think of guarantees w.r.t resources as the lower limit on the resource. > Guarantees and limits can be thought of as the range (guarantee, limit] > for the usage of the resource. > >> >> I think of it as: "I will be allowed to use this many total pages, and >> they are guaranteed not to fail." (1), I think. The sum of all of the >> system's guarantees must be less than or equal to the amount of free >> memory on the machine. > > Yes, totally agree. Such a guarantee is really a limit and this limit is even harder than BC's one :) E.g. I have a node with 1Gb of ram and 10 containers with 100Mb guarantee each. I want to start one more. What shall I do not to break guarantees? > >> If we knew to which NUMA node the memory was going to go, we might as >> well take the pages out of the allocator. >> >> -- Dave >>
creator: Pavel Emelianov
date: 2006-09-11T06:56:59-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Chandra Seetharaman wrote: > On Fri, 2006-09-08 at 11:22 +0400, Pavel Emelianov wrote: > >> Chandra Seetharaman wrote: >> >> [snip] >> >>>>>> The question is - whether web server is multithreaded or not... >>>>>> If it is not - then no problem here, you can change current >>>>>> context and new resources will be charged accordingly. >>>>>> >>>>>> And current BC code is _able_ to handle it with _minor_ changes. >>>>>> (One just need to save bc not on mm struct, but rather on vma struct >>>>>> and change mm->bc on set_bc_id()). >>>>>> >>>>>> However, no one (can some one from CKRM team please?) explained so far >>>>>> what to do with threads. Consider the following example. >>>>>> >>>>>> 1. Threaded web server spawns a child to serve a client. >>>>>> 2. child thread touches some pages and they are charged to child BC >>>>>> (which differs from parent's one) >>>>>> 3. child exits, but since its mm is shared with parent, these pages >>>>>> stay mapped and charged to child BC. >>>>>> >>>>>> So the question is: what to do with these pages? >>>>>> - should we recharge them to another BC? >>>>>> - leave them charged? >>>>>> >>>>>> >>>>>> >>>>> Leave them charged. It will be charged to the appropriate UBC when they >>>>> touch it again. >>>>> >>>>> >>>>> >>>> Do you mean that page must be re-charged each time someone touches it? >>>> >>>> >>> What I meant is that to leave them charged, and if when they are >>> ummapped and mapped later, charge it to the appropriate BC. >>> >>> >> In this case multithreaded apache that tries to serve each domain in >> separate BC will fill the memory with BC-s, held by pages allocated >> and mapped in threads. >> > > I do not understand how the memory will be filled with BCs. Can you > explain, please. > Sure. At the beginning I have one task with one BC. Then 1. A thread is spawned and new BC is created; 2. New thread touches a new page (e.g. maps a new file) which is charged to new BC (and this means that this BC's must stay in memory till …
creator: Pavel Emelianov
date: 2006-09-11T07:02:06-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Pavel Emelianov wrote: > Balbir Singh wrote: >> Dave Hansen wrote: >>> On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: >>>> I'm afraid we have different understandings of what a "guarantee" is. >>> It appears so. >>> >>>> Don't we? >>>> Guarantee may be one of >>>> >>>> 1. container will be able to touch that number of pages >>>> 2. container will be able to sys_mmap() that number of pages >>>> 3. container will not be killed unless it touches that number of >>>> pages >>> A "death sentence" guarantee? I like it. :) >>> >>>> 4. anything else >>>> >>>> Let's decide what kind of a guarantee we want. >> I think of guarantees w.r.t resources as the lower limit on the resource. >> Guarantees and limits can be thought of as the range (guarantee, limit] >> for the usage of the resource. >> >>> I think of it as: "I will be allowed to use this many total pages, and >>> they are guaranteed not to fail." (1), I think. The sum of all of the >>> system's guarantees must be less than or equal to the amount of free >>> memory on the machine. >> Yes, totally agree. > > Such a guarantee is really a limit and this limit is even harder than > BC's one :) > > E.g. I have a node with 1Gb of ram and 10 containers with 100Mb > guarantee each. > I want to start one more. What shall I do not to break guarantees? Don't start the new container or change the guarantees of the existing ones to accommodate this one :) The QoS design (done by the administrator) should take care of such use-cases. It would be perfectly ok to have a container that does not care about guarantees to set their guarantee to 0 and set their limit to the desired value. As Chandra has been stating we need two parameters (guarantee, limit), either can be optional, but not both. > >>> If we knew to which NUMA node the memory was going to go, we might as >>> well take the pages out of the allocator. >>> >>> -- Dave >>> -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-11T07:54:06-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Balbir Singh wrote: > Pavel Emelianov wrote: >> Balbir Singh wrote: >>> Dave Hansen wrote: >>>> On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: >>>>> I'm afraid we have different understandings of what a "guarantee" is. >>>> It appears so. >>>> >>>>> Don't we? >>>>> Guarantee may be one of >>>>> >>>>> 1. container will be able to touch that number of pages >>>>> 2. container will be able to sys_mmap() that number of pages >>>>> 3. container will not be killed unless it touches that number of >>>>> pages >>>> A "death sentence" guarantee? I like it. :) >>>> >>>>> 4. anything else >>>>> >>>>> Let's decide what kind of a guarantee we want. >>> I think of guarantees w.r.t resources as the lower limit on the >>> resource. >>> Guarantees and limits can be thought of as the range (guarantee, limit] >>> for the usage of the resource. >>> >>>> I think of it as: "I will be allowed to use this many total pages, and >>>> they are guaranteed not to fail." (1), I think. The sum of all of >>>> the >>>> system's guarantees must be less than or equal to the amount of free >>>> memory on the machine. >>> Yes, totally agree. >> >> Such a guarantee is really a limit and this limit is even harder than >> BC's one :) >> >> E.g. I have a node with 1Gb of ram and 10 containers with 100Mb >> guarantee each. >> I want to start one more. What shall I do not to break guarantees? > > Don't start the new container or change the guarantees of the existing > ones > to accommodate this one :) The QoS design (done by the administrator) > should > take care of such use-cases. It would be perfectly ok to have a container > that does not care about guarantees to set their guarantee to 0 and set > their limit to the desired value. As Chandra has been stating we need two > parameters (guarantee, limit), either can be optional, but not both. If I set up 9 groups to have 100Mb limit then I have 100Mb assured (on 1Gb node) for the 10th one exactly. And I do not have to set up any guarantee as it w…
creator: Pavel Emelianov
date: 2006-09-11T08:13:59-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Pavel Emelianov wrote: > Balbir Singh wrote: >> Pavel Emelianov wrote: >>> Balbir Singh wrote: >>>> Dave Hansen wrote: >>>>> On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: >>>>>> I'm afraid we have different understandings of what a "guarantee" is. >>>>> It appears so. >>>>> >>>>>> Don't we? >>>>>> Guarantee may be one of >>>>>> >>>>>> 1. container will be able to touch that number of pages >>>>>> 2. container will be able to sys_mmap() that number of pages >>>>>> 3. container will not be killed unless it touches that number of >>>>>> pages >>>>> A "death sentence" guarantee? I like it. :) >>>>> >>>>>> 4. anything else >>>>>> >>>>>> Let's decide what kind of a guarantee we want. >>>> I think of guarantees w.r.t resources as the lower limit on the >>>> resource. >>>> Guarantees and limits can be thought of as the range (guarantee, limit] >>>> for the usage of the resource. >>>> >>>>> I think of it as: "I will be allowed to use this many total pages, and >>>>> they are guaranteed not to fail." (1), I think. The sum of all of >>>>> the >>>>> system's guarantees must be less than or equal to the amount of free >>>>> memory on the machine. >>>> Yes, totally agree. >>> Such a guarantee is really a limit and this limit is even harder than >>> BC's one :) >>> >>> E.g. I have a node with 1Gb of ram and 10 containers with 100Mb >>> guarantee each. >>> I want to start one more. What shall I do not to break guarantees? >> Don't start the new container or change the guarantees of the existing >> ones >> to accommodate this one :) The QoS design (done by the administrator) >> should >> take care of such use-cases. It would be perfectly ok to have a container >> that does not care about guarantees to set their guarantee to 0 and set >> their limit to the desired value. As Chandra has been stating we need two >> parameters (guarantee, limit), either can be optional, but not both. > If I set up 9 groups to have 100Mb limit then I have 100Mb assured (on > 1Gb node) > for t…
creator: Balbir Singh
date: 2006-09-11T08:19:22-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, Sep 11, 2006 at 12:13:59PM +0400, Pavel Emelianov wrote: > If I set up 9 groups to have 100Mb limit then I have 100Mb assured (on > 1Gb node) > for the 10th one exactly. And I do not have to set up any guarantee as > it won't affect > anything. So what a guarantee parameter is needed for? I presume you are talking of hard-limiting each group to 100 MB here. In which case, wont the 100MB (reserved for 10th group) be unutilized untill 10th group is started (it may never be started for that matter!). IMO it would be better to go and use that free 100 MB for reclaimable memory and give that up when 10th group is started. -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2006-09-11T10:21:52-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, Sep 11, 2006 at 11:02:06AM +0400, Pavel Emelianov wrote: > Sure. At the beginning I have one task with one BC. Then > 1. A thread is spawned and new BC is created; Why do we have to create a BC for every new thread? A new BC is needed for every new service level instead IMO. And typically there wont be unlimited service levels. > 2. New thread touches a new page (e.g. maps a new file) which is charged > to new BC > (and this means that this BC's must stay in memory till page is > uncharged); > 3. Thread exits after serving the request, but since it's mm is shared > with parent > all the touched pages stay resident and, thus, the new BC is still > pinned in memory. > Steps 1-3 are done multiple times for new pages (new files). > Remember that we're discussing the case when pages are not recharged. -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2006-09-11T13:04:28-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Fri, 2006-09-08 at 14:43 -0700, Rohit Seth wrote: <snip> > > > Guarantee may be one of > > > > > > 1. container will be able to touch that number of pages > > > 2. container will be able to sys_mmap() that number of pages > > > 3. container will not be killed unless it touches that number of pages > > > 4. anything else > > > > I would say (1) with slight modification > > "container will be able to touch _at least_ that number of pages" > > > > Does this scheme support running of tasks outside of containers on the > same platform where you have tasks running inside containers. If so > then how will you ensure processes running out side any container will > not leave less than the total guaranteed memory to different containers. > There could be a default container which doesn't have any guarantee or limit. When you create containers and assign guarantees to each of them make sure that you leave some amount of resource unassigned. That unassigned resources can be used by the default container or can be used by containers that want more than their guarantee (and less than their limit). This is how CKRM/RG handles this issue. > > > -rohit > > > ------------------------------------------------------------ ------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&b id=263057&dat=121642 > _______________________________________________ > ckrm-tech mailing list > https://lists.sourceforge.net/lists/listinfo/ckrm-tech -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-11T18:25:07-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, 2006-09-11 at 10:56 +0400, Pavel Emelianov wrote: <snip> > >> I think of it as: "I will be allowed to use this many total pages, and > >> they are guaranteed not to fail." (1), I think. The sum of all of the > >> system's guarantees must be less than or equal to the amount of free > >> memory on the machine. > > > > Yes, totally agree. > > Such a guarantee is really a limit and this limit is even harder than > BC's one :) > > E.g. I have a node with 1Gb of ram and 10 containers with 100Mb > guarantee each. In the first place system administrator should not be configuring it that way, Then they are using it as a strict hard limit than guarantee (as the resources guaranteed to one container is _not_ available to others). Besides, the above configuration is clearly _not_ work conservative. They should use both guarantee and limit to associate resources to a container/RG. > I want to start one more. What shall I do not to break guarantees? CKRM/RG handles it this way: Amount of a resource a child RG gets is the ratio of its share value to the parent's total # of shares. Children's resource allocation can be changed just by changing the parent's total # of shares. If you case about initial situation would be: Total memory in the system 100MB parent's total # of shares: 100 (1 share == 1MB) 10 children with # of shares: 10 (i.e each children has 10MB) When I want to add another child, just change parent's total # of shares to be say 125: Total memory in the system 100MB parent's total # of shares: 125 (1 share == 0.8MB) 10 children with # of shares: 10 (i.e each children has 8MB) Now you are left with 25 shares (or 20MB) that you can assign to new child(ren) as you please. <snip> -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-11T18:44:22-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, 2006-09-11 at 11:02 +0400, Pavel Emelianov wrote: <snip> > >> In this case multithreaded apache that tries to serve each domain in > >> separate BC will fill the memory with BC-s, held by pages allocated > >> and mapped in threads. > >> > > > > I do not understand how the memory will be filled with BCs. Can you > > explain, please. > > > Sure. At the beginning I have one task with one BC. Then > 1. A thread is spawned and new BC is created; You do not have to create a new BC for each new thread, just associate the thread to an existing appropriate BC. > 2. New thread touches a new page (e.g. maps a new file) which is charged > to new BC > (and this means that this BC's must stay in memory till page is > uncharged); > 3. Thread exits after serving the request, but since it's mm is shared > with parent > all the touched pages stay resident and, thus, the new BC is still > pinned in memory. > Steps 1-3 are done multiple times for new pages (new files). > Remember that we're discussing the case when pages are not recharged. > > ------------------------------------------------------------ ------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&b id=263057&dat=121642 > _______________________________________________ > ckrm-tech mailing list > https://lists.sourceforge.net/lists/listinfo/ckrm-tech -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-11T18:47:09-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, 2006-09-11 at 12:13 +0400, Pavel Emelianov wrote: <snip> > > > > Don't start the new container or change the guarantees of the existing > > ones > > to accommodate this one :) The QoS design (done by the administrator) > > should > > take care of such use-cases. It would be perfectly ok to have a container > > that does not care about guarantees to set their guarantee to 0 and set > > their limit to the desired value. As Chandra has been stating we need two > > parameters (guarantee, limit), either can be optional, but not both. > If I set up 9 groups to have 100Mb limit then I have 100Mb assured (on > 1Gb node) > for the 10th one exactly. And I do not have to set up any guarantee as > it won't affect > anything. So what a guarantee parameter is needed for? I do not think it is that simple since - there is typically more than one class I want to set guarantee to - I will not able to use both limit and guarantee - Implementation will not be work-conserving. Also, How would you configure the following in your model ? 5 classes: Class A(10, 40), Class B(20, 100), Class C (30, 100), Class D (5, 100), Class E(15, 50); (class_name(guarantee, limit)) "Limit only" approach works for DoS prevention. But for providing QoS you would need guarantee. -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-11T18:49:50-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, 2006-09-11 at 11:25 -0700, Chandra Seetharaman wrote: > On Fri, 2006-09-08 at 14:43 -0700, Rohit Seth wrote: > <snip> > > > > > Guarantee may be one of > > > > > > > > 1. container will be able to touch that number of pages > > > > 2. container will be able to sys_mmap() that number of pages > > > > 3. container will not be killed unless it touches that number of pages > > > > 4. anything else > > > > > > I would say (1) with slight modification > > > "container will be able to touch _at least_ that number of pages" > > > > > > > Does this scheme support running of tasks outside of containers on the > > same platform where you have tasks running inside containers. If so > > then how will you ensure processes running out side any container will > > not leave less than the total guaranteed memory to different containers. > > > > There could be a default container which doesn't have any guarantee or > limit. First, I think it is critical that we allow processes to run outside of any container (unless we know for sure that the penalty of running a process inside a container is very very minimal). And anything running outside a container should be limited by default Linux settings. > When you create containers and assign guarantees to each of them > make sure that you leave some amount of resource unassigned. ^^^^^ This will force the "default" container with limits (indirectly). IMO, the whole guarantee feature gets defeated the moment you bring in this fuzziness. > That > unassigned resources can be used by the default container or can be used > by containers that want more than their guarantee (and less than their > limit). This is how CKRM/RG handles this issue. > > It seems that a single notion of limit should suffice, and that limit should more be treated as something beyond which that resource consumption in the container will be throttled/not_allowed. -rohit
creator: Rohit Seth
date: 2006-09-11T19:10:31-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, 2006-09-11 at 12:10 -0700, Rohit Seth wrote: > On Mon, 2006-09-11 at 11:25 -0700, Chandra Seetharaman wrote: > > On Fri, 2006-09-08 at 14:43 -0700, Rohit Seth wrote: > > <snip> > > > > > > > Guarantee may be one of > > > > > > > > > > 1. container will be able to touch that number of pages > > > > > 2. container will be able to sys_mmap() that number of pages > > > > > 3. container will not be killed unless it touches that number of pages > > > > > 4. anything else > > > > > > > > I would say (1) with slight modification > > > > "container will be able to touch _at least_ that number of pages" > > > > > > > > > > Does this scheme support running of tasks outside of containers on the > > > same platform where you have tasks running inside containers. If so > > > then how will you ensure processes running out side any container will > > > not leave less than the total guaranteed memory to different containers. > > > > > > > There could be a default container which doesn't have any guarantee or > > limit. > > First, I think it is critical that we allow processes to run outside of > any container (unless we know for sure that the penalty of running a > process inside a container is very very minimal). When I meant a default container I meant a default "resource group". In case of container that would be the default environment. I do not see any additional overhead associated with it, it is only associated with how resource are allocated/accounted. > > And anything running outside a container should be limited by default > Linux settings. note that the resource available to the default RG will be (total system resource - allocated to RGs). > > > When you create containers and assign guarantees to each of them > > make sure that you leave some amount of resource unassigned. > ^^^^^ This will force the "default" container > with limits (indirectly). IMO, the whole guarantee feature gets defeated You _will_ have limits for the default RG even if we don't have guaran…
creator: Chandra Seetharaman
date: 2006-09-11T19:42:05-00:00

Re: Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Rohit Seth wrote: > On Mon, 2006-09-11 at 11:25 -0700, Chandra Seetharaman wrote: > >> On Fri, 2006-09-08 at 14:43 -0700, Rohit Seth wrote: >> <snip> >> >> >>>>> Guarantee may be one of >>>>> >>>>> 1. container will be able to touch that number of pages >>>>> 2. container will be able to sys_mmap() that number of pages >>>>> 3. container will not be killed unless it touches that number of pages >>>>> 4. anything else >>>>> >>>> I would say (1) with slight modification >>>> "container will be able to touch _at least_ that number of pages" >>>> >>>> >>> Does this scheme support running of tasks outside of containers on the >>> same platform where you have tasks running inside containers. If so >>> then how will you ensure processes running out side any container will >>> not leave less than the total guaranteed memory to different containers. >>> >>> >> There could be a default container which doesn't have any guarantee or >> limit. >> > > First, I think it is critical that we allow processes to run outside of > any container (unless we know for sure that the penalty of running a > process inside a container is very very minimal). > (1) there is a set of processes running outside of any container. In OpenVZ we call that "VE0" or "host system", probably Chandra meant that by "default container". (2) The host system is used to manage the containers (start/stop/set parameters/create/destroy). (3) the penalty of running a process inside a container is indeed very low. > And anything running outside a container should be limited by default > Linux settings. > (4) due to (2), it is not recommended to run anything but the tasks used to manage the containers -- otherwise your gonna have security problems (5) "Default Linux settings" do not cover everything (for example -- dentry cache), thus the need for beancounters. >> When you create containers and assign guarantees to each of them >> make sure that you leave some amount of resource unassigned. >> > ^^^^^ This will force t…
creator: kir
date: 2006-09-11T19:47:24-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, 2006-09-11 at 12:42 -0700, Chandra Seetharaman wrote: > On Mon, 2006-09-11 at 12:10 -0700, Rohit Seth wrote: > > On Mon, 2006-09-11 at 11:25 -0700, Chandra Seetharaman wrote: > > > There could be a default container which doesn't have any guarantee or > > > limit. > > > > First, I think it is critical that we allow processes to run outside of > > any container (unless we know for sure that the penalty of running a > > process inside a container is very very minimal). > > When I meant a default container I meant a default "resource group". In > case of container that would be the default environment. I do not see > any additional overhead associated with it, it is only associated with > how resource are allocated/accounted. > There should be some cost when you do atomic inc/dec accounting and locks for add/remove resources from any container (including default resource group). No? > > > > And anything running outside a container should be limited by default > > Linux settings. > > note that the resource available to the default RG will be (total system > resource - allocated to RGs). I think it will be preferable to not change the existing behavior for applications that are running outside any container (in your case default resource group). > > > > > When you create containers and assign guarantees to each of them > > > make sure that you leave some amount of resource unassigned. > > ^^^^^ This will force the "default" container > > with limits (indirectly). IMO, the whole guarantee feature gets defeated > > You _will_ have limits for the default RG even if we don't have > guarantees. > > > the moment you bring in this fuzziness. > > Not really. > - Each RG will have a guarantee and limit of each resource. > - default RG will have (system resource - sum of guarantees) > - Every RG will be guaranteed some amount of resource to provide QoS > - Every RG will be limited at "limit" to prevent DoS attacks. > - Whoever doesn't care either of those set them to don't…
creator: Rohit Seth
date: 2006-09-11T23:58:19-00:00

Re: Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, 2006-09-11 at 23:48 +0400, Kir Kolyshkin wrote: > Rohit Seth wrote: > > On Mon, 2006-09-11 at 11:25 -0700, Chandra Seetharaman wrote: > > > >> On Fri, 2006-09-08 at 14:43 -0700, Rohit Seth wrote: > >> <snip> > >> > >> > >>>>> Guarantee may be one of > >>>>> > >>>>> 1. container will be able to touch that number of pages > >>>>> 2. container will be able to sys_mmap() that number of pages > >>>>> 3. container will not be killed unless it touches that number of pages > >>>>> 4. anything else > >>>>> > >>>> I would say (1) with slight modification > >>>> "container will be able to touch _at least_ that number of pages" > >>>> > >>>> > >>> Does this scheme support running of tasks outside of containers on the > >>> same platform where you have tasks running inside containers. If so > >>> then how will you ensure processes running out side any container will > >>> not leave less than the total guaranteed memory to different containers. > >>> > >>> > >> There could be a default container which doesn't have any guarantee or > >> limit. > >> > > > > First, I think it is critical that we allow processes to run outside of > > any container (unless we know for sure that the penalty of running a > > process inside a container is very very minimal). > > > (1) there is a set of processes running outside of any container. In > OpenVZ we call that "VE0" or "host system", probably Chandra meant that > by "default container". > (2) The host system is used to manage the containers (start/stop/set > parameters/create/destroy). > (3) the penalty of running a process inside a container is indeed very low. > > > And anything running outside a container should be limited by default > > Linux settings. > > > (4) due to (2), it is not recommended to run anything but the tasks used > to manage the containers -- otherwise your gonna have security problems Just like you want to run those special threads outside of any container, some sysadmin might be interested in running different pro…
creator: Rohit Seth
date: 2006-09-12T00:28:58-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Rohit Seth wrote: > If the limits are set appropriately so that containers total memory > consumption does not exceed the system memory then there shouldn't be > any QoS issue (to whatever extent it is applicable for specific > scenario). > > -rohit > What if the guarantee and limits are subject to change? Consider many groups, with changing limits - how do we provide guarantees then? Limit is the upper bound on resource utilization and guarantee is the lower bound. In a dynamic system, how can we provide a lower bound on a resource for a group by manipulating the upper bounds on the rest of the groups? Consider a system with 1GB of ram and two groups such that they need a guarantee of 100MB and 200MB of memory. How would you setup limits to ensure that the guarantees are met? The remaining groups will be limited to 700MB, but how do we ensure that these classes get 100MB and 200MB of the remaining 300MB respectively? -- Balbir Singh, Linux Technology Center, IBM Software Labs
creator: Balbir Singh
date: 2006-09-12T09:53:38-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Srivatsa Vaddagiri wrote: > On Mon, Sep 11, 2006 at 11:02:06AM +0400, Pavel Emelianov wrote: > >> Sure. At the beginning I have one task with one BC. Then >> 1. A thread is spawned and new BC is created; >> > > Why do we have to create a BC for every new thread? A new BC is needed > for every new service level instead IMO. And typically there wont be > unlimited service levels. > That's the scenario we started from - each domain is served in a separate BC with *threaded* Apache. > >> 2. New thread touches a new page (e.g. maps a new file) which is charged >> to new BC >> (and this means that this BC's must stay in memory till page is >> uncharged); >> 3. Thread exits after serving the request, but since it's mm is shared >> with parent >> all the touched pages stay resident and, thus, the new BC is still >> pinned in memory. >> Steps 1-3 are done multiple times for new pages (new files). >> Remember that we're discussing the case when pages are not recharged. >> > > >
creator: Pavel Emelianov
date: 2006-09-12T10:24:25-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, Sep 12, 2006 at 02:24:25PM +0400, Pavel Emelianov wrote: > Srivatsa Vaddagiri wrote: > > On Mon, Sep 11, 2006 at 11:02:06AM +0400, Pavel Emelianov wrote: > > > >> Sure. At the beginning I have one task with one BC. Then > >> 1. A thread is spawned and new BC is created; > >> > > > > Why do we have to create a BC for every new thread? A new BC is needed > > for every new service level instead IMO. And typically there wont be > > unlimited service levels. > > > That's the scenario we started from - each domain is served in a separate > BC with *threaded* Apache. Sure ..but you can still meet that requirement by creating fixed set of BCs (for each domain) and let each new thread be associated with a corresponding BC (w/o requiring to create BC for every new thread), depending on which domain's request it is serving? > > > >> 2. New thread touches a new page (e.g. maps a new file) which is charged > >> to new BC > >> (and this means that this BC's must stay in memory till page is > >> uncharged); > >> 3. Thread exits after serving the request, but since it's mm is shared > >> with parent > >> all the touched pages stay resident and, thus, the new BC is still > >> pinned in memory. > >> Steps 1-3 are done multiple times for new pages (new files). > >> Remember that we're discussing the case when pages are not recharged. > >> > > > > > > > > > ------------------------------------------------------------ ------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&b id=263057&dat=121642 > _______________________________________________ > ckrm-tech mailing list > https://lists.sourceforge.net/lists/listinfo/ckrm-tech -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2006-09-12T10:29:43-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Srivatsa Vaddagiri wrote: > On Mon, Sep 11, 2006 at 12:13:59PM +0400, Pavel Emelianov wrote: > >> If I set up 9 groups to have 100Mb limit then I have 100Mb assured (on >> 1Gb node) >> for the 10th one exactly. And I do not have to set up any guarantee as >> it won't affect >> anything. So what a guarantee parameter is needed for? >> > > I presume you are talking of hard-limiting each group to 100 MB here. In > which case, wont the 100MB (reserved for 10th group) be unutilized > untill 10th group is started (it may never be started for that matter!). > > IMO it would be better to go and use that free 100 MB for reclaimable memory > and give that up when 10th group is started. > Sure. I've talked about the unreclaimable memory. Sorry, for not specifying it explicitly.
creator: Pavel Emelianov
date: 2006-09-12T10:35:45-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Balbir Singh wrote: > Pavel Emelianov wrote: >> Balbir Singh wrote: >>> Pavel Emelianov wrote: >>>> Balbir Singh wrote: >>>>> Dave Hansen wrote: >>>>>> On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: >>>>>>> I'm afraid we have different understandings of what a >>>>>>> "guarantee" is. >>>>>> It appears so. >>>>>> >>>>>>> Don't we? >>>>>>> Guarantee may be one of >>>>>>> >>>>>>> 1. container will be able to touch that number of pages >>>>>>> 2. container will be able to sys_mmap() that number of pages >>>>>>> 3. container will not be killed unless it touches that number of >>>>>>> pages >>>>>> A "death sentence" guarantee? I like it. :) >>>>>> >>>>>>> 4. anything else >>>>>>> >>>>>>> Let's decide what kind of a guarantee we want. >>>>> I think of guarantees w.r.t resources as the lower limit on the >>>>> resource. >>>>> Guarantees and limits can be thought of as the range (guarantee, >>>>> limit] >>>>> for the usage of the resource. >>>>> >>>>>> I think of it as: "I will be allowed to use this many total >>>>>> pages, and >>>>>> they are guaranteed not to fail." (1), I think. The sum of all of >>>>>> the >>>>>> system's guarantees must be less than or equal to the amount of free >>>>>> memory on the machine. >>>>> Yes, totally agree. >>>> Such a guarantee is really a limit and this limit is even harder than >>>> BC's one :) >>>> >>>> E.g. I have a node with 1Gb of ram and 10 containers with 100Mb >>>> guarantee each. >>>> I want to start one more. What shall I do not to break guarantees? >>> Don't start the new container or change the guarantees of the existing >>> ones >>> to accommodate this one :) The QoS design (done by the administrator) >>> should >>> take care of such use-cases. It would be perfectly ok to have a >>> container >>> that does not care about guarantees to set their guarantee to 0 and set >>> their limit to the desired value. As Chandra has been stating we >>> need two >>> parameters (guarantee, limit), either can be optional, but not bo…
creator: Pavel Emelianov
date: 2006-09-12T10:39:56-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Balbir Singh wrote: > Pavel Emelianov wrote: >> Balbir Singh wrote: >>> Pavel Emelianov wrote: >>>> Balbir Singh wrote: >>>>> Dave Hansen wrote: >>>>>> On Fri, 2006-09-08 at 11:33 +0400, Pavel Emelianov wrote: >>>>>>> I'm afraid we have different understandings of what a >>>>>>> "guarantee" is. >>>>>> It appears so. >>>>>> >>>>>>> Don't we? >>>>>>> Guarantee may be one of >>>>>>> >>>>>>> 1. container will be able to touch that number of pages >>>>>>> 2. container will be able to sys_mmap() that number of pages >>>>>>> 3. container will not be killed unless it touches that number of >>>>>>> pages >>>>>> A "death sentence" guarantee? I like it. :) >>>>>> >>>>>>> 4. anything else >>>>>>> >>>>>>> Let's decide what kind of a guarantee we want. >>>>> I think of guarantees w.r.t resources as the lower limit on the >>>>> resource. >>>>> Guarantees and limits can be thought of as the range (guarantee, >>>>> limit] >>>>> for the usage of the resource. >>>>> >>>>>> I think of it as: "I will be allowed to use this many total >>>>>> pages, and >>>>>> they are guaranteed not to fail." (1), I think. The sum of all of >>>>>> the >>>>>> system's guarantees must be less than or equal to the amount of free >>>>>> memory on the machine. >>>>> Yes, totally agree. >>>> Such a guarantee is really a limit and this limit is even harder than >>>> BC's one :) >>>> >>>> E.g. I have a node with 1Gb of ram and 10 containers with 100Mb >>>> guarantee each. >>>> I want to start one more. What shall I do not to break guarantees? >>> Don't start the new container or change the guarantees of the existing >>> ones >>> to accommodate this one :) The QoS design (done by the administrator) >>> should >>> take care of such use-cases. It would be perfectly ok to have a >>> container >>> that does not care about guarantees to set their guarantee to 0 and set >>> their limit to the desired value. As Chandra has been stating we >>> need two >>> parameters (guarantee, limit), either can be optional, but not bo…
creator: Pavel Emelianov
date: 2006-09-12T10:40:03-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, Sep 11, 2006 at 12:10:31PM -0700, Rohit Seth wrote: > It seems that a single notion of limit should suffice, and that limit > should more be treated as something beyond which that resource > consumption in the container will be throttled/not_allowed. The big question is : are containers/RG allowed to use *upto* their limit always? In other words, will you typically setup limits such that sum of all limits = max resource capacity? If it is setup like that, then what you are considering as limit is actually guar no? If it wont be setup like that, then I dont see how one can provide QoS. -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2006-09-12T10:44:10-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Chandra Seetharaman wrote: > On Mon, 2006-09-11 at 12:13 +0400, Pavel Emelianov wrote: > > <snip> > >>> Don't start the new container or change the guarantees of the existing >>> ones >>> to accommodate this one :) The QoS design (done by the administrator) >>> should >>> take care of such use-cases. It would be perfectly ok to have a container >>> that does not care about guarantees to set their guarantee to 0 and set >>> their limit to the desired value. As Chandra has been stating we need two >>> parameters (guarantee, limit), either can be optional, but not both. >>> >> If I set up 9 groups to have 100Mb limit then I have 100Mb assured (on >> 1Gb node) >> for the 10th one exactly. And I do not have to set up any guarantee as >> it won't affect >> anything. So what a guarantee parameter is needed for? >> > > I do not think it is that simple since > - there is typically more than one class I want to set guarantee to > - I will not able to use both limit and guarantee > - Implementation will not be work-conserving. > > Also, How would you configure the following in your model ? > > 5 classes: Class A(10, 40), Class B(20, 100), Class C (30, 100), Class D > (5, 100), Class E(15, 50); (class_name(guarantee, limit)) > What's the total memory amount on the node? Without it it's hard to make any guarantee. > "Limit only" approach works for DoS prevention. But for providing QoS > you would need guarantee. > You may not provide guarantee on physycal resource for a particular group without limiting its usage by other groups. That's my major idea.
creator: Pavel Emelianov
date: 2006-09-12T10:48:18-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Srivatsa Vaddagiri wrote: > On Tue, Sep 12, 2006 at 02:24:25PM +0400, Pavel Emelianov wrote: > >> Srivatsa Vaddagiri wrote: >> >>> On Mon, Sep 11, 2006 at 11:02:06AM +0400, Pavel Emelianov wrote: >>> >>> >>>> Sure. At the beginning I have one task with one BC. Then >>>> 1. A thread is spawned and new BC is created; >>>> >>>> >>> Why do we have to create a BC for every new thread? A new BC is needed >>> for every new service level instead IMO. And typically there wont be >>> unlimited service levels. >>> >>> >> That's the scenario we started from - each domain is served in a separate >> BC with *threaded* Apache. >> > > Sure ..but you can still meet that requirement by creating fixed set of > BCs (for each domain) and let each new thread be associated with a > corresponding BC (w/o requiring to create BC for every new thread), > depending on which domain's request it is serving? > Hmmm... Beancounters can provide this after trivial changes. We may schedule them in current set of "pending" features (http://wiki.openvz.org/UBC_discussion) But this can create a kind of DoS within an application: A thread continuously touches new and new pages to it's BC and these pages are get touched by other threads also. Sooner or later this BC will hit it's limit and reclaiming this set of pages would affect all the other threads. Also such accounting reveals you NOTHING about real memory usage. E.g. 100Mb charged for one BC can mean "this BC ate 100Mb of memory" as well as "this BC uses one page really, but all the others are just used by other threads" and anything between these two corner cases. Well. We've digressed from our main thread - discussing (dis)advantages of current BC implemenation. > >>> >>> >>>> 2. New thread touches a new page (e.g. maps a new file) which is charged >>>> to new BC >>>> (and this means that this BC's must stay in memory till page is >>>> uncharged); >>>> 3. Thread exits after serving the request, but since it's mm is shared >>>> with parent >>>> all…
creator: Pavel Emelianov
date: 2006-09-12T11:06:35-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Pavel Emelianov wrote: <snip> >>>>> E.g. I have a node with 1Gb of ram and 10 containers with 100Mb >>>>> guarantee each. >>>>> I want to start one more. What shall I do not to break guarantees? >>>> Don't start the new container or change the guarantees of the existing >>>> ones >>>> to accommodate this one :) The QoS design (done by the administrator) >>>> should >>>> take care of such use-cases. It would be perfectly ok to have a >>>> container >>>> that does not care about guarantees to set their guarantee to 0 and set >>>> their limit to the desired value. As Chandra has been stating we >>>> need two >>>> parameters (guarantee, limit), either can be optional, but not both. >>> If I set up 9 groups to have 100Mb limit then I have 100Mb assured (on >>> 1Gb node) >>> for the 10th one exactly. And I do not have to set up any guarantee as >>> it won't affect >>> anything. So what a guarantee parameter is needed for? >> This use case works well for providing guarantee to one container. >> What if >> I want guarantees of 100Mb and 200Mb for two containers? How do I setup >> the system using limits? > You may set any value from 100 up to 800 Mb for the first one and > 200-900Mb for > the second. In case of no other groups first will receive its 100Mb for > sure and > so does the second. If there are other groups - their guarantees should > be concerned. If I add another group with a guarantee of 100MB, then its limit will be anywhere between 100MB-800MB ? I do not understand the guarantees being concerned part. >> Even I restrict everyone else to 700Mb. With this I cannot be sure that >> the remaining 300Mb will be distributed as 100Mb and 200Mb. > There's no "everyone else" here - we're talking about a "static" case. > When new group arrives we need to recalculate guarantees as you said. I was speaking in general where we have 'n' groups, so thats why I had "everyone else". > And here's my next question - what to do if the new guarantee would become > lower that curre…
creator: Balbir Singh
date: 2006-09-12T12:01:43-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, Sep 12, 2006 at 03:06:35PM +0400, Pavel Emelianov wrote: > Hmmm... Beancounters can provide this after trivial changes. All that is needed is some interface to set a thread's BC id (which you seem to have already - sys_set_bcid) > We may schedule them in current set of "pending" features > (http://wiki.openvz.org/UBC_discussion) > > But this can create a kind of DoS within an application: > A thread continuously touches new and new pages to it's BC and > these pages are get touched by other threads also. Sooner or later Any good reason why threads will touch each other's working set? Sure nothing prevents them from touching, but I would expect each thread (serving a separate domain) to work on its own set of private pages? > this BC will hit it's limit and reclaiming this set of pages would affect > all the other threads. -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2006-09-12T12:04:07-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 16:14 +0530, Srivatsa Vaddagiri wrote: > On Mon, Sep 11, 2006 at 12:10:31PM -0700, Rohit Seth wrote: > > It seems that a single notion of limit should suffice, and that limit > > should more be treated as something beyond which that resource > > consumption in the container will be throttled/not_allowed. > > The big question is : are containers/RG allowed to use *upto* their > limit always? In other words, will you typically setup limits such that > sum of all limits = max resource capacity? > If a user is really interested in ensuring that all scheduled jobs (or containers) get what they have asked for (guarantees) then making the sum of all container limits equal to total system limit is the right thing to do. > If it is setup like that, then what you are considering as limit is > actually guar no? > Right. And if we do it like this then it is up to sysadmin to configure the thing right without adding additional logic in kernel. -rohit
creator: Rohit Seth
date: 2006-09-12T17:22:32-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, Sep 12, 2006 at 10:22:32AM -0700, Rohit Seth wrote: > On Tue, 2006-09-12 at 16:14 +0530, Srivatsa Vaddagiri wrote: > > On Mon, Sep 11, 2006 at 12:10:31PM -0700, Rohit Seth wrote: > > > It seems that a single notion of limit should suffice, and that limit > > > should more be treated as something beyond which that resource > > > consumption in the container will be throttled/not_allowed. > > > > The big question is : are containers/RG allowed to use *upto* their > > limit always? In other words, will you typically setup limits such that > > sum of all limits = max resource capacity? > > > > If a user is really interested in ensuring that all scheduled jobs (or > containers) get what they have asked for (guarantees) then making the > sum of all container limits equal to total system limit is the right > thing to do. > > > If it is setup like that, then what you are considering as limit is > > actually guar no? > > > Right. And if we do it like this then it is up to sysadmin to configure > the thing right without adding additional logic in kernel. Perhaps calling it as "limit" in confusing then (otoh it may go down well with Linus!). I perhaps agree we need to go with one for now (in the interest of making some progress), but we probably will come back to this at a later point. For ex, I chanced upon this document: www.vmware.com/pdf/vmware_drs_wp.pdf which explains how supporting a hard limit (in contrast to guar as we have been discussing) can be usefull sometimes. -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2006-09-12T17:40:58-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Mon, 2006-09-11 at 16:58 -0700, Rohit Seth wrote: > On Mon, 2006-09-11 at 12:42 -0700, Chandra Seetharaman wrote: > > On Mon, 2006-09-11 at 12:10 -0700, Rohit Seth wrote: > > > On Mon, 2006-09-11 at 11:25 -0700, Chandra Seetharaman wrote: > > > > > There could be a default container which doesn't have any guarantee or > > > > limit. > > > > > > First, I think it is critical that we allow processes to run outside of > > > any container (unless we know for sure that the penalty of running a > > > process inside a container is very very minimal). > > > > When I meant a default container I meant a default "resource group". In > > case of container that would be the default environment. I do not see > > any additional overhead associated with it, it is only associated with > > how resource are allocated/accounted. > > > > There should be some cost when you do atomic inc/dec accounting and > locks for add/remove resources from any container (including default > resource group). No? yes, it would be there, but is not heavy, IMO. > > > > > > > And anything running outside a container should be limited by default > > > Linux settings. > > > > note that the resource available to the default RG will be (total system > > resource - allocated to RGs). > > I think it will be preferable to not change the existing behavior for > applications that are running outside any container (in your case > default resource group). hmm, when you provide QoS for a set of apps, you will affect (the resource availability of) other apps. I don't see any way around it. Any ideas ? > > > > > > > > When you create containers and assign guarantees to each of them > > > > make sure that you leave some amount of resource unassigned. > > > ^^^^^ This will force the "default" container > > > with limits (indirectly). IMO, the whole guarantee feature gets defeated > > > > You _will_ have limits for the default RG even if we don't have > > guarantees. > > > > > the moment you bring in this fuzziness. > >…
creator: Chandra Seetharaman
date: 2006-09-12T23:54:13-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 14:48 +0400, Pavel Emelianov wrote: <snip> > > I do not think it is that simple since > > - there is typically more than one class I want to set guarantee to > > - I will not able to use both limit and guarantee > > - Implementation will not be work-conserving. > > > > Also, How would you configure the following in your model ? > > > > 5 classes: Class A(10, 40), Class B(20, 100), Class C (30, 100), Class D > > (5, 100), Class E(15, 50); (class_name(guarantee, limit)) > > > What's the total memory amount on the node? Without it it's hard to make > any > guarantee. I wrote the example treating them as %, so 100 would be the total amount of memory. > > "Limit only" approach works for DoS prevention. But for providing QoS > > you would need guarantee. > > > You may not provide guarantee on physycal resource for a particular group > without limiting its usage by other groups. That's my major idea. I agree with that, but the other way around (i.e provide guarantee for everyone by imposing limits on everyone) is what I am saying is not possible. > > ------------------------------------------------------------ ------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&b id=263057&dat=121642 > _______________________________________________ > ckrm-tech mailing list > https://lists.sourceforge.net/lists/listinfo/ckrm-tech -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-12T23:58:08-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 10:22 -0700, Rohit Seth wrote: > On Tue, 2006-09-12 at 16:14 +0530, Srivatsa Vaddagiri wrote: > > On Mon, Sep 11, 2006 at 12:10:31PM -0700, Rohit Seth wrote: > > > It seems that a single notion of limit should suffice, and that limit > > > should more be treated as something beyond which that resource > > > consumption in the container will be throttled/not_allowed. > > > > The big question is : are containers/RG allowed to use *upto* their > > limit always? In other words, will you typically setup limits such that > > sum of all limits = max resource capacity? > > > > If a user is really interested in ensuring that all scheduled jobs (or > containers) get what they have asked for (guarantees) then making the > sum of all container limits equal to total system limit is the right > thing to do. > > > If it is setup like that, then what you are considering as limit is > > actually guar no? > > > Right. And if we do it like this then it is up to sysadmin to configure > the thing right without adding additional logic in kernel. It won't be a complete solution, as the user won't be able to - set both guarantee and limit for a resource group - use limit on some and guarantee on some - optimize the usage of available resources > > -rohit > > > > ------------------------------------------------------------ ------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&b id=263057&dat=121642 > _______________________________________________ > ckrm-tech mailing list > https://lists.sourceforge.net/lists/listinfo/ckrm-tech -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ----------------------------…
creator: Chandra Seetharaman
date: 2006-09-13T00:02:12-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 16:54 -0700, Chandra Seetharaman wrote: > On Mon, 2006-09-11 at 16:58 -0700, Rohit Seth wrote: > > On Mon, 2006-09-11 at 12:42 -0700, Chandra Seetharaman wrote: > > > On Mon, 2006-09-11 at 12:10 -0700, Rohit Seth wrote: > > > > On Mon, 2006-09-11 at 11:25 -0700, Chandra Seetharaman wrote: > > > > > > > There could be a default container which doesn't have any guarantee or > > > > > limit. > > > > > > > > First, I think it is critical that we allow processes to run outside of > > > > any container (unless we know for sure that the penalty of running a > > > > process inside a container is very very minimal). > > > > > > When I meant a default container I meant a default "resource group". In > > > case of container that would be the default environment. I do not see > > > any additional overhead associated with it, it is only associated with > > > how resource are allocated/accounted. > > > > > > > There should be some cost when you do atomic inc/dec accounting and > > locks for add/remove resources from any container (including default > > resource group). No? > > yes, it would be there, but is not heavy, IMO. I think anything greater than 1% could be a concern for people who are not very interested in containers but would be forced to live with them. > > > > > > > > > > And anything running outside a container should be limited by default > > > > Linux settings. > > > > > > note that the resource available to the default RG will be (total system > > > resource - allocated to RGs). > > > > I think it will be preferable to not change the existing behavior for > > applications that are running outside any container (in your case > > default resource group). > > hmm, when you provide QoS for a set of apps, you will affect (the > resource availability of) other apps. I don't see any way around it. Any > ideas ? When I say, existing behavior, I mean not getting impacted by some artificial limits that are imposed by container subsystem. IOW, if a sysa…
creator: Rohit Seth
date: 2006-09-13T00:39:08-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 17:02 -0700, Chandra Seetharaman wrote: > On Tue, 2006-09-12 at 10:22 -0700, Rohit Seth wrote: > > On Tue, 2006-09-12 at 16:14 +0530, Srivatsa Vaddagiri wrote: > > > On Mon, Sep 11, 2006 at 12:10:31PM -0700, Rohit Seth wrote: > > > > It seems that a single notion of limit should suffice, and that limit > > > > should more be treated as something beyond which that resource > > > > consumption in the container will be throttled/not_allowed. > > > > > > The big question is : are containers/RG allowed to use *upto* their > > > limit always? In other words, will you typically setup limits such that > > > sum of all limits = max resource capacity? > > > > > > > If a user is really interested in ensuring that all scheduled jobs (or > > containers) get what they have asked for (guarantees) then making the > > sum of all container limits equal to total system limit is the right > > thing to do. > > > > > If it is setup like that, then what you are considering as limit is > > > actually guar no? > > > > > Right. And if we do it like this then it is up to sysadmin to configure > > the thing right without adding additional logic in kernel. > > It won't be a complete solution, as the user won't be able to > - set both guarantee and limit for a resource group > - use limit on some and guarantee on some > - optimize the usage of available resources I think, if we have some of the dynamic resource limit adjustments possible then some of the above functionality could be achieved. And I think that could be a good start point. -rohit
creator: Rohit Seth
date: 2006-09-13T00:43:23-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 17:39 -0700, Rohit Seth wrote: <snip> > > yes, it would be there, but is not heavy, IMO. > > I think anything greater than 1% could be a concern for people who are > not very interested in containers but would be forced to live with them. If they are not interested in resource management and/or containers, i do not think they need to pay. > > > > > > > > > > > > > > And anything running outside a container should be limited by default > > > > > Linux settings. > > > > > > > > note that the resource available to the default RG will be (total system > > > > resource - allocated to RGs). > > > > > > I think it will be preferable to not change the existing behavior for > > > applications that are running outside any container (in your case > > > default resource group). > > > > hmm, when you provide QoS for a set of apps, you will affect (the > > resource availability of) other apps. I don't see any way around it. Any > > ideas ? > > When I say, existing behavior, I mean not getting impacted by some > artificial limits that are imposed by container subsystem. IOW, if a That is what I understood and replied above. > sysadmin is okay to have certain apps running outside of container then > he is basically forgoing any QoS for any container on that system. Not at all. If the container they are interested in is guaranteed, I do not see how apps running outside a container would affect them. <snip> > > > > Not really. > > > > - Each RG will have a guarantee and limit of each resource. > > > > - default RG will have (system resource - sum of guarantees) > > > > - Every RG will be guaranteed some amount of resource to provide QoS > > > > - Every RG will be limited at "limit" to prevent DoS attacks. > > > > - Whoever doesn't care either of those set them to don't care values. > > > > > > > > > > For the cases that put this don't care, do you depend on existing > > > reclaim algorithm (for memory) in kernel? > > > > Yes. > > So one container with these don'…
creator: Chandra Seetharaman
date: 2006-09-13T01:10:18-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 17:43 -0700, Rohit Seth wrote: <snip> > > It won't be a complete solution, as the user won't be able to > > - set both guarantee and limit for a resource group > > - use limit on some and guarantee on some > > - optimize the usage of available resources > > I think, if we have some of the dynamic resource limit adjustments > possible then some of the above functionality could be achieved. And I > think that could be a good start point. Yes, dynamic resource adjustments should be available. But, you can't expect the sysadmin to sit around and keep tweaking the limits so as to achieve the QoS he wants. (Even if you have an application sitting and doing it, as I pointed in other email it may not be possible for different scenarios). > > -rohit > > > ------------------------------------------------------------ ------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&b id=263057&dat=121642 > _______________________________________________ > ckrm-tech mailing list > https://lists.sourceforge.net/lists/listinfo/ckrm-tech -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-13T01:13:10-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 18:10 -0700, Chandra Seetharaman wrote: > On Tue, 2006-09-12 at 17:39 -0700, Rohit Seth wrote: > <snip> > > > yes, it would be there, but is not heavy, IMO. > > > > I think anything greater than 1% could be a concern for people who are > > not very interested in containers but would be forced to live with them. > > If they are not interested in resource management and/or containers, i > do not think they need to pay. > > Think of a single kernel from a vendor that has container support built in. > > > > > > > > > > > > > > > > And anything running outside a container should be limited by default > > > > > > Linux settings. > > > > > > > > > > note that the resource available to the default RG will be (total system > > > > > resource - allocated to RGs). > > > > > > > > I think it will be preferable to not change the existing behavior for > > > > applications that are running outside any container (in your case > > > > default resource group). > > > > > > hmm, when you provide QoS for a set of apps, you will affect (the > > > resource availability of) other apps. I don't see any way around it. Any > > > ideas ? > > > > When I say, existing behavior, I mean not getting impacted by some > > artificial limits that are imposed by container subsystem. IOW, if a > > That is what I understood and replied above. > > sysadmin is okay to have certain apps running outside of container then > > he is basically forgoing any QoS for any container on that system. > > Not at all. If the container they are interested in is guaranteed, I do > not see how apps running outside a container would affect them. > Because the kernel (outside the container subsystem) doesn't know of these guarantees...unless you modify the page allocator to have another variant of overcommit memory. > <snip> > > > > > Not really. > > > > > - Each RG will have a guarantee and limit of each resource. > > > > > - default RG will have (system resource - sum of guarantees) > > > > > - Every R…
creator: Rohit Seth
date: 2006-09-13T01:25:51-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, Sep 12, 2006 at 06:25:51PM -0700, Rohit Seth wrote: > And how will the kernel reclaimer know which RGs are don't care? The UBC code can provide this information certainly. For ex: in my CPU controller patch, I do keep track of such don't care groups in the controller itself: http://lkml.org/lkml/2006/8/20/120 -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2006-09-13T04:41:24-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Chandra Seetharaman wrote: > On Tue, 2006-09-12 at 14:48 +0400, Pavel Emelianov wrote: > <snip> > >>> I do not think it is that simple since >>> - there is typically more than one class I want to set guarantee to >>> - I will not able to use both limit and guarantee >>> - Implementation will not be work-conserving. >>> >>> Also, How would you configure the following in your model ? >>> >>> 5 classes: Class A(10, 40), Class B(20, 100), Class C (30, 100), Class D >>> (5, 100), Class E(15, 50); (class_name(guarantee, limit)) >>> >>> >> What's the total memory amount on the node? Without it it's hard to make >> any >> guarantee. >> > > I wrote the example treating them as %, so 100 would be the total amount > of memory. > OK. Then limiting must be done this way (unreclaimable limit/total limit) A (15/40) B (25/100) C (35/100) D (10/100) E (20/50) In this case each group will receive it's guarantee for sure. E.g. even if A, B, E and D will eat all it's unreclaimable memory then we'll have 100 - 15 - 25 - 20 - 10 = 30% of memory left (maybe after reclaiming) which is perfectly enough for C's guarantee. > >>> "Limit only" approach works for DoS prevention. But for providing QoS >>> you would need guarantee. >>> >>> >> You may not provide guarantee on physycal resource for a particular group >> without limiting its usage by other groups. That's my major idea. >> > > I agree with that, but the other way around (i.e provide guarantee for > everyone by imposing limits on everyone) is what I am saying is not > possible. Then how do you make sure that memory WILL be available when the group needs it without limiting the others in a proper way?
creator: Pavel Emelianov
date: 2006-09-13T08:06:41-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Wed, Sep 13, 2006 at 12:06:41PM +0400, Pavel Emelianov wrote: > OK. Then limiting must be done this way (unreclaimable limit/total limit) > A (15/40) > B (25/100) > C (35/100) s/35/30? Also the different b/n total and unreclaimable limits goes towards limiting reclaimable memory i suppose? And 1st limit seems to be a hard-limit while the 2nd one is soft? > D (10/100) > E (20/50) > In this case each group will receive it's guarantee for sure. > > E.g. even if A, B, E and D will eat all it's unreclaimable memory then > we'll have > 100 - 15 - 25 - 20 - 10 = 30% of memory left (maybe after reclaiming) which > is perfectly enough for C's guarantee. I agree by carefully choosing these limits, we can provide some sort of QoS, which is a good step to begin with. -- Regards, vatsa
creator: Srivatsa Vaddagiri
date: 2006-09-13T12:15:25-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Srivatsa Vaddagiri wrote: > On Wed, Sep 13, 2006 at 12:06:41PM +0400, Pavel Emelianov wrote: >> OK. Then limiting must be done this way (unreclaimable limit/total limit) >> A (15/40) >> B (25/100) >> C (35/100) > > s/35/30? Hmmm... No, it must be 35. It IS higher than guarantee you proposed, but that's OK to have a limit higher than guarantee, isn't it? > > Also the different b/n total and unreclaimable limits goes towards > limiting reclaimable memory i suppose? And 1st limit seems to be a > hard-limit while the 2nd one is soft? The first limit (let's call it soft one) is limit for unreclaimable memory, the second (hard limit) - for booth reclaimable and not. The ploicy is 1. if BC tries to *mmap()* unreclaimable region (e.g. w/o backed file as moving page to swap is not a pure "reclamation") then check the soft limit and prohibit mapping in case it is hit; 2. if BC tries to *touch* a page - then check for the hard limit and start reclaiming this BC's pages if the limit is hit. That's how guarantees can be met. Current BC code does perform the first check and gives you all the levers for the second one - just the patch(es) with reclamation mechanism is required. > >> D (10/100) >> E (20/50) >> In this case each group will receive it's guarantee for sure. >> >> E.g. even if A, B, E and D will eat all it's unreclaimable memory then >> we'll have >> 100 - 15 - 25 - 20 - 10 = 30% of memory left (maybe after reclaiming) which >> is perfectly enough for C's guarantee. > > I agree by carefully choosing these limits, we can provide some sort of > QoS, which is a good step to begin with. Sure. As I've said - soft limiting is already done with BC patches, the hard one is not prohibited by BC (BCs even prepare a good pad for it). When reclaiming is done we'll have a hard limit described above.
creator: Pavel Emelianov
date: 2006-09-13T13:35:30-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: Balbir Singh wrote: > Pavel Emelianov wrote: > <snip> > >>>>>> E.g. I have a node with 1Gb of ram and 10 containers with 100Mb >>>>>> guarantee each. >>>>>> I want to start one more. What shall I do not to break guarantees? >>>>> Don't start the new container or change the guarantees of the >>>>> existing >>>>> ones >>>>> to accommodate this one :) The QoS design (done by the administrator) >>>>> should >>>>> take care of such use-cases. It would be perfectly ok to have a >>>>> container >>>>> that does not care about guarantees to set their guarantee to 0 >>>>> and set >>>>> their limit to the desired value. As Chandra has been stating we >>>>> need two >>>>> parameters (guarantee, limit), either can be optional, but not both. >>>> If I set up 9 groups to have 100Mb limit then I have 100Mb assured (on >>>> 1Gb node) >>>> for the 10th one exactly. And I do not have to set up any guarantee as >>>> it won't affect >>>> anything. So what a guarantee parameter is needed for? >>> This use case works well for providing guarantee to one container. >>> What if >>> I want guarantees of 100Mb and 200Mb for two containers? How do I setup >>> the system using limits? >> You may set any value from 100 up to 800 Mb for the first one and >> 200-900Mb for >> the second. In case of no other groups first will receive its 100Mb for >> sure and >> so does the second. If there are other groups - their guarantees should >> be concerned. > > If I add another group with a guarantee of 100MB, then its limit will > be anywhere between 100MB-800MB ? I've described this in details in my letter to sekharan@. > > I do not understand the guarantees being concerned part. > >>> Even I restrict everyone else to 700Mb. With this I cannot be sure that >>> the remaining 300Mb will be distributed as 100Mb and 200Mb. >> There's no "everyone else" here - we're talking about a "static" case. >> When new group arrives we need to recalculate guarantees as you said. > > I was speaking in general where we have…
creator: Pavel Emelianov
date: 2006-09-13T13:39:22-00:00

Re: [ckrm-tech] [PATCH] BC: resource beancounters (v4) (added user memory)
type: item
description: On Tue, 2006-09-12 at 18:25 -0700, Rohit Seth wrote: > On Tue, 2006-09-12 at 18:10 -0700, Chandra Seetharaman wrote: > > On Tue, 2006-09-12 at 17:39 -0700, Rohit Seth wrote: > > <snip> > > > > yes, it would be there, but is not heavy, IMO. > > > > > > I think anything greater than 1% could be a concern for people who are > > > not very interested in containers but would be forced to live with them. > > > > If they are not interested in resource management and/or containers, i > > do not think they need to pay. > > > > > Think of a single kernel from a vendor that has container support built > in. Ok. Understood. Here are results of some of the benchmarks we have run in the past (April 2005) with CKRM which showed no/negligible performance impact in that scenario. http://marc.theaimsgroup.com/?l=ckrm-tech&m=111325064322 305&w=2 http://marc.theaimsgroup.com/?l=ckrm-tech&m=111385973226 267&w=2 http://marc.theaimsgroup.com/?l=ckrm-tech&m=111291409731 929&w=2 > <snip> > > Not at all. If the container they are interested in is guaranteed, I do > > not see how apps running outside a container would affect them. > > > > Because the kernel (outside the container subsystem) doesn't know of The core resource subsystem (VM subsystem for memory) would know about the guarantees and don't cares, and it would handle it appropriately. > these guarantees...unless you modify the page allocator to have another > variant of overcommit memory. > <snip> > > > No, the reclaimer would free up pages associated with the don't care RGs > > ( as the user don't care about the resource made available to them). > > > > And how will the kernel reclaimer know which RGs are don't care? By looking into the beancounter associated with the container/RG -- ------------------------------------------------------------ ---------- Chandra Seetharaman | Be careful what you choose.... - sekharan@us.ibm.com | .......you may get it. ------------------------------------------------------------ ----------
creator: Chandra Seetharaman
date: 2006-09-13T22:20:46-00:00