OpenVZ Forum - RDF feed
type: channel
items: [PATCH v4 00/19] slab accounting for memcg
items: [PATCH v4 01/19] slab: Ignore internal flags in cache creation
items: [PATCH v4 02/19] move slabinfo processing to slab_common.c
items: [PATCH v4 03/19] move print_slabinfo_header to slab_common.c
items: [PATCH v4 04/19] sl[au]b: process slabinfo_show in common code
items: [PATCH v4 05/19] slab: don't preemptively remove element from list in cache destroy
items: [PATCH v4 06/19] slab/slub: struct memcg_params
items: [PATCH v4 07/19] consider a memcg parameter in kmem_create_cache
items: [PATCH v4 08/19] Allocate memory for memcg caches whenever a new memcg appears
items: [PATCH v4 09/19] memcg: infrastructure to match an allocation to the right cache
items: [PATCH v4 10/19] memcg: skip memcg kmem allocations in specified code regions
items: [PATCH v4 11/19] sl[au]b: always get the cache from its page in kfree
items: [PATCH v4 12/19] sl[au]b: Allocate objects from memcg cache
items: [PATCH v4 13/19] memcg: destroy memcg caches
items: [PATCH v4 14/19] memcg/sl[au]b Track all the memcg children of a kmem_cache.
items: [PATCH v4 15/19] memcg/sl[au]b: shrink dead caches
items: [PATCH v4 16/19] Aggregate memcg cache values in slabinfo
items: [PATCH v4 17/19] slab: propagate tunables values
items: [PATCH v4 18/19] slub: slub-specific propagation changes.
items: [PATCH v4 19/19] Add slab-specific documentation about the kmem controller
items: Re: [PATCH v4 00/19] slab accounting for memcg

[PATCH v4 00/19] slab accounting for memcg
type: item
description: This is a followup to the previous kmem series. I divided them logically so it gets easier for reviewers. But I believe they are ready to be merged together (although we can do a two-pass merge if people would prefer) Throwaway git tree found at: git://git.kernel.org/pub/scm/linux/kernel/git/glommer/memcg. git kmemcg-slab I've bundled the following important changes since last submission: * no more messing with the cache name after destruction: aggregated figures are shown in /proc/slabinfo. * memory.kmem.slabinfo file with memcg-specific cache information during its lifespan. * full slub attribute propagation. * reusing the standard workqueue mechanism. * cache-side indexing, instead of memcg-side indexing. The memcg css_id serves as an index, and we don't need extra indexes for that. * struct memcg_cache_params no longer bundled in struct kmem_cache: We now will have only a pointer in the struct, allowing memory consumption when disable to fall down ever further. Patches need to be adjusted to cope with those changes, but other than that, look the same - just a lot simpler. I also put quite some effort to overcome my writing disability and get some decent changelogs in place. For a detailed explanation about this whole effort, please refer to my previous post (https://lkml.org/lkml/2012/10/8/119) *** BLURB HERE *** Glauber Costa (19): slab: Ignore internal flags in cache creation move slabinfo processing to slab_common.c move print_slabinfo_header to slab_common.c sl[au]b: process slabinfo_show in common code slab: don't preemptively remove element from list in cache destroy slab/slub: struct memcg_params consider a memcg parameter in kmem_create_cache Allocate memory for memcg caches whenever a new memcg appears memcg: infrastructure to match an allocation to the right cache memcg: skip memcg kmem allocations in specified code regions sl[au]b: always get the cache from its page in kfree sl[au]b: Allocate objects from memcg cache memcg: destroy memcg caches memcg/…
creator: Glauber Costa
date: 2012-10-12T13:40:54-00:00

[PATCH v4 01/19] slab: Ignore internal flags in cache creation
type: item
description: Some flags are used internally by the allocators for management purposes. One example of that is the CFLGS_OFF_SLAB flag that slab uses to mark that the metadata for that cache is stored outside of the slab. No cache should ever pass those as a creation flags. We can just ignore this bit if it happens to be passed (such as when duplicating a cache in the kmem memcg patches). Because such flags can vary from allocator to allocator, we allow them to make their own decisions on that, defining SLAB_AVAILABLE_FLAGS with all flags that are valid at creation time. Allocators that doesn't have any specific flag requirement should define that to mean all flags. Common code will mask out all flags not belonging to that set. [ v2: leave the mask out decision up to the allocators ] [ v3: define flags for all allocators ] [ v4: move all definitions to slab.h ] Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: David Rientjes <rientjes@google.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> --- mm/slab.c | 22 ---------------------- mm/slab.h | 25 +++++++++++++++++++++++++ mm/slab_common.c | 7 +++++++ mm/slub.c | 3 --- 4 files changed, 32 insertions(+), 25 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 87569af..eafef58 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -162,23 +162,6 @@ */ static bool pfmemalloc_active __read_mostly; -/* Legal flag mask for kmem_cache_create(). */ -#if DEBUG -# define CREATE_MASK (SLAB_RED_ZONE | \ - SLAB_POISON | SLAB_HWCACHE_ALIGN | \ - SLAB_CACHE_DMA | \ - SLAB_STORE_USER | \ - SLAB_RECLAIM_ACCOUNT | SLAB_PANIC | \ - SLAB_DESTROY_BY_RCU | SLAB_MEM_SPREAD | \ - SLAB_DEBUG_OBJECTS | SLAB_NOLEAKTRACE | SLAB_NOTRACK) -#else -# define CREATE_MASK (SLAB_HWCACHE_ALIGN | \ - SLAB_CACHE_DMA | \ - SLAB_RECLAIM_ACCOUNT | SLAB_PANIC | \ - SLAB_DESTROY_BY_RCU | SLAB_MEM_SPREAD | \ - SLAB_DEBUG_OBJECTS | SLAB_NOLEAKTRACE | SLAB_NOTRACK) -#endif - /* * kmem_bufctl_t: * @@ -2385,11 +2368,6 @@ __kmem_cache_create (struct k…
creator: Glauber Costa
date: 2012-10-12T13:40:55-00:00

[PATCH v4 02/19] move slabinfo processing to slab_common.c
type: item
description: This patch moves all the common machinery to slabinfo processing to slab_common.c. We can do better by noticing that the output is heavily common, and having the allocators to just provide finished information about this. But after this first step, this can be done easier. Signed-off-by: Glauber Costa <glommer@parallels.com> Acked-by: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: David Rientjes <rientjes@google.com> --- mm/slab.c | 72 ++++++++++---------------------------------------------- mm/slab.h | 8 +++++++ mm/slab_common.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 51 ++++----------------------------------- 4 files changed, 96 insertions(+), 105 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index eafef58..e35970a 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -4263,7 +4263,7 @@ out: #ifdef CONFIG_SLABINFO -static void print_slabinfo_header(struct seq_file *m) +void print_slabinfo_header(struct seq_file *m) { /* * Output format version, so at least we can change it @@ -4286,28 +4286,7 @@ static void print_slabinfo_header(struct seq_file *m) seq_putc(m, '\n'); } -static void *s_start(struct seq_file *m, loff_t *pos) -{ - loff_t n = *pos; - - mutex_lock(&slab_mutex); - if (!n) - print_slabinfo_header(m); - - return seq_list_start(&slab_caches, *pos); -} - -static void *s_next(struct seq_file *m, void *p, loff_t *pos) -{ - return seq_list_next(p, &slab_caches, pos); -} - -static void s_stop(struct seq_file *m, void *p) -{ - mutex_unlock(&slab_mutex); -} - -static int s_show(struct seq_file *m, void *p) +int slabinfo_show(struct seq_file *m, void *p) { struct kmem_cache *cachep = list_entry(p, struct kmem_cache, list); struct slab *slabp; @@ -4404,27 +4383,6 @@ static int s_show(struct seq_file *m, void *p) return 0; } -/* - * slabinfo_op - iterator that generates /proc/slabinfo - * - * Output layout: - * cache-name - * num-active-objs - * total-objs - * object size - * num-active-slabs - * total-slabs - * …
creator: Glauber Costa
date: 2012-10-12T13:40:56-00:00

[PATCH v4 03/19] move print_slabinfo_header to slab_common.c
type: item
description: The header format is highly similar between slab and slub. The main difference lays in the fact that slab may optionally have statistics added here in case of CONFIG_SLAB_DEBUG, while the slub will stick them somewhere else. By making sure that information conditionally lives inside a globally-visible CONFIG_DEBUG_SLAB switch, we can move the header printing to a common location. Signed-off-by: Glauber Costa <glommer@parallels.com> Acked-by: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: David Rientjes <rientjes@google.com> --- mm/slab.c | 24 ------------------------ mm/slab.h | 2 -- mm/slab_common.c | 23 +++++++++++++++++++++++ mm/slub.c | 10 ---------- 4 files changed, 23 insertions(+), 36 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index e35970a..864a9e9 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -4262,30 +4262,6 @@ out: } #ifdef CONFIG_SLABINFO - -void print_slabinfo_header(struct seq_file *m) -{ - /* - * Output format version, so at least we can change it - * without _too_ many complaints. - */ -#if STATS - seq_puts(m, "slabinfo - version: 2.1 (statistics)\n"); -#else - seq_puts(m, "slabinfo - version: 2.1\n"); -#endif - seq_puts(m, "# name <active_objs> <num_objs> <objsize> " - "<objperslab> <pagesperslab>"); - seq_puts(m, " : tunables <limit> <batchcount> <sharedfactor>"); - seq_puts(m, " : slabdata <active_slabs> <num_slabs> <sharedavail>"); -#if STATS - seq_puts(m, " : globalstat <listallocs> <maxobjs> <grown> <reaped> " - "<error> <maxfreeable> <nodeallocs> <remotefrees> <alienoverflow>"); - seq_puts(m, " : cpustat <allochit> <allocmiss> <freehit> <freemiss>"); -#endif - seq_putc(m, '\n'); -} - int slabinfo_show(struct seq_file *m, void *p) { struct kmem_cache *cachep = list_entry(p, struct kmem_cache, list); diff --git a/mm/slab.h b/mm/slab.h index 4156d21..e9ba23f 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -74,8 +74,6 @@ int __kmem_cache_shutdown(struct kmem_cache *); struct seq_file; struct file; -void print_slabinfo_he…
creator: Glauber Costa
date: 2012-10-12T13:40:57-00:00

[PATCH v4 04/19] sl[au]b: process slabinfo_show in common code
type: item
description: With all the infrastructure in place, we can now have slabinfo_show done from slab_common.c. A cache-specific function is called to grab information about the cache itself, since that is still heavily dependent on the implementation. But with the values produced by it, all the printing and handling is done from common code. [ v2: moved objects_per_slab and cache_order to slabinfo ] Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: David Rientjes <rientjes@google.com> --- mm/slab.c | 26 +++++++++++++++----------- mm/slab.h | 16 +++++++++++++++- mm/slab_common.c | 18 +++++++++++++++++- mm/slub.c | 24 ++++++++++-------------- 4 files changed, 57 insertions(+), 27 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 864a9e9..98b3460 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -4262,9 +4262,8 @@ out: } #ifdef CONFIG_SLABINFO -int slabinfo_show(struct seq_file *m, void *p) +void get_slabinfo(struct kmem_cache *cachep, struct slabinfo *sinfo) { - struct kmem_cache *cachep = list_entry(p, struct kmem_cache, list); struct slab *slabp; unsigned long active_objs; unsigned long num_objs; @@ -4319,13 +4318,20 @@ int slabinfo_show(struct seq_file *m, void *p) if (error) printk(KERN_ERR "slab: cache %s error: %s\n", name, error); - seq_printf(m, "%-17s %6lu %6lu %6u %4u %4d", - name, active_objs, num_objs, cachep->size, - cachep->num, (1 << cachep->gfporder)); - seq_printf(m, " : tunables %4u %4u %4u", - cachep->limit, cachep->batchcount, cachep->shared); - seq_printf(m, " : slabdata %6lu %6lu %6lu", - active_slabs, num_slabs, shared_avail); + sinfo->active_objs = active_objs; + sinfo->num_objs = num_objs; + sinfo->active_slabs = active_slabs; + sinfo->num_slabs = num_slabs; + sinfo->shared_avail = shared_avail; + sinfo->limit = cachep->limit; + sinfo->batchcount = cachep->batchcount; + sinfo->shared = cachep->shared; + sinfo->objects_per_slab = cachep->num; + sinfo->cache_order = cachep->gfporder…
creator: Glauber Costa
date: 2012-10-12T13:40:58-00:00

[PATCH v4 05/19] slab: don't preemptively remove element from list in cache destroy
type: item
description: After the slab/slub/slob merge, we are deleting the element from the slab_cache lists, and then if the destruction fail, we add it back again. This behavior was present in some caches, but not in others, if my memory doesn't fail me. I, however, see no reason why we need to do so, since we are now locked during the whole deletion (which wasn't necessarily true before). I propose a simplification in which we delete it only when there is no more going back, so we don't need to add it again. Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- mm/slab_common.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 1ee1d6f..bf4b4f1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -174,16 +174,15 @@ void kmem_cache_destroy(struct kmem_cache *s) mutex_lock(&slab_mutex); s->refcount--; if (!s->refcount) { - list_del(&s->list); - if (!__kmem_cache_shutdown(s)) { if (s->flags & SLAB_DESTROY_BY_RCU) rcu_barrier(); + list_del(&s->list); + kfree(s->name); kmem_cache_free(kmem_cache, s); } else { - list_add(&s->list, &slab_caches); printk(KERN_ERR "kmem_cache_destroy %s: Slab cache still has objects\n", s->name); dump_stack(); -- 1.7.11.4
creator: Glauber Costa
date: 2012-10-12T13:40:59-00:00

[PATCH v4 06/19] slab/slub: struct memcg_params
type: item
description: For the kmem slab controller, we need to record some extra information in the kmem_cache structure. Signed-off-by: Glauber Costa <glommer@parallels.com> Signed-off-by: Suleiman Souhlal <suleiman@google.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Tejun Heo <tj@kernel.org> --- include/linux/slab.h | 25 +++++++++++++++++++++++++ include/linux/slab_def.h | 3 +++ include/linux/slub_def.h | 3 +++ mm/slab.h | 13 +++++++++++++ 4 files changed, 44 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 0dd2dfa..e4ea48a 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -177,6 +177,31 @@ unsigned int kmem_cache_size(struct kmem_cache *); #define ARCH_SLAB_MINALIGN __alignof__(unsigned long long) #endif +#include <linux/workqueue.h> +/* + * This is the main placeholder for memcg-related information in kmem caches. + * struct kmem_cache will hold a pointer to it, so the memory cost while + * disabled is 1 pointer. The runtime cost while enabled, gets bigger than it + * would otherwise be if that would be bundled in kmem_cache: we'll need an + * extra pointer chase. But the trade off clearly lays in favor of not + * penalizing non-users. + * + * Both the root cache and the child caches will have it. For the root cache, + * this will hold a dynamically allocated array large enough to hold + * information about the currently limited memcgs in the system. + * + * Child caches will hold extra metadata needed for its operation. Fields are: + * + * @memcg: pointer to the memcg this cache belongs to + */ +struct memcg_cache_params { + bool is_root_cache; + union { + struct kmem_cache *memcg_caches[0]; + struct mem_cgroup *memcg; + }; +}; + /* * Common kmalloc functions provided by all allocators */ diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h index 36d7031..665a…
creator: Glauber Costa
date: 2012-10-12T13:41:00-00:00

[PATCH v4 07/19] consider a memcg parameter in kmem_create_cache
type: item
description: Allow a memcg parameter to be passed during cache creation. When the slub allocator is being used, it will only merge caches that belong to the same memcg. Default function is created as a wrapper, passing NULL to the memcg version. We only merge caches that belong to the same memcg. A helper is provided, memcg_css_id: because slub needs a unique cache name for sysfs. Since this is visible, but not the canonical location for slab data, the cache name is not used, the css_id should suffice. [ v2: moved to idr/ida instead of redoing the indexes ] [ v3: moved call to ida_init away from cgroup creation to fix a bug ] [ v4: no longer using the index mechanism ] Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/memcontrol.h | 22 +++++++++++++++++++++ include/linux/slab.h | 12 +++++++++++- mm/memcontrol.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 23 ++++++++++++++++++---- mm/slab_common.c | 40 +++++++++++++++++++++++++++---------- mm/slub.c | 16 +++++++++++---- 6 files changed, 143 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 34e96cf..957da60 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -28,6 +28,7 @@ struct mem_cgroup; struct page_cgroup; struct page; struct mm_struct; +struct kmem_cache; /* Stats that can be updated by kernel. */ enum mem_cgroup_page_stat_item { @@ -414,6 +415,11 @@ void __memcg_kmem_commit_charge(struct page *page, struct mem_cgroup *memcg, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +int memcg_css_id(struct mem_cgroup *memcg); +int memcg_register_cache(struct mem_cgroup *memcg, struct kmem_cache *s); +void memcg_…
creator: Glauber Costa
date: 2012-10-12T13:41:01-00:00

[PATCH v4 08/19] Allocate memory for memcg caches whenever a new memcg appears
type: item
description: Every cache that is considered a root cache (basically the "original" caches, tied to the root memcg/no-memcg) will have an array that should be large enough to store a cache pointer per each memcg in the system. Theoreticaly, this is as high as 1 << sizeof(css_id), which is currently in the 64k pointers range. Most of the time, we won't be using that much. What goes in this patch, is a simple scheme to dynamically allocate such an array, in order to minimize memory usage for memcg caches. Because we would also like to avoid allocations all the time, at least for now, the array will only grow. It will tend to be big enough to hold the maximum number of kmem-limited memcgs ever achieved. We'll allocate it to be a minimum of 64 kmem-limited memcgs. When we have more than that, we'll start doubling the size of this array every time the limit is reached. Because we are only considering kmem limited memcgs, a natural point for this to happen is when we write to the limit. At that point, we already have set_limit_mutex held, so that will become our natural synchronization mechanism. Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/memcontrol.h | 2 + mm/memcontrol.c | 171 ++++++++++++++++++++++++++++++++++++++++----- mm/slab_common.c | 25 +++++++ 3 files changed, 181 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 957da60..e8d0571 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -420,6 +420,8 @@ int memcg_register_cache(struct mem_cgroup *memcg, struct kmem_cache *s); void memcg_release_cache(struct kmem_cache *cachep); void memcg_cache_list_add(struct mem_cgroup *memcg, struct kmem_cache *c…
creator: Glauber Costa
date: 2012-10-12T13:41:02-00:00

[PATCH v4 09/19] memcg: infrastructure to match an allocation to the right cache
type: item
description: The page allocator is able to bind a page to a memcg when it is allocated. But for the caches, we'd like to have as many objects as possible in a page belonging to the same cache. This is done in this patch by calling memcg_kmem_get_cache in the beginning of every allocation function. This routing is patched out by static branches when kernel memory controller is not being used. It assumes that the task allocating, which determines the memcg in the page allocator, belongs to the same cgroup throughout the whole process. Misacounting can happen if the task calls memcg_kmem_get_cache() while belonging to a cgroup, and later on changes. This is considered acceptable, and should only happen upon task migration. Before the cache is created by the memcg core, there is also a possible imbalance: the task belongs to a memcg, but the cache being allocated from is the global cache, since the child cache is not yet guaranteed to be ready. This case is also fine, since in this case the GFP_KMEMCG will not be passed and the page allocator will not attempt any cgroup accounting. [ v4: use a standard workqueue mechanism, create right away if possible, index from cache side ] Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/memcontrol.h | 42 ++++++++++ init/Kconfig | 2 +- mm/memcontrol.c | 188 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 231 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e8d0571..c7886aa 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -422,6 +422,10 @@ void memcg_cache_list_add(struct mem_cgroup *memcg, struct kmem_cache *cachep); int memcg_update_cache_size(struct …
creator: Glauber Costa
date: 2012-10-12T13:41:03-00:00

[PATCH v4 10/19] memcg: skip memcg kmem allocations in specified code regions
type: item
description: This patch creates a mechanism that skip memcg allocations during certain pieces of our core code. It basically works in the same way as preempt_disable()/preempt_enable(): By marking a region under which all allocations will be accounted to the root memcg. We need this to prevent races in early cache creation, when we allocate data using caches that are not necessarily created already. [ v2: wrap the whole enqueue process, INIT_WORK can alloc memory ] Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/sched.h | 1 + mm/memcontrol.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 62 insertions(+), 2 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 0d907e1..9fad6c1 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1581,6 +1581,7 @@ struct task_struct { unsigned long nr_pages; /* uncharged usage */ unsigned long memsw_nr_pages; /* uncharged mem+swap usage */ } memcg_batch; + unsigned int memcg_kmem_skip_account; #endif #ifdef CONFIG_HAVE_HW_BREAKPOINT atomic_t ptrace_bp_refcnt; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 148baad..96916bd 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2949,6 +2949,41 @@ out: kfree(s->memcg_params); } +/* + * During the creation a new cache, we need to disable our accounting mechanism + * altogether. This is true even if we are not creating, but rather just + * enqueing new caches to be created. + * + * This is because that process will trigger allocations; some visible, like + * explicit kmallocs to auxiliary data structures, name strings and internal + * cache structures; some well concealed, like INIT_WORK() that can allocate + * objects during debug.…
creator: Glauber Costa
date: 2012-10-12T13:41:04-00:00

[PATCH v4 11/19] sl[au]b: always get the cache from its page in kfree
type: item
description: struct page already have this information. If we start chaining caches, this information will always be more trustworthy than whatever is passed into the function A parent pointer is added to the slub structure, so we can make sure the freeing comes from either the right slab, or from its rightful parent. [ v3: added parent testing with VM_BUG_ON ] [ v4: make it faster when kmemcg not in use ] Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/memcontrol.h | 4 ++++ mm/slab.c | 17 ++++++++++++++++- mm/slab.h | 13 +++++++++++++ mm/slub.c | 14 ++++++++++++-- 4 files changed, 45 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index c7886aa..4c94182 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -536,6 +536,10 @@ static inline void sock_release_memcg(struct sock *sk) { } +static inline bool memcg_kmem_enabled(void) +{ + return false; +} static inline bool memcg_kmem_newpage_charge(gfp_t gfp, struct mem_cgroup **memcg, int order) { diff --git a/mm/slab.c b/mm/slab.c index 98b3460..6f22067 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3911,9 +3911,24 @@ EXPORT_SYMBOL(__kmalloc); * Free an object which was previously allocated from this * cache. */ -void kmem_cache_free(struct kmem_cache *cachep, void *objp) +void kmem_cache_free(struct kmem_cache *s, void *objp) { unsigned long flags; + struct kmem_cache *cachep; + + /* + * When kmemcg is not being used, both assignments should return the + * same value. but we don't want to pay the assignment price in that + * case. If it is not compiled in, the compiler should be smart en…
creator: Glauber Costa
date: 2012-10-12T13:41:05-00:00

[PATCH v4 12/19] sl[au]b: Allocate objects from memcg cache
type: item
description: We are able to match a cache allocation to a particular memcg. If the task doesn't change groups during the allocation itself - a rare event, this will give us a good picture about who is the first group to touch a cache page. This patch uses the now available infrastructure by calling memcg_kmem_get_cache() before all the cache allocations. Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/slub_def.h | 15 ++++++++++----- mm/memcontrol.c | 3 +++ mm/slab.c | 6 +++++- mm/slub.c | 5 +++-- 4 files changed, 21 insertions(+), 8 deletions(-) diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 961e72e..ed330df 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -13,6 +13,8 @@ #include <linux/kobject.h> #include <linux/kmemleak.h> +#include <linux/memcontrol.h> +#include <linux/mm.h> enum stat_item { ALLOC_FASTPATH, /* Allocation from cpu slab */ @@ -209,14 +211,14 @@ static __always_inline int kmalloc_index(size_t size) * This ought to end up with a global pointer to the right cache * in kmalloc_caches. */ -static __always_inline struct kmem_cache *kmalloc_slab(size_t size) +static __always_inline struct kmem_cache *kmalloc_slab(gfp_t flags, size_t size) { int index = kmalloc_index(size); if (index == 0) return NULL; - return kmalloc_caches[index]; + return memcg_kmem_get_cache(kmalloc_caches[index], flags); } void *kmem_cache_alloc(struct kmem_cache *, gfp_t); @@ -225,7 +227,10 @@ void *__kmalloc(size_t size, gfp_t flags); static __always_inline void * kmalloc_order(size_t size, gfp_t flags, unsigned int order) { - void *ret = (void *) __get_free_pages(flags | __GFP_COMP, order); + void *ret; + + flags |= (__GFP_COMP | __GFP_KMEMCG); + r…
creator: Glauber Costa
date: 2012-10-12T13:41:06-00:00

[PATCH v4 13/19] memcg: destroy memcg caches
type: item
description: This patch implements destruction of memcg caches. Right now, only caches where our reference counter is the last remaining are deleted. If there are any other reference counters around, we just leave the caches lying around until they go away. When that happen, a destruction function is called from the cache code. Caches are only destroyed in process context, so we queue them up for later processing in the general case. Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/memcontrol.h | 2 ++ include/linux/slab.h | 9 +++++++++ mm/memcontrol.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++ mm/slab.c | 3 +++ mm/slab.h | 24 +++++++++++++++++++++++ mm/slub.c | 7 ++++++- 6 files changed, 91 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4c94182..9ac12cb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -426,6 +426,8 @@ void memcg_update_array_size(int num_groups); struct kmem_cache * __memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp); +void mem_cgroup_destroy_cache(struct kmem_cache *cachep); + /** * memcg_kmem_newpage_charge: verify if a new kmem allocation is allowed. * @gfp: the gfp allocation flags. diff --git a/include/linux/slab.h b/include/linux/slab.h index b22a158..e17d348 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -198,6 +198,11 @@ unsigned int kmem_cache_size(struct kmem_cache *); * * @memcg: pointer to the memcg this cache belongs to * @root_cache: pointer to the global, root cache, this cache was derived from + * @cachep: backpointer to the kmem_cache structure that hold us. + * @dead: set to true after the memcg dies; the cache may still be aroun…
creator: Glauber Costa
date: 2012-10-12T13:41:07-00:00

[PATCH v4 14/19] memcg/sl[au]b Track all the memcg children of a kmem_cache.
type: item
description: This enables us to remove all the children of a kmem_cache being destroyed, if for example the kernel module it's being used in gets unloaded. Otherwise, the children will still point to the destroyed parent. Signed-off-by: Suleiman Souhlal <suleiman@google.com> Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Tejun Heo <tj@kernel.org> --- include/linux/memcontrol.h | 6 ++++++ mm/memcontrol.c | 32 ++++++++++++++++++++++++++++++-- mm/slab_common.c | 3 +++ 3 files changed, 39 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 9ac12cb..6c8156b 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -427,6 +427,7 @@ struct kmem_cache * __memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp); void mem_cgroup_destroy_cache(struct kmem_cache *cachep); +void kmem_cache_destroy_memcg_children(struct kmem_cache *s); /** * memcg_kmem_newpage_charge: verify if a new kmem allocation is allowed. @@ -578,6 +579,11 @@ memcg_kmem_get_cache(struct kmem_cache *cachep, gfp_t gfp) { return cachep; } + + +static inline void kmem_cache_destroy_memcg_children(struct kmem_cache *s) +{ +} #endif /* CONFIG_MEMCG_KMEM */ #endif /* _LINUX_MEMCONTROL_H */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f744305..8cf8b4d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2716,6 +2716,8 @@ static void __mem_cgroup_commit_charge(struct mem_cgroup *memcg, memcg_check_events(memcg, page); } +static DEFINE_MUTEX(set_limit_mutex); + #ifdef CONFIG_MEMCG_KMEM static inline bool memcg_can_account_kmem(struct mem_cgroup *memcg) { @@ -3093,6 +3095,34 @@ out: return new_cachep; } +void kmem_cache_destroy_memcg_children(struct kmem_cache *s) +{ + struct kmem_cache *c; + int i; + + if (!s->memcg_params) + return…
creator: Glauber Costa
date: 2012-10-12T13:41:08-00:00

[PATCH v4 15/19] memcg/sl[au]b: shrink dead caches
type: item
description: In the slub allocator, when the last object of a page goes away, we don't necessarily free it - there is not necessarily a test for empty page in any slab_free path. This means that when we destroy a memcg cache that happened to be empty, those caches may take a lot of time to go away: removing the memcg reference won't destroy them - because there are pending references, and the empty pages will stay there, until a shrinker is called upon for any reason. This patch marks all memcg caches as dead. kmem_cache_shrink is called for the ones who are not yet dead - this will force internal cache reorganization, and then all references to empty pages will be removed. An unlikely branch is used to make sure this case does not affect performance in the usual slab_free path. The slab allocator has a time based reaper that would eventually get rid of the objects, but we can also call it explicitly, since dead caches are not a likely event. [ v2: also call verify_dead for the slab ] [ v3: use delayed_work to avoid calling verify_dead at every free] Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/slab.h | 2 +- mm/memcontrol.c | 28 ++++++++++++++++++++++------ 2 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index e17d348..9ed1a98 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -214,7 +214,7 @@ struct memcg_cache_params { struct kmem_cache *cachep; bool dead; atomic_t nr_pages; - struct work_struct destroy; + struct delayed_work destroy; }; }; }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8cf8b4d..b52e6b9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2955,11 +2955,27 @@ static void kmem_c…
creator: Glauber Costa
date: 2012-10-12T13:41:09-00:00

[PATCH v4 16/19] Aggregate memcg cache values in slabinfo
type: item
description: When we create caches in memcgs, we need to display their usage information somewhere. We'll adopt a scheme similar to /proc/meminfo, with aggregate totals shown in the global file, and per-group information stored in the group itself. For the time being, only reads are allowed in the per-group cache. Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/memcontrol.h | 8 ++++++++ include/linux/slab.h | 3 +++ mm/memcontrol.c | 34 +++++++++++++++++++++++++++++++++- mm/slab.h | 27 +++++++++++++++++++++++++++ mm/slab_common.c | 42 ++++++++++++++++++++++++++++++++++++++---- 5 files changed, 109 insertions(+), 5 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6c8156b..5dd366e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -404,6 +404,11 @@ void sock_update_memcg(struct sock *sk); void sock_release_memcg(struct sock *sk); extern struct static_key memcg_kmem_enabled_key; + +extern int memcg_limited_groups_array_size; +#define for_each_memcg_cache_index(_idx) \ + for ((_idx) = 0; i < memcg_limited_groups_array_size; (_idx)++) + static inline bool memcg_kmem_enabled(void) { return static_key_false(&memcg_kmem_enabled_key); @@ -539,6 +544,9 @@ static inline void sock_release_memcg(struct sock *sk) { } +#define for_each_memcg_cache_index(_idx) \ + for (; NULL; ) + static inline bool memcg_kmem_enabled(void) { return false; diff --git a/include/linux/slab.h b/include/linux/slab.h index 9ed1a98..b7eea06 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -221,6 +221,9 @@ struct memcg_cache_params { int memcg_update_all_caches(int num_memcgs); +int cache_show(struct kmem_cache *s, struct seq_file *m); +void …
creator: Glauber Costa
date: 2012-10-12T13:41:10-00:00

[PATCH v4 17/19] slab: propagate tunables values
type: item
description: SLAB allows us to tune a particular cache behavior with tunables. When creating a new memcg cache copy, we'd like to preserve any tunables the parent cache already had. This could be done by an explicit call to do_tune_cpucache() after the cache is created. But this is not very convenient now that the caches are created from common code, since this function is SLAB-specific. Another method of doing that is taking advantage of the fact that do_tune_cpucache() is always called from enable_cpucache(), which is called at cache initialization. We can just preset the values, and then things work as expected. It can also happen that a root cache has its tunables updated during normal system operation. In this case, we will propagate the change to all caches that are already active. This change will require us to move the assignment of root_cache in memcg_params a bit earlier. We need this to be already set - which memcg_kmem_register_cache will do - when we reach __kmem_cache_create() Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/memcontrol.h | 8 +++++--- include/linux/slab.h | 2 +- mm/memcontrol.c | 10 ++++++---- mm/slab.c | 44 +++++++++++++++++++++++++++++++++++++++++--- mm/slab.h | 12 ++++++++++++ mm/slab_common.c | 7 ++++--- 6 files changed, 69 insertions(+), 14 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 5dd366e..09c2917 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -421,7 +421,8 @@ void __memcg_kmem_commit_charge(struct page *page, void __memcg_kmem_uncharge_page(struct page *page, int order); int memcg_css_id(struct mem_cgroup *memcg); -int memcg_register_cache(struct mem_cgroup *memcg, struc…
creator: Glauber Costa
date: 2012-10-12T13:41:11-00:00

[PATCH v4 18/19] slub: slub-specific propagation changes.
type: item
description: SLUB allows us to tune a particular cache behavior with sysfs-based tunables. When creating a new memcg cache copy, we'd like to preserve any tunables the parent cache already had. This can be done by tapping into the store attribute function provided by the allocator. We of course don't need to mess with read-only fields. Since the attributes can have multiple types and are stored internally by sysfs, the best strategy is to issue a ->show() in the root cache, and then ->store() in the memcg cache. The drawback of that, is that sysfs can allocate up to a page in buffering for show(), that we are likely not to need, but also can't guarantee. To avoid always allocating a page for that, we can update the caches at store time with the maximum attribute size ever stored to the root cache. We will then get a buffer big enough to hold it. The corolary to this, is that if no stores happened, nothing will be propagated. It can also happen that a root cache has its tunables updated during normal system operation. In this case, we will propagate the change to all caches that are already active. Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- include/linux/slub_def.h | 1 + mm/slub.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index ed330df..f41acb9 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -105,6 +105,7 @@ struct kmem_cache { #endif #ifdef CONFIG_MEMCG_KMEM struct memcg_cache_params *memcg_params; + int max_attr_size; /* for propagation, maximum size of a stored attr */ #endif #ifdef CONFIG_NUMA diff --git a/mm/slub.c b/mm/slub.c index e98fdf0..617d7ad 10…
creator: Glauber Costa
date: 2012-10-12T13:41:12-00:00

[PATCH v4 19/19] Add slab-specific documentation about the kmem controller
type: item
description: Signed-off-by: Glauber Costa <glommer@parallels.com> CC: Christoph Lameter <cl@linux.com> CC: Pekka Enberg <penberg@cs.helsinki.fi> CC: Michal Hocko <mhocko@suse.cz> CC: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> CC: Johannes Weiner <hannes@cmpxchg.org> CC: Suleiman Souhlal <suleiman@google.com> CC: Tejun Heo <tj@kernel.org> --- Documentation/cgroups/memory.txt | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/Documentation/cgroups/memory.txt b/Documentation/cgroups/memory.txt index dd15be8..95e4809 100644 --- a/Documentation/cgroups/memory.txt +++ b/Documentation/cgroups/memory.txt @@ -300,6 +300,13 @@ to trigger slab reclaim when those limits are reached. kernel memory, we prevent new processes from being created when the kernel memory usage is too high. +* slab pages: pages allocated by the SLAB or SLUB allocator are tracked. A copy +of each kmem_cache is created everytime the cache is touched by the first time +from inside the memcg. The creation is done lazily, so some objects can still be +skipped while the cache is being created. All objects in a slab page should +belong to the same memcg. This only fails to hold when a task is migrated to a +different memcg during the page allocation by the cache. + * sockets memory pressure: some sockets protocols have memory pressure thresholds. The Memory Controller allows them to be controlled individually per cgroup, instead of globally. -- 1.7.11.4
creator: Glauber Costa
date: 2012-10-12T13:41:13-00:00

Re: [PATCH v4 00/19] slab accounting for memcg
type: item
description: On 10/12/2012 05:40 PM, Glauber Costa wrote: > This is a followup to the previous kmem series. I divided them logically > so it gets easier for reviewers. But I believe they are ready to be merged > together (although we can do a two-pass merge if people would prefer) > > Throwaway git tree found at: > > git://git.kernel.org/pub/scm/linux/kernel/git/glommer/memcg. git kmemcg-slab > To all reviewers of my previous series (and others as well), I'd like to draw attention to this one. This is the follow up to the kmemcg-stack series, to be applied right ontop. I believe the first series (kmemcg-stack) got more attention (which is good), but this one got a bunch of fixes and reviews in the process as well. Thanks
creator: Glauber Costa
date: 2012-10-18T11:38:44-00:00