Home » Mailing lists » Users » URGENT! process in container invokes PF to OOM-KILLER with no clear reason???
| URGENT! process in container invokes PF to OOM-KILLER with no clear reason??? [message #47677] |
Fri, 24 August 2012 09:13  |
Stoyan Stoyanov
Messages: 11
Registered: November 2011
|
Junior Member |
|
|
Hi list,
We used openvz in production servers from 1-2 years ago, but now
really need your suggestions and help. So this is the situation.
We have setup with two identical openvz servers, and they was working
perfectly fine until a the problem rise, so this is the problem:
In a random time interval ( 1-2-3 hours ), the system (on the HN)
lunch OOM-KILLER and start killing processes in different containers
(so far not so unusual) , but the real problem is that 1. on the HN
there is a lot of free memory resources and 2. in the VPS containers
there is also free memory resources.
From the log analyze it's clear that random processes do PF , which
invoke the oom-killer , but it's really must NOT to do it.
So, there are the /var/log/messages messages:
Aug 23 21:40:09 vz2 kernel: [13444137.871993] postgres invoked oom-
killer: gfp_mask=0x0, order=0, oom_adj=0
Aug 23 21:40:09 vz2 kernel: [13444137.871995] Pid: 653, comm: postgres
Not tainted 2.6.32-5-openvz-amd64 #1
Aug 23 21:40:09 vz2 kernel: [13444137.871997] Call Trace:
Aug 23 21:40:09 vz2 kernel: [13444137.872002] [<ffffffff810b8eec>] ?
oom_kill_process+0x75/0x22c
Aug 23 21:40:09 vz2 kernel: [13444137.872004] [<ffffffff810b940e>] ?
__out_of_memory+0x93/0xad
Aug 23 21:40:09 vz2 kernel: [13444137.872006] [<ffffffff810b969a>] ?
pagefault_out_of_memory+0xba/0xe8
Aug 23 21:40:09 vz2 kernel: [13444137.872009] [<ffffffff81032612>] ?
mm_fault_error+0x5c/0x142
Aug 23 21:40:09 vz2 kernel: [13444137.872012] [<ffffffff812ed805>] ?
do_page_fault+0x2e0/0x2fc
Aug 23 21:40:09 vz2 kernel: [13444137.872016] [<ffffffff812eb695>] ?
page_fault+0x25/0x30
Aug 23 21:40:09 vz2 kernel: [13444137.872017] Mem-Info:
Aug 23 21:40:09 vz2 kernel: [13444137.872018] Node 0 DMA per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.872020] CPU 0: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872021] CPU 1: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872022] CPU 2: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872024] CPU 3: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872025] CPU 4: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872026] CPU 5: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872028] CPU 6: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872029] CPU 7: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872030] Node 0 DMA32 per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.872031] CPU 0: hi: 186,
btch: 31 usd: 165
Aug 23 21:40:09 vz2 kernel: [13444137.872033] CPU 1: hi: 186,
btch: 31 usd: 12
Aug 23 21:40:09 vz2 kernel: [13444137.872034] CPU 2: hi: 186,
btch: 31 usd: 170
Aug 23 21:40:09 vz2 kernel: [13444137.872035] CPU 3: hi: 186,
btch: 31 usd: 156
Aug 23 21:40:09 vz2 kernel: [13444137.872037] CPU 4: hi: 186,
btch: 31 usd: 157
Aug 23 21:40:09 vz2 kernel: [13444137.872038] CPU 5: hi: 186,
btch: 31 usd: 155
Aug 23 21:40:09 vz2 kernel: [13444137.872039] CPU 6: hi: 186,
btch: 31 usd: 157
Aug 23 21:40:09 vz2 kernel: [13444137.872040] CPU 7: hi: 186,
btch: 31 usd: 174
Aug 23 21:40:09 vz2 kernel: [13444137.872041] Node 0 Normal per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.872043] CPU 0: hi: 186,
btch: 31 usd: 108
Aug 23 21:40:09 vz2 kernel: [13444137.872044] CPU 1: hi: 186,
btch: 31 usd: 161
Aug 23 21:40:09 vz2 kernel: [13444137.872045] CPU 2: hi: 186,
btch: 31 usd: 163
Aug 23 21:40:09 vz2 kernel: [13444137.872046] CPU 3: hi: 186,
btch: 31 usd: 127
Aug 23 21:40:09 vz2 kernel: [13444137.872048] CPU 4: hi: 186,
btch: 31 usd: 139
Aug 23 21:40:09 vz2 kernel: [13444137.872049] CPU 5: hi: 186,
btch: 31 usd: 171
Aug 23 21:40:09 vz2 kernel: [13444137.872050] CPU 6: hi: 186,
btch: 31 usd: 133
Aug 23 21:40:09 vz2 kernel: [13444137.872052] CPU 7: hi: 186,
btch: 31 usd: 172
Aug 23 21:40:09 vz2 kernel: [13444137.872055] active_anon:127099
inactive_anon:16297 isolated_anon:0
Aug 23 21:40:09 vz2 kernel: [13444137.872056] active_file:668874
inactive_file:2839950 isolated_file:0
Aug 23 21:40:09 vz2 kernel: [13444137.872056] unevictable:0 dirty:
3089 writeback:0 unstable:0
Aug 23 21:40:09 vz2 kernel: [13444137.872057] free:225934
slab_reclaimable:166494 slab_unreclaimable:13922
Aug 23 21:40:09 vz2 kernel: [13444137.872058] mapped:13973 shmem:1139
pagetables:8974 bounce:0
Aug 23 21:40:09 vz2 kernel: [13444137.872059] Node 0 DMA free:15800kB
min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15188kB mlocked:0kB dirty:0kB writeback:0kB
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:
0kB pages_scanned:0 all_unreclaimable? yes
Aug 23 21:40:09 vz2 kernel: [13444137.872067] lowmem_reserve[]: 0 2982
16086 16086
Aug 23 21:40:09 vz2 kernel: [13444137.872069] Node 0 DMA32 free:
313340kB min:3008kB low:3760kB high:4512kB active_anon:844kB
inactive_anon:3524kB active_file:259008kB inactive_file:2010296kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:
3054420kB mlocked:0kB dirty:32kB writeback:0kB mapped:4344kB shmem:0kB
slab_reclaimable:181804kB slab_unreclaimable:9844kB kernel_stack:408kB
pagetables:156kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
Aug 23 21:40:09 vz2 kernel: [13444137.872077] lowmem_reserve[]: 0 0
13104 13104
Aug 23 21:40:09 vz2 kernel: [13444137.872078] Node 0 Normal free:
574596kB min:13216kB low:16520kB high:19824kB active_anon:507552kB
inactive_anon:61664kB active_file:2416488kB inactive_file:9349504kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:
13418496kB mlocked:0kB dirty:12324kB writeback:0kB mapped:51548kB
shmem:4556kB slab_reclaimable:484172kB slab_unreclaimable:45844kB
kernel_stack:4464kB pagetables:35740kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug 23 21:40:09 vz2 kernel: [13444137.872086] lowmem_reserve[]: 0 0 0 0
Aug 23 21:40:09 vz2 kernel: [13444137.872088] Node 0 DMA: 2*4kB 2*8kB
0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
3*4096kB = 15800kB
Aug 23 21:40:09 vz2 kernel: [13444137.872093] Node 0 DMA32: 20691*4kB
8508*8kB 4799*16kB 1613*32kB 405*64kB 67*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 313724kB
Aug 23 21:40:09 vz2 kernel: [13444137.872099] Node 0 Normal: 71219*4kB
27806*8kB 1748*16kB 1207*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
0*2048kB 0*4096kB = 574940kB
Aug 23 21:40:09 vz2 kernel: [13444137.872104] 3512939 total pagecache
pages
Aug 23 21:40:09 vz2 kernel: [13444137.872105] 2963 pages in swap cache
Aug 23 21:40:09 vz2 kernel: [13444137.872106] Swap cache stats: add
58675, delete 55712, find 294564024/294568150
Aug 23 21:40:09 vz2 kernel: [13444137.872108] Free swap = 31207600kB
Aug 23 21:40:09 vz2 kernel: [13444137.872109] Total swap = 31247348kB
Aug 23 21:40:09 vz2 kernel: [13444137.909766] 4194288 pages RAM
Aug 23 21:40:09 vz2 kernel: [13444137.909768] 86418 pages reserved
Aug 23 21:40:09 vz2 kernel: [13444137.909769] 1510860 pages shared
Aug 23 21:40:09 vz2 kernel: [13444137.909770] 2659168 pages non-shared
Aug 23 21:40:09 vz2 kernel: [13444137.909845] Mem-Info:
Aug 23 21:40:09 vz2 kernel: [13444137.909847] Node 0 DMA per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.909850] CPU 0: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909852] CPU 1: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909855] CPU 2: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909857] CPU 3: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909859] CPU 4: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909862] CPU 5: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909864] CPU 6: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909866] CPU 7: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909868] Node 0 DMA32 per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.909871] CPU 0: hi: 186,
btch: 31 usd: 165
Aug 23 21:40:09 vz2 kernel: [13444137.909873] CPU 1: hi: 186,
btch: 31 usd: 12
Aug 23 21:40:09 vz2 kernel: [13444137.909876] CPU 2: hi: 186,
btch: 31 usd: 170
Aug 23 21:40:09 vz2 kernel: [13444137.909878] CPU 3: hi: 186,
btch: 31 usd: 156
Aug 23 21:40:09 vz2 kernel: [13444137.909880] CPU 4: hi: 186,
btch: 31 usd: 157
Aug 23 21:40:09 vz2 kernel: [13444137.909883] CPU 5: hi: 186,
btch: 31 usd: 155
Aug 23 21:40:09 vz2 kernel: [13444137.909885] CPU 6: hi: 186,
btch: 31 usd: 157
Aug 23 21:40:09 vz2 kernel: [13444137.909887] CPU 7: hi: 186,
btch: 31 usd: 174
Aug 23 21:40:09 vz2 kernel: [13444137.909889] Node 0 Normal per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.909892] CPU 0: hi: 186,
btch: 31 usd: 108
Aug 23 21:40:09 vz2 kernel: [13444137.909894] CPU 1: hi: 186,
btch: 31 usd: 161
Aug 23 21:40:09 vz2 kernel: [13444137.909897] CPU 2: hi: 186,
btch: 31 usd: 163
Aug 23 21:40:09 vz2 kernel: [13444137.909899] CPU 3: hi: 186,
btch: 31 usd: 127
Aug 23 21:40:09 vz2 kernel: [13444137.909901] CPU 4: hi: 186,
btch: 31 usd: 139
Aug 23 21:40:09 vz2 kernel: [13444137.909903] CPU 5: hi: 186,
btch: 31 usd: 171
Aug 23 21:40:09 vz2 kernel: [13444137.909906] CPU 6: hi: 186,
btch: 31 usd: 133
Aug 23 21:40:09 vz2 kernel: [13444137.909908] CPU 7: hi: 186,
btch: 31 usd: 173
Aug 23 21:40:09 vz2 kernel: [13444137.909912] active_anon:127099
inactive_anon:16297 isolated_anon:0
Aug 23 21:40:09 vz2 kernel: [13444137.909914] active_file:668874
inactive_file:2839950 iso
...
|
|
|
|
Current Time: Fri Nov 08 22:33:51 GMT 2024
Total time taken to generate the page: 0.03219 seconds
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
