Home » Mailing lists » Users » URGENT! process in container invokes PF to OOM-KILLER with no clear reason???
| URGENT! process in container invokes PF to OOM-KILLER with no clear reason??? [message #47677] |
Fri, 24 August 2012 09:13  |
Stoyan Stoyanov
Messages: 11
Registered: November 2011
|
Junior Member |
|
|
Hi list,
We used openvz in production servers from 1-2 years ago, but now
really need your suggestions and help. So this is the situation.
We have setup with two identical openvz servers, and they was working
perfectly fine until a the problem rise, so this is the problem:
In a random time interval ( 1-2-3 hours ), the system (on the HN)
lunch OOM-KILLER and start killing processes in different containers
(so far not so unusual) , but the real problem is that 1. on the HN
there is a lot of free memory resources and 2. in the VPS containers
there is also free memory resources.
From the log analyze it's clear that random processes do PF , which
invoke the oom-killer , but it's really must NOT to do it.
So, there are the /var/log/messages messages:
Aug 23 21:40:09 vz2 kernel: [13444137.871993] postgres invoked oom-
killer: gfp_mask=0x0, order=0, oom_adj=0
Aug 23 21:40:09 vz2 kernel: [13444137.871995] Pid: 653, comm: postgres
Not tainted 2.6.32-5-openvz-amd64 #1
Aug 23 21:40:09 vz2 kernel: [13444137.871997] Call Trace:
Aug 23 21:40:09 vz2 kernel: [13444137.872002] [<ffffffff810b8eec>] ?
oom_kill_process+0x75/0x22c
Aug 23 21:40:09 vz2 kernel: [13444137.872004] [<ffffffff810b940e>] ?
__out_of_memory+0x93/0xad
Aug 23 21:40:09 vz2 kernel: [13444137.872006] [<ffffffff810b969a>] ?
pagefault_out_of_memory+0xba/0xe8
Aug 23 21:40:09 vz2 kernel: [13444137.872009] [<ffffffff81032612>] ?
mm_fault_error+0x5c/0x142
Aug 23 21:40:09 vz2 kernel: [13444137.872012] [<ffffffff812ed805>] ?
do_page_fault+0x2e0/0x2fc
Aug 23 21:40:09 vz2 kernel: [13444137.872016] [<ffffffff812eb695>] ?
page_fault+0x25/0x30
Aug 23 21:40:09 vz2 kernel: [13444137.872017] Mem-Info:
Aug 23 21:40:09 vz2 kernel: [13444137.872018] Node 0 DMA per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.872020] CPU 0: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872021] CPU 1: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872022] CPU 2: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872024] CPU 3: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872025] CPU 4: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872026] CPU 5: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872028] CPU 6: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872029] CPU 7: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.872030] Node 0 DMA32 per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.872031] CPU 0: hi: 186,
btch: 31 usd: 165
Aug 23 21:40:09 vz2 kernel: [13444137.872033] CPU 1: hi: 186,
btch: 31 usd: 12
Aug 23 21:40:09 vz2 kernel: [13444137.872034] CPU 2: hi: 186,
btch: 31 usd: 170
Aug 23 21:40:09 vz2 kernel: [13444137.872035] CPU 3: hi: 186,
btch: 31 usd: 156
Aug 23 21:40:09 vz2 kernel: [13444137.872037] CPU 4: hi: 186,
btch: 31 usd: 157
Aug 23 21:40:09 vz2 kernel: [13444137.872038] CPU 5: hi: 186,
btch: 31 usd: 155
Aug 23 21:40:09 vz2 kernel: [13444137.872039] CPU 6: hi: 186,
btch: 31 usd: 157
Aug 23 21:40:09 vz2 kernel: [13444137.872040] CPU 7: hi: 186,
btch: 31 usd: 174
Aug 23 21:40:09 vz2 kernel: [13444137.872041] Node 0 Normal per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.872043] CPU 0: hi: 186,
btch: 31 usd: 108
Aug 23 21:40:09 vz2 kernel: [13444137.872044] CPU 1: hi: 186,
btch: 31 usd: 161
Aug 23 21:40:09 vz2 kernel: [13444137.872045] CPU 2: hi: 186,
btch: 31 usd: 163
Aug 23 21:40:09 vz2 kernel: [13444137.872046] CPU 3: hi: 186,
btch: 31 usd: 127
Aug 23 21:40:09 vz2 kernel: [13444137.872048] CPU 4: hi: 186,
btch: 31 usd: 139
Aug 23 21:40:09 vz2 kernel: [13444137.872049] CPU 5: hi: 186,
btch: 31 usd: 171
Aug 23 21:40:09 vz2 kernel: [13444137.872050] CPU 6: hi: 186,
btch: 31 usd: 133
Aug 23 21:40:09 vz2 kernel: [13444137.872052] CPU 7: hi: 186,
btch: 31 usd: 172
Aug 23 21:40:09 vz2 kernel: [13444137.872055] active_anon:127099
inactive_anon:16297 isolated_anon:0
Aug 23 21:40:09 vz2 kernel: [13444137.872056] active_file:668874
inactive_file:2839950 isolated_file:0
Aug 23 21:40:09 vz2 kernel: [13444137.872056] unevictable:0 dirty:
3089 writeback:0 unstable:0
Aug 23 21:40:09 vz2 kernel: [13444137.872057] free:225934
slab_reclaimable:166494 slab_unreclaimable:13922
Aug 23 21:40:09 vz2 kernel: [13444137.872058] mapped:13973 shmem:1139
pagetables:8974 bounce:0
Aug 23 21:40:09 vz2 kernel: [13444137.872059] Node 0 DMA free:15800kB
min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:15188kB mlocked:0kB dirty:0kB writeback:0kB
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB
kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:
0kB pages_scanned:0 all_unreclaimable? yes
Aug 23 21:40:09 vz2 kernel: [13444137.872067] lowmem_reserve[]: 0 2982
16086 16086
Aug 23 21:40:09 vz2 kernel: [13444137.872069] Node 0 DMA32 free:
313340kB min:3008kB low:3760kB high:4512kB active_anon:844kB
inactive_anon:3524kB active_file:259008kB inactive_file:2010296kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:
3054420kB mlocked:0kB dirty:32kB writeback:0kB mapped:4344kB shmem:0kB
slab_reclaimable:181804kB slab_unreclaimable:9844kB kernel_stack:408kB
pagetables:156kB unstable:0kB bounce:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
Aug 23 21:40:09 vz2 kernel: [13444137.872077] lowmem_reserve[]: 0 0
13104 13104
Aug 23 21:40:09 vz2 kernel: [13444137.872078] Node 0 Normal free:
574596kB min:13216kB low:16520kB high:19824kB active_anon:507552kB
inactive_anon:61664kB active_file:2416488kB inactive_file:9349504kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:
13418496kB mlocked:0kB dirty:12324kB writeback:0kB mapped:51548kB
shmem:4556kB slab_reclaimable:484172kB slab_unreclaimable:45844kB
kernel_stack:4464kB pagetables:35740kB unstable:0kB bounce:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Aug 23 21:40:09 vz2 kernel: [13444137.872086] lowmem_reserve[]: 0 0 0 0
Aug 23 21:40:09 vz2 kernel: [13444137.872088] Node 0 DMA: 2*4kB 2*8kB
0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
3*4096kB = 15800kB
Aug 23 21:40:09 vz2 kernel: [13444137.872093] Node 0 DMA32: 20691*4kB
8508*8kB 4799*16kB 1613*32kB 405*64kB 67*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 313724kB
Aug 23 21:40:09 vz2 kernel: [13444137.872099] Node 0 Normal: 71219*4kB
27806*8kB 1748*16kB 1207*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
0*2048kB 0*4096kB = 574940kB
Aug 23 21:40:09 vz2 kernel: [13444137.872104] 3512939 total pagecache
pages
Aug 23 21:40:09 vz2 kernel: [13444137.872105] 2963 pages in swap cache
Aug 23 21:40:09 vz2 kernel: [13444137.872106] Swap cache stats: add
58675, delete 55712, find 294564024/294568150
Aug 23 21:40:09 vz2 kernel: [13444137.872108] Free swap = 31207600kB
Aug 23 21:40:09 vz2 kernel: [13444137.872109] Total swap = 31247348kB
Aug 23 21:40:09 vz2 kernel: [13444137.909766] 4194288 pages RAM
Aug 23 21:40:09 vz2 kernel: [13444137.909768] 86418 pages reserved
Aug 23 21:40:09 vz2 kernel: [13444137.909769] 1510860 pages shared
Aug 23 21:40:09 vz2 kernel: [13444137.909770] 2659168 pages non-shared
Aug 23 21:40:09 vz2 kernel: [13444137.909845] Mem-Info:
Aug 23 21:40:09 vz2 kernel: [13444137.909847] Node 0 DMA per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.909850] CPU 0: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909852] CPU 1: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909855] CPU 2: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909857] CPU 3: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909859] CPU 4: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909862] CPU 5: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909864] CPU 6: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909866] CPU 7: hi: 0,
btch: 1 usd: 0
Aug 23 21:40:09 vz2 kernel: [13444137.909868] Node 0 DMA32 per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.909871] CPU 0: hi: 186,
btch: 31 usd: 165
Aug 23 21:40:09 vz2 kernel: [13444137.909873] CPU 1: hi: 186,
btch: 31 usd: 12
Aug 23 21:40:09 vz2 kernel: [13444137.909876] CPU 2: hi: 186,
btch: 31 usd: 170
Aug 23 21:40:09 vz2 kernel: [13444137.909878] CPU 3: hi: 186,
btch: 31 usd: 156
Aug 23 21:40:09 vz2 kernel: [13444137.909880] CPU 4: hi: 186,
btch: 31 usd: 157
Aug 23 21:40:09 vz2 kernel: [13444137.909883] CPU 5: hi: 186,
btch: 31 usd: 155
Aug 23 21:40:09 vz2 kernel: [13444137.909885] CPU 6: hi: 186,
btch: 31 usd: 157
Aug 23 21:40:09 vz2 kernel: [13444137.909887] CPU 7: hi: 186,
btch: 31 usd: 174
Aug 23 21:40:09 vz2 kernel: [13444137.909889] Node 0 Normal per-cpu:
Aug 23 21:40:09 vz2 kernel: [13444137.909892] CPU 0: hi: 186,
btch: 31 usd: 108
Aug 23 21:40:09 vz2 kernel: [13444137.909894] CPU 1: hi: 186,
btch: 31 usd: 161
Aug 23 21:40:09 vz2 kernel: [13444137.909897] CPU 2: hi: 186,
btch: 31 usd: 163
Aug 23 21:40:09 vz2 kernel: [13444137.909899] CPU 3: hi: 186,
btch: 31 usd: 127
Aug 23 21:40:09 vz2 kernel: [13444137.909901] CPU 4: hi: 186,
btch: 31 usd: 139
Aug 23 21:40:09 vz2 kernel: [13444137.909903] CPU 5: hi: 186,
btch: 31 usd: 171
Aug 23 21:40:09 vz2 kernel: [13444137.909906] CPU 6: hi: 186,
btch: 31 usd: 133
Aug 23 21:40:09 vz2 kernel: [13444137.909908] CPU 7: hi: 186,
btch: 31 usd: 173
Aug 23 21:40:09 vz2 kernel: [13444137.909912] active_anon:127099
inactive_anon:16297 isolated_anon:0
Aug 23 21:40:09 vz2 kernel: [13444137.909914] active_file:668874
inactive_file:2839950 iso
...
|
|
|
|
| Re: URGENT! process in container invokes PF to OOM-KILLER with no clear reason??? [message #47679 is a reply to message #47677] |
Fri, 24 August 2012 09:56  |
Martin Dobrev
Messages: 14
Registered: November 2006
|
Junior Member |
|
|
Hi,
I'd suggest you to start with a supported RHEL 5/6 kernel. Yours seems
to me a bit outdated thus buggy ;). You can find a recent version of it
and instructions how to install/convert RPM->DEB in the wiki.
Just an suggestion: If you don't use any Debian specific services on the
HN, why not directly using RHEL/CentOS for the HN? You'll be on the main
track with OVZ.
Regards,
Martin
HD Media
:)
On 24.8.2012 ?. 12:13 ?., Stoyan Stoyanov wrote:
> Hi list,
>
> We used openvz in production servers from 1-2 years ago, but now
> really need your suggestions and help. So this is the situation.
>
> We have setup with two identical openvz servers, and they was working
> perfectly fine until a the problem rise, so this is the problem:
>
> In a random time interval ( 1-2-3 hours ), the system (on the HN)
> lunch OOM-KILLER and start killing processes in different containers
> (so far not so unusual) , but the real problem is that 1. on the HN
> there is a lot of free memory resources and 2. in the VPS containers
> there is also free memory resources.
> From the log analyze it's clear that random processes do PF , which
> invoke the oom-killer , but it's really must NOT to do it.
>
> So, there are the /var/log/messages messages:
>
> Aug 23 21:40:09 vz2 kernel: [13444137.871993] postgres invoked
> oom-killer: gfp_mask=0x0, order=0, oom_adj=0
> Aug 23 21:40:09 vz2 kernel: [13444137.871995] Pid: 653, comm: postgres
> Not tainted 2.6.32-5-openvz-amd64 #1
> Aug 23 21:40:09 vz2 kernel: [13444137.871997] Call Trace:
> Aug 23 21:40:09 vz2 kernel: [13444137.872002] [<ffffffff810b8eec>] ?
> oom_kill_process+0x75/0x22c
> Aug 23 21:40:09 vz2 kernel: [13444137.872004] [<ffffffff810b940e>] ?
> __out_of_memory+0x93/0xad
> Aug 23 21:40:09 vz2 kernel: [13444137.872006] [<ffffffff810b969a>] ?
> pagefault_out_of_memory+0xba/0xe8
> Aug 23 21:40:09 vz2 kernel: [13444137.872009] [<ffffffff81032612>] ?
> mm_fault_error+0x5c/0x142
> Aug 23 21:40:09 vz2 kernel: [13444137.872012] [<ffffffff812ed805>] ?
> do_page_fault+0x2e0/0x2fc
> Aug 23 21:40:09 vz2 kernel: [13444137.872016] [<ffffffff812eb695>] ?
> page_fault+0x25/0x30
> Aug 23 21:40:09 vz2 kernel: [13444137.872017] Mem-Info:
> Aug 23 21:40:09 vz2 kernel: [13444137.872018] Node 0 DMA per-cpu:
> Aug 23 21:40:09 vz2 kernel: [13444137.872020] CPU 0: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872021] CPU 1: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872022] CPU 2: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872024] CPU 3: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872025] CPU 4: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872026] CPU 5: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872028] CPU 6: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872029] CPU 7: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872030] Node 0 DMA32 per-cpu:
> Aug 23 21:40:09 vz2 kernel: [13444137.872031] CPU 0: hi: 186,
> btch: 31 usd: 165
> Aug 23 21:40:09 vz2 kernel: [13444137.872033] CPU 1: hi: 186,
> btch: 31 usd: 12
> Aug 23 21:40:09 vz2 kernel: [13444137.872034] CPU 2: hi: 186,
> btch: 31 usd: 170
> Aug 23 21:40:09 vz2 kernel: [13444137.872035] CPU 3: hi: 186,
> btch: 31 usd: 156
> Aug 23 21:40:09 vz2 kernel: [13444137.872037] CPU 4: hi: 186,
> btch: 31 usd: 157
> Aug 23 21:40:09 vz2 kernel: [13444137.872038] CPU 5: hi: 186,
> btch: 31 usd: 155
> Aug 23 21:40:09 vz2 kernel: [13444137.872039] CPU 6: hi: 186,
> btch: 31 usd: 157
> Aug 23 21:40:09 vz2 kernel: [13444137.872040] CPU 7: hi: 186,
> btch: 31 usd: 174
> Aug 23 21:40:09 vz2 kernel: [13444137.872041] Node 0 Normal per-cpu:
> Aug 23 21:40:09 vz2 kernel: [13444137.872043] CPU 0: hi: 186,
> btch: 31 usd: 108
> Aug 23 21:40:09 vz2 kernel: [13444137.872044] CPU 1: hi: 186,
> btch: 31 usd: 161
> Aug 23 21:40:09 vz2 kernel: [13444137.872045] CPU 2: hi: 186,
> btch: 31 usd: 163
> Aug 23 21:40:09 vz2 kernel: [13444137.872046] CPU 3: hi: 186,
> btch: 31 usd: 127
> Aug 23 21:40:09 vz2 kernel: [13444137.872048] CPU 4: hi: 186,
> btch: 31 usd: 139
> Aug 23 21:40:09 vz2 kernel: [13444137.872049] CPU 5: hi: 186,
> btch: 31 usd: 171
> Aug 23 21:40:09 vz2 kernel: [13444137.872050] CPU 6: hi: 186,
> btch: 31 usd: 133
> Aug 23 21:40:09 vz2 kernel: [13444137.872052] CPU 7: hi: 186,
> btch: 31 usd: 172
> Aug 23 21:40:09 vz2 kernel: [13444137.872055] active_anon:127099
> inactive_anon:16297 isolated_anon:0
> Aug 23 21:40:09 vz2 kernel: [13444137.872056] active_file:668874
> inactive_file:2839950 isolated_file:0
> Aug 23 21:40:09 vz2 kernel: [13444137.872056] unevictable:0
> dirty:3089 writeback:0 unstable:0
> Aug 23 21:40:09 vz2 kernel: [13444137.872057] free:225934
> slab_reclaimable:166494 slab_unreclaimable:13922
> Aug 23 21:40:09 vz2 kernel: [13444137.872058] mapped:13973 shmem:1139
> pagetables:8974 bounce:0
> Aug 23 21:40:09 vz2 kernel: [13444137.872059] Node 0 DMA free:15800kB
> min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB
> active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15188kB mlocked:0kB dirty:0kB writeback:0kB
> mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB
> kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Aug 23 21:40:09 vz2 kernel: [13444137.872067] lowmem_reserve[]: 0 2982
> 16086 16086
> Aug 23 21:40:09 vz2 kernel: [13444137.872069] Node 0 DMA32
> free:313340kB min:3008kB low:3760kB high:4512kB active_anon:844kB
> inactive_anon:3524kB active_file:259008kB inactive_file:2010296kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB
> present:3054420kB mlocked:0kB dirty:32kB writeback:0kB mapped:4344kB
> shmem:0kB slab_reclaimable:181804kB slab_unreclaimable:9844kB
> kernel_stack:408kB pagetables:156kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Aug 23 21:40:09 vz2 kernel: [13444137.872077] lowmem_reserve[]: 0 0
> 13104 13104
> Aug 23 21:40:09 vz2 kernel: [13444137.872078] Node 0 Normal
> free:574596kB min:13216kB low:16520kB high:19824kB
> active_anon:507552kB inactive_anon:61664kB active_file:2416488kB
> inactive_file:9349504kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:13418496kB mlocked:0kB dirty:12324kB
> writeback:0kB mapped:51548kB shmem:4556kB slab_reclaimable:484172kB
> slab_unreclaimable:45844kB kernel_stack:4464kB pagetables:35740kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Aug 23 21:40:09 vz2 kernel: [13444137.872086] lowmem_reserve[]: 0 0 0 0
> Aug 23 21:40:09 vz2 kernel: [13444137.872088] Node 0 DMA: 2*4kB 2*8kB
> 0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
> 3*4096kB = 15800kB
> Aug 23 21:40:09 vz2 kernel: [13444137.872093] Node 0 DMA32: 20691*4kB
> 8508*8kB 4799*16kB 1613*32kB 405*64kB 67*128kB 0*256kB 0*512kB
> 0*1024kB 0*2048kB 0*4096kB = 313724kB
> Aug 23 21:40:09 vz2 kernel: [13444137.872099] Node 0 Normal: 71219*4kB
> 27806*8kB 1748*16kB 1207*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
> 0*2048kB 0*4096kB = 574940kB
> Aug 23 21:40:09 vz2 kernel: [13444137.872104] 3512939 total pagecache
> pages
> Aug 23 21:40:09 vz2 kernel: [13444137.872105] 2963 pages in swap cache
> Aug 23 21:40:09 vz2 kernel: [13444137.872106] Swap cache stats: add
> 58675, delete 55712, find 294564024/294568150
> Aug 23 21:40:09 vz2 kernel: [13444137.872108] Free swap = 31207600kB
> Aug 23 21:40:09 vz2 kernel: [13444137.872109] Total swap = 31247348kB
> Aug 23 21:40:09 vz2 kernel: [13444137.909766] 4194288 pages RAM
> Aug 23 21:40:09 vz2 kernel: [13444137.909768] 86418 pages reserved
> Aug 23 21:40:09 vz2 kernel: [13444137.909769] 1510860 pages shared
> Aug 23 21:40:09 vz2 kernel: [13444137.909770] 2659168 pages non-shared
> Aug 23 21:40:09 vz2 kernel: [13444137.909845] Mem-Info:
> Aug 23 21:40:09 vz2 kernel: [13444137.909847] Node 0 DMA per-cpu:
> Aug 23 21:40:09 vz2 kernel: [13444137.909850] CPU 0: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909852] CPU 1: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909855] CPU 2: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909857] CPU 3: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909859] CPU 4: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909862] CPU 5: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909864] CPU 6: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909866] CPU 7: hi: 0,
> btch: 1 usd: 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909868] Node 0 DMA32 per-cpu:
> Aug 23 21:40:09 vz2 kernel: [13444137.909871] CPU 0: hi: 186,
> btch: 31 usd: 165
> Aug 23 21:40:09 vz2 kernel: [13444137.909873] CPU 1: hi: 186,
> btch: 31 usd: 12
> Aug 23 21:40:09 vz2 kernel: [13444137.909876] CPU 2: hi: 186,
> btch: 31 usd: 170
> Aug 23 21:40:09 vz2 kernel: [13444137.909878] CPU 3: hi: 186,
> btch: 31 usd: 156
> Aug 23 21:40:09 vz2 kernel: [13444137.909880] CPU 4: hi: 186,
> btch: 31 usd: 157
> Aug 23 21:40:09 vz2 kernel: [13444137.909883] CPU 5: hi: 186,
> btch: 31 usd: 155
> Aug 23 21:40:09 vz2 kernel: [13444137.909885] CPU 6: hi: 186,
> btch: 31 usd: 157
> Aug 23 21:40:09 vz2 kernel: [13444137.909887] CPU 7: hi: 186,
> btch: 31 usd: 174
> Aug 23 21:40:09 vz2 kernel: [13444137.909889] Node 0 Normal per-cpu:
> Aug 23 21:40:09 vz2 kernel: [13444137.909892] CPU 0: hi: 186,
> btch: 31 usd: 108
> Aug 23 21:40:09 vz2 kernel: [13444137.909894] CPU 1: hi: 186,
> btch: 31 usd: 161
> Aug 23 21:40:09 vz2 kernel: [13444137.909897] CPU 2: hi: 186,
> btch: 31 usd: 163
> Aug 23 21:40:09 vz2 kernel: [13444137.909899] CPU 3: hi: 186,
> btch: 31 usd: 127
> Aug 23 21:40:09 vz2 kernel: [13444137.909901] CPU 4: hi: 186,
> btch: 31 usd: 139
> Aug 23 21:40:09 vz2 kernel: [13444137.909903] CPU 5: hi: 186,
> btch: 31 usd: 171
> Aug 23 21:40:09 vz2 kernel: [13444137.909906] CPU 6: hi: 186,
> btch: 31 usd: 133
> Aug 23 21:40:09 vz2 kernel: [13444137.909908] CPU 7: hi: 186,
> btch: 31 usd: 173
> Aug 23 21:40:09 vz2 kernel: [13444137.909912] active_anon:127099
> inactive_anon:16297 isolated_anon:0
> Aug 23 21:40:09 vz2 kernel: [13444137.909914] active_file:668874
> inactive_file:2839950 isolated_file:0
> Aug 23 21:40:09 vz2 kernel: [13444137.909915] unevictable:0
> dirty:3089 writeback:0 unstable:0
> Aug 23 21:40:09 vz2 kernel: [13444137.909916] free:225934
> slab_reclaimable:166494 slab_unreclaimable:13922
> Aug 23 21:40:09 vz2 kernel: [13444137.909917] mapped:13973 shmem:1139
> pagetables:8974 bounce:0
> Aug 23 21:40:09 vz2 kernel: [13444137.909920] Node 0 DMA free:15800kB
> min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB
> active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:15188kB mlocked:0kB dirty:0kB writeback:0kB
> mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB
> kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Aug 23 21:40:09 vz2 kernel: [13444137.909931] lowmem_reserve[]: 0 2982
> 16086 16086
> Aug 23 21:40:09 vz2 kernel: [13444137.909936] Node 0 DMA32
> free:313340kB min:3008kB low:3760kB high:4512kB active_anon:844kB
> inactive_anon:3524kB active_file:259008kB inactive_file:2010296kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB
> present:3054420kB mlocked:0kB dirty:32kB writeback:0kB mapped:4344kB
> shmem:0kB slab_reclaimable:181804kB slab_unreclaimable:9844kB
> kernel_stack:408kB pagetables:156kB unstable:0kB bounce:0kB
> writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Aug 23 21:40:09 vz2 kernel: [13444137.909948] lowmem_reserve[]: 0 0
> 13104 13104
> Aug 23 21:40:09 vz2 kernel: [13444137.909953] Node 0 Normal
> free:574596kB min:13216kB low:16520kB high:19824kB
> active_anon:507552kB inactive_anon:61664kB active_file:2416488kB
> inactive_file:9349504kB unevictable:0kB isolated(anon):0kB
> isolated(file):0kB present:13418496kB mlocked:0kB dirty:12324kB
> writeback:0kB mapped:51548kB shmem:4556kB slab_reclaimable:484172kB
> slab_unreclaimable:45844kB kernel_stack:4464kB pagetables:35740kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> Aug 23 21:40:09 vz2 kernel: [13444137.909965] lowmem_reserve[]: 0 0 0 0
> Aug 23 21:40:09 vz2 kernel: [13444137.909967] Node 0 DMA: 2*4kB 2*8kB
> 0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB
> 3*4096kB = 15800kB
> Aug 23 21:40:09 vz2 kernel: [13444137.909975] Node 0 DMA32: 20691*4kB
> 8508*8kB 4799*16kB 1613*32kB 405*64kB 67*128kB 0*256kB 0*512kB
> 0*1024kB 0*2048kB 0*4096kB = 313724kB
> Aug 23 21:40:09 vz2 kernel: [13444137.909983] Node 0 Normal: 71219*4kB
> 27806*8kB 1749*16kB 1207*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB
> 0*2048kB 0*4096kB = 574956kB
> Aug 23 21:40:09 vz2 kernel: [13444137.909991] 3512939 total pagecache
> pages
> Aug 23 21:40:09 vz2 kernel: [13444137.909993] 2963 pages in swap cache
> Aug 23 21:40:09 vz2 kernel: [13444137.909995] Swap cache stats: add
> 58675, delete 55712, find 294564024/294568150
> Aug 23 21:40:09 vz2 kernel: [13444137.909997] Free swap = 31207600kB
> Aug 23 21:40:09 vz2 kernel: [13444137.909998] Total swap = 31247348kB
> Aug 23 21:40:10 vz2 kernel: [13444137.959017] 4194288 pages RAM
> Aug 23 21:40:10 vz2 kernel: [13444137.959019] 86418 pages reserved
> Aug 23 21:40:10 vz2 kernel: [13444137.959020] 1510878 pages shared
> Aug 23 21:40:10 vz2 kernel: [13444137.959021] 2659168 pages non-shared
> Aug 23 21:40:10 vz2 kernel: [13444137.964021] OOM killed process
> apache2 (pid=13259, ve=202) exited, free=230863 gen=22391.
> Aug 23 21:40:10 vz2 kernel: [13444137.972302] OOM killed process
> apache2 (pid=13239, ve=202) exited, free=235866 gen=22392.
> Aug 23 21:40:10 vz2 kernel: [13444137.976072] OOM killed process
> apache2 (pid=27304, ve=201) exited, free=237075 gen=22393.
> Aug 23 21:40:10 vz2 kernel: [13444137.983820] OOM killed process
> apache2 (pid=13247, ve=202) exited, free=242072 gen=22394.
> Aug 23 21:40:10 vz2 kernel: [13444137.987171] OOM killed process udevd
> (pid=29123, ve=211) exited, free=242167 gen=22395.
> Aug 23 21:40:10 vz2 kernel: [13444137.991245] OOM killed process
> rsyslogd (pid=29416, ve=211) exited, free=242167 gen=22396.
> Aug 23 21:40:10 vz2 kernel: [13444137.995244] OOM killed process sshd
> (pid=29429, ve=211) exited, free=242332 gen=22397.
> Aug 23 21:40:10 vz2 kernel: [13444137.999140] OOM killed process
> xinetd (pid=29436, ve=211) exited, free=242408 gen=22398.
> Aug 23 21:40:10 vz2 kernel: [13444138.003141] OOM killed process
> saslauthd (pid=29443, ve=211) exited, free=242408 gen=22399.
> Aug 23 21:40:10 vz2 kernel: [13444138.007302] OOM killed process
> sendmail (pid=29459, ve=211) exited, free=242856 gen=22400.
> Aug 23 21:40:10 vz2 kernel: [13444138.011214] OOM killed process
> sendmail (pid=29466, ve=211) exited, free=243228 gen=22401.
> Aug 23 21:40:10 vz2 kernel: [13444138.015316] OOM killed process httpd
> (pid=29474, ve=211) exited, free=243414 gen=22402.
> Aug 23 21:40:10 vz2 kernel: [13444138.023087] OOM killed process crond
> (pid=29482, ve=211) exited, free=243600 gen=22403.
> Aug 23 21:40:10 vz2 kernel: [13444138.027811] OOM killed process
> radiusd (pid=3940, ve=211) exited, free=244904 gen=22404.
> Aug 23 21:40:10 vz2 kernel: [13444138.039797] OOM killed process
> apache2 (pid=13234, ve=202) exited, free=249923 gen=22405.
> Aug 23 21:40:10 vz2 kernel: [13444138.043952] OOM killed process
> apache2 (pid=29056, ve=201) exited, free=251318 gen=22406.
> Aug 23 21:40:10 vz2 kernel: [13444138.051801] OOM killed process
> apache2 (pid=13271, ve=202) exited, free=256367 gen=22407.
> Aug 23 21:40:10 vz2 kernel: [13444138.059855] OOM killed process
> apache2 (pid=27702, ve=202) exited, free=263823 gen=22408.
> Aug 23 21:40:10 vz2 kernel: [13444138.067754] OOM killed process
> apache2 (pid=27701, ve=202) exited, free=271078 gen=22409.
> Aug 23 21:40:10 vz2 kernel: [13444138.075214] OOM killed process
> apache2 (pid=32672, ve=202) exited, free=276172 gen=22410.
> Aug 23 21:40:10 vz2 kernel: [13444138.083472] OOM killed process
> apache2 (pid=16198, ve=202) exited, free=283240 gen=22411.
> Aug 23 21:40:10 vz2 kernel: [13444138.090803] OOM killed process
> apache2 (pid=19018, ve=202) exited, free=288169 gen=22412.
> Aug 23 21:40:10 vz2 kernel: [13444138.099506] OOM killed process
> apache2 (pid=6824, ve=202) exited, free=293052 gen=22413.
> Aug 23 21:40:10 vz2 kernel: [13444138.107363] OOM killed process
> apache2 (pid=21691, ve=202) exited, free=297671 gen=22414.
> Aug 23 21:40:10 vz2 kernel: [13444138.114606] OOM killed process
> apache2 (pid=26866, ve=202) exited, free=302015 gen=22415.
> Aug 23 21:40:10 vz2 kernel: [13444138.122742] OOM killed process
> apache2 (pid=29596, ve=202) exited, free=306201 gen=22416.
> Aug 23 21:40:10 vz2 kernel: [13444138.129498] OOM killed process
> apache2 (pid=32155, ve=202) exited, free=306736 gen=22417.
> Aug 23 21:40:10 vz2 kernel: [13444138.133860] OOM killed process
> mysqld (pid=18656, ve=202) exited, free=314273 gen=22418.
> Aug 23 21:40:10 vz2 kernel: [13444138.135274] OOM killed process httpd
> (pid=29476, ve=211) exited, free=314981 gen=22419.
> Aug 23 21:40:10 vz2 kernel: [13444138.140485] OOM killed process
> postgres (pid=653, ve=206) exited, free=316294 gen=22420.
>
>
> as you see there in that particular situation the postgre process (now
> is prostgre, but it's random process I mean, not everytime this
> process invokes OOM) invokes the killer, do call trace, and then with
> no meaning the HN do OOM killing on a lot of processes in different
> containers. (because the kernel things that this process should be
> killed - ok) , BUT in the time frames of oom killing on the system
> there is a lot of free memory resources ( no swap used, a lot of
> cached ram , etc.). I am attaching a munin memory graph , so you can
> see what i'm talking about:
>
>
>
> There is the HN OS information:
>
> root@vz2:~# uname -a
> Linux vz2 2.6.32-5-openvz-amd64 #1 SMP Mon Oct 3 05:12:50 UTC 2011
> x86_64 GNU/Linux
> root@vz2:~# cat /proc/version
> Linux version 2.6.32-5-openvz-amd64 (Debian 2.6.32-38)
> (ben@decadent.org.uk <mailto:ben@decadent.org.uk>) (gcc version 4.3.5
> (Debian 4.3.5-4) ) #1 SMP Mon Oct 3 05:12:50 UTC 2011
>
> *Please reply if you need more information , or anything I can provide.*
> *The problem is urgent, because the kernel kill some of the critical
> services for us.*
>
> OK, of course there is NO failcnt in the beancounters , there is no
> overcommit or something like this on vzmemcheck and other vz tools.
>
> And Not least, we have two absolute identical servers (same hardware,
> same install, same kernels, just different VEs in that), but the
> problem rise only on one of them.
>
>
>
>
>
> Stoyan Stoyanov
> Core System Administrator
>
>
>
> CONFIDENTIAL
> The information contained in this email and any attachment is
> confidential. It is intended only for the named addressee(s). If you
> are not the named addressee(s) please notify the sender immediately
> and do not disclose, copy or distribute the contents to any other
> person other than the intended addressee(s).
>
>
>
|
|
|
|
Goto Forum:
Current Time: Wed Mar 29 23:10:29 GMT 2023
Total time taken to generate the page: 0.00993 seconds
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
