OpenVZ Forum: Support » Openvz7 container hangs at out of memory

Home » General » Support » Openvz7 container hangs at out of memory

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Openvz7 container hangs at out of memory [message #52907]

Wed, 02 August 2017 15:15

wishd
Messages: 14
Registered: June 2017

Junior Member

Odd issue coming up. If a container runs out of memory it hangs. I can not stop, enter or shut it down. dmesg shows errors like:

Running kernel 3.10.0-514.16.1.vz7.30.15

[31650.198123] Task in /machine.slice/806bac77-11c9-4300-9959-32f4795ab5c3 killed as a result of limit of /machine.slice/806bac77-11c9-4300-9959-32f4795ab5c3
[31650.198125] memory: usage 2097164kB, limit 2097152kB, failcnt 10087303
[31650.198127] memory+swap: usage 3145872kB, limit 3145728kB, failcnt 218116
[31650.198129] kmem: usage 130864kB, limit 9007199254740988kB, failcnt 0
[31650.198130] Memory cgroup stats for /machine.slice/806bac77-11c9-4300-9959-32f4795ab5c3: cache:468KB rss:1965832KB rss_huge:0KB mapped_file:232KB shmem:256KB slab_reclaimable:37376KB slab_unreclaimable:41552KB swap:1048708KB inactive_anon:846572KB active_anon:1119516KB inactive_file:8KB active_file:0KB unevictable:0KB
[31650.198143] Memory cgroup out of memory: Kill process 865079 (mysqld) score 66 or sacrifice child
[31650.198186] Killed process 9910 (mysqld) in VE "806bac77-11c9-4300-9959-32f4795ab5c3" total-vm:908868kB, anon-rss:31720kB, file-rss:0kB, shmem-rss:0kB
[31801.312842] INFO: task ps:882799 blocked for more than 120 seconds.

I have these limits:
CT-806bac77 /# free
total used free shared buff/cache available
Mem: 2097152 291924 248116 316 1557112 1550724
Swap: 1048576 7044 1041532
CT-806bac77 /# df -m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/ploop32004p1 50267 22346 25755 47% /
none 32075 0 32075 0% /sys/fs/cgroup
none 32075 0 32075 0% /dev
tmpfs 1024 0 1024 0% /dev/shm
tmpfs 1024 1 1024 1% /run

806bac77-11c9-4300-9959-32f4795ab5c3: kmemsize 56086528 58318848 9223372036854775807 9223372036854775807 0
lockedpages 0 0 524288 524288 0
privvmpages 266606 276619 9223372036854775807 9223372036854775807 0
shmpages 420 2058 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numproc 99 171 131072 131072 0
physpages 521225 524288 524288 524288 0
vmguarpages 0 0 786432 786432 0
oomguarpages 523579 526664 0 0 0
numtcpsock 0 0 9223372036854775807 9223372036854775807 0
numflock 16 21 9223372036854775807 9223372036854775807 0
numpty 0 0 9223372036854775807 9223372036854775807 0
numsiginfo 0 57 9223372036854775807 9223372036854775807 0
tcpsndbuf 0 0 9223372036854775807 9223372036854775807 0
tcprcvbuf 0 0 9223372036854775807 9223372036854775807 0
othersockbuf 0 0 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 0 9223372036854775807 9223372036854775807 0
numothersock 0 0 9223372036854775807 9223372036854775807 0
dcachesize 16924672 16924672 9223372036854775807 9223372036854775807 0
numfile 1642 1985 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numiptent 544 547 2000 2000 0

The load on the system runs up as well when this happens. All other vm's work fine and I can enter them. Only containers are running on this, 16 with overall plenty of free CPU and ram otherwise.

Report message to a moderator

Re: Openvz7 container hangs at out of memory [message #52908 is a reply to message #52907]

Wed, 02 August 2017 15:26

khorenko
Messages: 533
Registered: January 2006
Location: Moscow, Russia

Senior Member

Hi,

please try latest kernel from https://download.openvz.org/virtuozzo/factory/x86_64/os/Pack ages/v/

And let us know in case you observe the same behavior here.
(i guess you won't, there were fixed for similar issues)

If your problem is solved - please, report it!
It's even more important than reporting the problem itself...

Report message to a moderator

Re: Openvz7 container hangs at out of memory [message #52909 is a reply to message #52908]

Wed, 02 August 2017 15:47

wishd
Messages: 14
Registered: June 2017

Junior Member

Updated and booted into this. I will report back what happens after 24 hours.

Report message to a moderator

Re: Openvz7 container hangs at out of memory [message #52911 is a reply to message #52908]

Thu, 03 August 2017 19:51

wishd
Messages: 14
Registered: June 2017

Junior Member

Yes its resolved.

Report message to a moderator

Re: Openvz7 container hangs at out of memory [message #52976 is a reply to message #52907]

Mon, 11 September 2017 09:53

save92
Messages: 1
Registered: September 2017

Junior Member

We are experiencing the same problem.
Kernel is 3.10.0-514.26.1.vz7.35.6

When a CT runs out of memory sometimes (not always), it freezes. When it happens, load avarage goes high and vzlist reports the same LA for each node:

vzlist -o ctid,laverage
CTID LAVERAGE
1075 51.21/47.98/73.69
1190 51.21/47.98/73.69
1324 51.21/47.98/73.69
1327 51.21/47.98/73.69
1332 51.21/47.98/73.69
1343 51.21/47.98/73.69

dmesg gets full of errors like:

[Mon Sep 11 11:27:10 2017] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[Mon Sep 11 11:27:10 2017] java cpuset=1652 mems_allowed=0-1
[Mon Sep 11 11:27:10 2017] CPU: 7 PID: 248313 Comm: java ve: 1652 Not tainted 3.10.0-514.26.1.vz7.35.6 #1 35.6
[Mon Sep 11 11:27:10 2017] Hardware name: Supermicro SYS-1028TP-DTR/X10DRT-P, BIOS 2.0a 08/12/2016
[Mon Sep 11 11:27:10 2017] 0000000000000000 000000007626cf68 ffff881eeebd3c98 ffffffff816838f0
[Mon Sep 11 11:27:10 2017] ffff881eeebd3d28 ffffffff816801eb 0000000000000000 00000000000001f9
[Mon Sep 11 11:27:10 2017] ffff8812e044b890 ffff881eeebd3d20 ffffffff81191507 ffff881eeebd3fd8
[Mon Sep 11 11:27:10 2017] Call Trace:
[Mon Sep 11 11:27:10 2017] [<ffffffff816838f0>] dump_stack+0x19/0x1b
[Mon Sep 11 11:27:10 2017] [<ffffffff816801eb>] dump_header+0x8e/0x225
[Mon Sep 11 11:27:10 2017] [<ffffffff81191507>] ? __add_to_page_cache_locked+0x97/0x310
[Mon Sep 11 11:27:10 2017] [<ffffffff8119596c>] oom_kill_process+0x5ec/0x640
[Mon Sep 11 11:27:10 2017] [<ffffffff811bdbae>] ? get_task_oom_score_adj+0xee/0x100
[Mon Sep 11 11:27:10 2017] [<ffffffff8120a749>] mem_cgroup_oom_synchronize+0x499/0x4e0
[Mon Sep 11 11:27:10 2017] [<ffffffff811c5e14>] ? handle_mm_fault+0x9e4/0xdc0
[Mon Sep 11 11:27:10 2017] [<ffffffff81195f13>] pagefault_out_of_memory+0x13/0x50
[Mon Sep 11 11:27:10 2017] [<ffffffff8167e56a>] mm_fault_error+0x68/0x12b
[Mon Sep 11 11:27:10 2017] [<ffffffff8168f025>] __do_page_fault+0x395/0x450
[Mon Sep 11 11:27:10 2017] [<ffffffff8168f115>] do_page_fault+0x35/0x90
[Mon Sep 11 11:27:10 2017] [<ffffffff8168b2c8>] page_fault+0x28/0x30
[Mon Sep 11 11:27:10 2017] Task in /machine.slice/1652 killed as a result of limit of /machine.slice/1652
[Mon Sep 11 11:27:10 2017] memory: usage 16699852kB, limit 16777216kB, failcnt 0
[Mon Sep 11 11:27:10 2017] memory+swap: usage 16908288kB, limit 16908288kB, failcnt 63525
[Mon Sep 11 11:27:10 2017] kmem: usage 146276kB, limit 9007199254740988kB, failcnt 0
[Mon Sep 11 11:27:10 2017] Memory cgroup stats for /machine.slice/1652: rss_huge:7802880KB mapped_file:36KB shmem:348KB slab_unreclaimable:55796KB swap:208436KB cache:556KB rss:16553020KB slab_reclaim
able:34504KB inactive_anon:1985016KB active_anon:14568352KB inactive_file:12KB active_file:8KB unevictable:0KB
[Mon Sep 11 11:27:10 2017] Memory cgroup out of memory: Kill process 1008195 (java) score 217 or sacrifice child
[Mon Sep 11 11:27:10 2017] Killed process 37119 (java) in VE "1652" total-vm:9411892kB, anon-rss:3677412kB, file-rss:0kB, shmem-rss:0kB

Report message to a moderator