OpenVZ Forum: Support » Fsck needed after out of memory

Home » General » Support » Fsck needed after out of memory

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Fsck needed after out of memory [message #53580]

Mon, 30 September 2019 15:13

andre
Messages: 36
Registered: January 2008

Member

Hello,

We are seeing lot of cases where fsck is needed after out of memory situations (different hardwares/ves/etc)

============================================================ ============================================
Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_free_blocks:4879: Out of memory
Sep 30 10:18:34 kernel: Aborting journal on device ploop43739p1-8.
Sep 30 10:18:34 kernel: EXT4-fs (ploop43739p1): Remounting filesystem read-only
Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_ext_remove_space:3060: Journal has aborted
Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_ext_truncate:4698: Journal has aborted
Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_truncate:4139: Journal has aborted
Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
Sep 30 10:18:35 kernel: httpd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Sep 30 10:18:35 kernel: httpd cpuset=3657 mems_allowed=0
Sep 30 10:18:35 kernel: CPU: 1 PID: 828588 Comm: httpd ve: 3657 Kdump: loaded Not tainted 3.10.0-957.12.2.vz7.96.21 #1 96.21
Sep 30 10:18:35 kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.0b 09/17/2012
Sep 30 10:18:35 kernel: Call Trace:
Sep 30 10:18:35 kernel: [<ffffffff8b594677>] dump_stack+0x19/0x1b
Sep 30 10:18:35 kernel: [<ffffffff8b58ef52>] dump_header+0x90/0x229
Sep 30 10:18:35 kernel: [<ffffffff8afcdcf8>] oom_kill_process+0x5e8/0x640
Sep 30 10:18:35 kernel: [<ffffffff8aff82ce>] ? get_task_oom_score_adj+0xee/0x100
Sep 30 10:18:35 kernel: [<ffffffff8b0514e9>] mem_cgroup_oom_synchronize+0x4a9/0x500
Sep 30 10:18:35 kernel: [<ffffffff8afce2a3>] pagefault_out_of_memory+0x13/0x50
Sep 30 10:18:35 kernel: [<ffffffff8b58d027>] mm_fault_error+0x6a/0x157
Sep 30 10:18:35 kernel: [<ffffffff8b5a28a8>] __do_page_fault+0x3c8/0x4f0
Sep 30 10:18:35 kernel: [<ffffffff8b5a2a05>] do_page_fault+0x35/0x90
Sep 30 10:18:35 kernel: [<ffffffff8b59e768>] page_fault+0x28/0x30
Sep 30 10:18:35 kernel: Task in /machine.slice/3657 killed as a result of limit of /machine.slice/3657
Sep 30 10:18:35 kernel: memory: usage 2097056kB, limit 2097152kB, failcnt 45598816
Sep 30 10:18:35 kernel: memory+swap: usage 2621444kB, limit 2621440kB, failcnt 13806662
Sep 30 10:18:35 kernel: kmem: usage 407384kB, limit 9007199254740988kB, failcnt 0
Sep 30 10:18:35 kernel: Memory cgroup stats for /machine.slice/3657: rss_huge:0KB mapped_file:338136KB shmem:406204KB slab_unreclaimable:13824KB swap:524388KB cache:406476KB rss:1283196KB slab_reclaimable:355864KB inactive_anon:432588KB active_anon:1256812KB inactive_file:40KB active_file:12KB unevictable:0KB
Sep 30 10:18:35 kernel: Memory cgroup out of memory: Kill process 828656 (clamd) score 283 or sacrifice child
Sep 30 10:18:35 kernel: Killed process 794108 (clamd) in VE "3657" total-vm:1032160kB, anon-rss:628748kB, file-rss:0kB, shmem-rss:0kB
Sep 30 10:18:35 kernel: rsyslogd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Sep 30 10:18:35 kernel: rsyslogd cpuset=3657 mems_allowed=0
Sep 30 10:18:35 kernel: CPU: 1 PID: 72938 Comm: rsyslogd ve: 3657 Kdump: loaded Not tainted 3.10.0-957.12.2.vz7.96.21 #1 96.21
Sep 30 10:18:35 kernel: Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.0b 09/17/2012
Sep 30 10:18:35 kernel: Call Trace:
Sep 30 10:18:35 kernel: [<ffffffff8b594677>] dump_stack+0x19/0x1b
Sep 30 10:18:35 kernel: [<ffffffff8b58ef52>] dump_header+0x90/0x229
Sep 30 10:18:35 kernel: [<ffffffff8afcdcf8>] oom_kill_process+0x5e8/0x640
Sep 30 10:18:35 kernel: [<ffffffff8aff82ce>] ? get_task_oom_score_adj+0xee/0x100
Sep 30 10:18:35 kernel: [<ffffffff8b0514e9>] mem_cgroup_oom_synchronize+0x4a9/0x500
Sep 30 10:18:35 kernel: [<ffffffff8afce2a3>] pagefault_out_of_memory+0x13/0x50
Sep 30 10:18:35 kernel: [<ffffffff8b58d027>] mm_fault_error+0x6a/0x157
Sep 30 10:18:35 kernel: [<ffffffff8b5a28a8>] __do_page_fault+0x3c8/0x4f0
Sep 30 10:18:35 kernel: [<ffffffff8b5a2a05>] do_page_fault+0x35/0x90
Sep 30 10:18:35 kernel: [<ffffffff8b59e768>] page_fault+0x28/0x30
Sep 30 10:18:35 kernel: Task in /machine.slice/3657 killed as a result of limit of /machine.slice/3657
Sep 30 10:18:35 kernel: memory: usage 2097056kB, limit 2097152kB, failcnt 45598831
Sep 30 10:18:35 kernel: memory+swap: usage 2621444kB, limit 2621440kB, failcnt 13806683
Sep 30 10:18:35 kernel: kmem: usage 407384kB, limit 9007199254740988kB, failcnt 0
Sep 30 10:18:35 kernel: Memory cgroup stats for /machine.slice/3657: rss_huge:0KB mapped_file:338136KB shmem:406204KB slab_unreclaimable:13824KB swap:524388KB cache:406476KB rss:1283196KB slab_reclaimable:355864KB inactive_anon:432588KB active_anon:1256812KB inactive_file:40KB active_file:12KB unevictable:0KB
Sep 30 10:18:35 kernel: Memory cgroup out of memory: Kill process 828656 (clamd) score 283 or sacrifice child
Sep 30 10:18:35 kernel: Killed process 828656 (clamd) in VE "3657" total-vm:1032160kB, anon-rss:628756kB, file-rss:120kB, shmem-rss:0kB
Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_orphan_del:2723: Journal has aborted
Sep 30 10:18:35 kernel: EXT4-fs error (device ploop43739p1) in ext4_reserve_inode_write:5354: Journal has aborted
============================================================ ============================================

After that, VE will only mount with an fsck.

Looks like a bug.

Report message to a moderator

Re: Fsck needed after out of memory [message #53582 is a reply to message #53580]

Wed, 02 October 2019 06:14

vaverin
Messages: 708
Registered: September 2005

Senior Member

Dear andre,
thank you for reporting the problem,
but it's better to submit bug into bugs.openvz.org

I've submitted it as
https://bugs.openvz.org/browse/OVZ-7135

I would like to clarify: do you have any ext4 messages right before
Sep 30 10:18:34 kernel: EXT4-fs error (device ploop43739p1) in ext4_free_blocks:4879: Out of memory

?

Thank you,
Vasily Averin

Report message to a moderator

Re: Fsck needed after out of memory [message #53585 is a reply to message #53582]

Mon, 07 October 2019 00:13

andre
Messages: 36
Registered: January 2008

Member

Thanks for opening the bug report.

I have just browsed the logs and couldn't find any ext4 message before the "EXT4-fs error in ext4_free_blocks:4879: Out of memory". There are just out of memory messages before this line.

[Updated on: Mon, 07 October 2019 00:14]

Report message to a moderator

Previous Topic:	[OpenVZ 7] KVM KSM Support?
Next Topic:	Migration Fails with vzmigrate: vzmdest invoked oom-killer [ovz7]

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Nov 06 03:18:52 GMT 2024

Total time taken to generate the page: 0.03417 seconds