OpenVZ Forum: Support » Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL)

Home » General » Support » Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL)

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35906]

Thu, 30 April 2009 06:58

januszzz
Messages: 50
Registered: January 2007
Location: Opole, Poland

Member

Hi,

I have reproducible problem which takes my OpenVZ installation out of production.

I got HN with ~10 VEs. One of those VEs is syslog-ng which listens for remote connections and gathers all log messages from several dozens of machines, including some of those VEs.

This syslog-ng gathers information from HN too.

And here is the real problem:

- when I restart syslog-ng on the VE, the syslog-ng on HN silently starts to eat memory and after 6-12 hours or so eats every memory bit machine has (24GB plus 10GB swap).

Needlees to say OOM killer enters then and kills all syslog-ng instances on every VE. Besides, it randomly kills some more programs, including critical MySQLs and even ssh.

Then machine gets back to life, but the services needs restarting.

I got two identical machines (thanks for vzmigrate tool, btw!) and on both machines all goes the same. THe situation has happen 4th time, always when network admin gets his hands into his syslog-ng config file, and after when he restarts it (it looks like the trigger for eating ram is syslog restart in VE, but I cannot be certain).

I can clone machines and simulate this behaviour, but please let me know what should I catch so I can give some feedback.

Thanks, Janusz.

[Updated on: Thu, 30 April 2009 09:40]

Report message to a moderator

Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35908 is a reply to message #35906]

Thu, 30 April 2009 08:04

kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia

Senior Member

What is your UBC configuration?

Kir Kolyshkin

http://static.openvz.org/userbars/openvz-developer.png

Report message to a moderator

Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35911 is a reply to message #35908]

Thu, 30 April 2009 08:37

januszzz
Messages: 50
Registered: January 2007
Location: Opole, Poland

Member

Hi, sorry, here it goes:

OpenVZ2 conf # ls -1 *conf | tail -n 12| while read x; do vzcfgvalidate $x; done
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
OpenVZ2 conf #

uname -a
Linux OpenVZ2 2.6.18-028stab056-3 #4 SMP Thu Mar 12 16:18:18 CET 2009 x86_64 Intel(R) Xeon(R) CPU E5430 @ 2.66GHz GenuineIntel GNU/Linux

Kernel Config included.

Janusz.

Attachment: config.gz
(Size: 8.03KB, Downloaded 215 times)
Attachment: userbeancounters.txt
(Size: 39.75KB, Downloaded 272 times)

[Updated on: Thu, 30 April 2009 08:38]

Report message to a moderator

Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35912 is a reply to message #35911]

Thu, 30 April 2009 08:40

kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia

Senior Member

Thanks! And the output of vzmemcheck -v, please.

Kir Kolyshkin

Report message to a moderator

Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35913 is a reply to message #35912]

Thu, 30 April 2009 08:49

januszzz
Messages: 50
Registered: January 2007
Location: Opole, Poland

Member

Here it goes.

Attachment: vzmemcheck.txt
(Size: 1.22KB, Downloaded 272 times)

Report message to a moderator

Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35915 is a reply to message #35913]

Thu, 30 April 2009 09:09

kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia

Senior Member

januszzz wrote on Thu, 30 April 2009 12:49

Here it goes.

This pretty much answers it. Your mem+swap commitment is more than 100%, meaning you allow to allocate more RAM+swap than you actually have.

See http://wiki.openvz.org/UBC_systemwide_configuration which explains it in great details. In particular, it says "Commitment levels more than 1 means that the containers are guaranteed more memory than the system has. Such overcommitment is strongly not recommended, because in that case if all the memory is consumed, random applications, including the ones belonging to the host system, may be killed and the system may become inaccessible by ssh(1) and lose other important functionality". See, this precisely describes your situation.

Kir Kolyshkin

Report message to a moderator

Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35916 is a reply to message #35915]

Thu, 30 April 2009 09:16

januszzz
Messages: 50
Registered: January 2007
Location: Opole, Poland

Member

Huh, I expect that is true.

When I had 16GB RAM I've used vzsplit -n8 and based on this I've created the configs.

Later, I've added more 8GB more and tuned the next configs accordingly, expecting that this won't fail.

EDIT: what is bothering me is that those machines should not use more than say, 8GB ram and the only process which is balooning (it drinks 20-25 GB of RAM) is syslog-ng in HN. Why?

EDIT 2: As far as I understand this is more syslog-ng issue that it uses lots of ram. But while it goes in HN all of syslog-ng in each of the VE gets killed, am I right?

[Updated on: Thu, 30 April 2009 09:22]

Report message to a moderator

Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35917 is a reply to message #35916]

Thu, 30 April 2009 09:31

kir
Messages: 1645
Registered: August 2005
Location: Moscow, Russia

Senior Member

Quote:

But while it goes in HN all of syslog-ng in each of the VE gets killed, am I right?

No.

What can happen though is if the init script of syslog-ng has invalid/bad logic (i.e when restarting it kills processes by name not by PID it obtained when the process was starting) then it can kill all VE's syslog-ng processes. Solution to that problem is to fix initscript to save/use PIDs.

Kir Kolyshkin

Report message to a moderator

Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35918 is a reply to message #35917]

Thu, 30 April 2009 09:36

januszzz
Messages: 50
Registered: January 2007
Location: Opole, Poland

Member

>> But while it goes in HN all of syslog-ng in each of the VE gets
>> killed, am I right?

> No.

Hmm, this is strange and exactly opposite to what I observe.

As I've already said, syslog-ng is balooning in HN, not in a VE, and oom killer kills everything what is large, in all VEs. All mysql get killed, all sshd, all syslog-ng, all apaches, all Oracle listeners etc.

EDIT: COngrats to me, I've just edit the main post to clarify that this restart in VE of the syslog-ng triggers memory consuming in HN.
EDIT 2: So it looks like action taken inside VE silently touches a program in HN.

[Updated on: Thu, 30 April 2009 09:43]

Report message to a moderator

Previous Topic:	shell used by 'vzctl enter'
Next Topic:	df showing wrong output - SOLVED

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Mon Nov 18 19:32:07 GMT 2024

Total time taken to generate the page: 0.02926 seconds