Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35906] |
Thu, 30 April 2009 06:58 ![Go to next message Go to next message](/theme/ovz3/images/down.png) |
januszzz
Messages: 50 Registered: January 2007 Location: Opole, Poland
|
Member |
|
|
Hi,
I have reproducible problem which takes my OpenVZ installation out of production.
I got HN with ~10 VEs. One of those VEs is syslog-ng which listens for remote connections and gathers all log messages from several dozens of machines, including some of those VEs.
This syslog-ng gathers information from HN too.
And here is the real problem:
- when I restart syslog-ng on the VE, the syslog-ng on HN silently starts to eat memory and after 6-12 hours or so eats every memory bit machine has (24GB plus 10GB swap).
Needlees to say OOM killer enters then and kills all syslog-ng instances on every VE. Besides, it randomly kills some more programs, including critical MySQLs and even ssh.
Then machine gets back to life, but the services needs restarting.
I got two identical machines (thanks for vzmigrate tool, btw!) and on both machines all goes the same. THe situation has happen 4th time, always when network admin gets his hands into his syslog-ng config file, and after when he restarts it (it looks like the trigger for eating ram is syslog restart in VE, but I cannot be certain).
I can clone machines and simulate this behaviour, but please let me know what should I catch so I can give some feedback.
Thanks, Janusz.
[Updated on: Thu, 30 April 2009 09:40] Report message to a moderator
|
|
|
|
Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35911 is a reply to message #35908] |
Thu, 30 April 2009 08:37 ![Go to previous message Go to previous message](/theme/ovz3/images/up.png) ![Go to next message Go to next message](/theme/ovz3/images/down.png) |
januszzz
Messages: 50 Registered: January 2007 Location: Opole, Poland
|
Member |
|
|
Hi, sorry, here it goes:
OpenVZ2 conf # ls -1 *conf | tail -n 12| while read x; do vzcfgvalidate $x; done
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
Validation completed: success
OpenVZ2 conf #
uname -a
Linux OpenVZ2 2.6.18-028stab056-3 #4 SMP Thu Mar 12 16:18:18 CET 2009 x86_64 Intel(R) Xeon(R) CPU E5430 @ 2.66GHz GenuineIntel GNU/Linux
Kernel Config included.
Janusz.
[Updated on: Thu, 30 April 2009 08:38] Report message to a moderator
|
|
|
|
|
|
Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35916 is a reply to message #35915] |
Thu, 30 April 2009 09:16 ![Go to previous message Go to previous message](/theme/ovz3/images/up.png) ![Go to next message Go to next message](/theme/ovz3/images/down.png) |
januszzz
Messages: 50 Registered: January 2007 Location: Opole, Poland
|
Member |
|
|
Huh, I expect that is true.
When I had 16GB RAM I've used vzsplit -n8 and based on this I've created the configs.
Later, I've added more 8GB more and tuned the next configs accordingly, expecting that this won't fail.
EDIT: what is bothering me is that those machines should not use more than say, 8GB ram and the only process which is balooning (it drinks 20-25 GB of RAM) is syslog-ng in HN. Why?
EDIT 2: As far as I understand this is more syslog-ng issue that it uses lots of ram. But while it goes in HN all of syslog-ng in each of the VE gets killed, am I right?
[Updated on: Thu, 30 April 2009 09:22] Report message to a moderator
|
|
|
|
Re: Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35918 is a reply to message #35917] |
Thu, 30 April 2009 09:36 ![Go to previous message Go to previous message](/theme/ovz3/images/up.png) |
januszzz
Messages: 50 Registered: January 2007 Location: Opole, Poland
|
Member |
|
|
>> But while it goes in HN all of syslog-ng in each of the VE gets
>> killed, am I right?
> No.
Hmm, this is strange and exactly opposite to what I observe.
As I've already said, syslog-ng is balooning in HN, not in a VE, and oom killer kills everything what is large, in all VEs. All mysql get killed, all sshd, all syslog-ng, all apaches, all Oracle listeners etc.
EDIT: COngrats to me, I've just edit the main post to clarify that this restart in VE of the syslog-ng triggers memory consuming in HN.
EDIT 2: So it looks like action taken inside VE silently touches a program in HN.
[Updated on: Thu, 30 April 2009 09:43] Report message to a moderator
|
|
|