| Syslog-ng on HN eats RAM and OOM Killer kills VE processess (CRITICAL) [message #35906] |
Thu, 30 April 2009 06:58  |
januszzz
Messages: 50
Registered: January 2007
Location: Opole, Poland
|
Member |
|
|
Hi,
I have reproducible problem which takes my OpenVZ installation out of production.
I got HN with ~10 VEs. One of those VEs is syslog-ng which listens for remote connections and gathers all log messages from several dozens of machines, including some of those VEs.
This syslog-ng gathers information from HN too.
And here is the real problem:
- when I restart syslog-ng on the VE, the syslog-ng on HN silently starts to eat memory and after 6-12 hours or so eats every memory bit machine has (24GB plus 10GB swap).
Needlees to say OOM killer enters then and kills all syslog-ng instances on every VE. Besides, it randomly kills some more programs, including critical MySQLs and even ssh.
Then machine gets back to life, but the services needs restarting.
I got two identical machines (thanks for vzmigrate tool, btw!) and on both machines all goes the same. THe situation has happen 4th time, always when network admin gets his hands into his syslog-ng config file, and after when he restarts it (it looks like the trigger for eating ram is syslog restart in VE, but I cannot be certain).
I can clone machines and simulate this behaviour, but please let me know what should I catch so I can give some feedback.
Thanks, Janusz.
[Updated on: Thu, 30 April 2009 09:40] Report message to a moderator |
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
