| Performance problems on heavily-loaded VE [message #33018] |
Wed, 17 September 2008 19:09  |
seanfulton
Messages: 105
Registered: May 2007
|
Senior Member |
|
|
We recently moved our busy, spam-fighting mail server from a dual-XEON machine with 2 G of RAM to a CentOS-4 VE on an Intel Dual Core machine with 4 G of RAM.
At the time of the move, we had two Windows nodes running on the machine with Xen, but after NUMEROUS kernel crashes I ditched the XEN kernel and am now just using the 2.6.18-53.1.19.el5.028stab053.14PAE kernel and have not had any crashes.
In any event, there are only two VEs running on this machine; one is a dev VE with one person using it as an FTP source and next to no activity. The other is the mail VE.
During the move we generated a HUGE mail spool of mostly spam. What I am seeing now is that whenever the system load climbs above say 4 or so within the VE, it effectively stops responding to POP/IMAP/SMTP connetions. The connection opens, but the appropriate service just doesn't come up. Usually this is when sendmail starts trying to process that backed up mail queue.
What I find strange is that on the dual XEON machine with less RAM, we had no problems even as the load climed to 20 or above. We only started having connection problems when the load got up around 40 or so.
I've tuned everything I can think of an the mail VE has effectively all of the resources available. /proc/user_beancounters shows no failures, and the VE shows 4 G of RAM.
Has anyone else experienced anything like this or is able to point me in a direction? I was thinking that *perhaps* if the system load is due to processing the mail spool and that once we clear that out, it will be fine, but I am concerned.
We're currently running more than 50 other VEs on other machines in all sorts of configuration with no problems, but those are mainly web servers with static and dynamic content, and rarely push that kind of load.
Any ideas or pointers would be appreciated.
Oh yes, running apps inside the VE are sendmail, cyrus-imapd, spamassassin, spamass-milter, saslauthd.
sean
|
|
|
|
|
|
|
|
| Re: Performance problems on heavily-loaded VE [message #33026 is a reply to message #33021] |
Thu, 18 September 2008 09:35  |
piavlo
Messages: 159
Registered: January 2007
|
Senior Member |
|
|
Can the VE be stoped/restarted with vzctl after the problems begin? If the POP/IMAP/SMTP daemons be restarted insive VE do they resume to work ok?
Do you have any nfs mount in VE? What is the state of the
POP/IMAP/SMTP daemon on VE? Maybe they are stuck on IO in D state?
try to strace/lsof them.
[Updated on: Thu, 18 September 2008 18:32] Report message to a moderator |
|
|
|
| Re: Performance problems on heavily-loaded VE [message #33746 is a reply to message #33026] |
Thu, 06 November 2008 16:23 |
seanfulton
Messages: 105
Registered: May 2007
|
Senior Member |
|
|
| piavlo wrote on Thu, 18 September 2008 05:35 |
Can the VE be stoped/restarted with vzctl after the problems begin? If the POP/IMAP/SMTP daemons be restarted insive VE do they resume to work ok?
Do you have any nfs mount in VE? What is the state of the
POP/IMAP/SMTP daemon on VE? Maybe they are stuck on IO in D state?
try to strace/lsof them.
|
The VE can be restarted, but restarting the daemons inside the VE also works. There is no NFS inside the VE. The only thing I can find is that during the time when they hang, the sendmail processes are stuck but not in the D state, they seem to be running from all appearances. Also, this happens while we are running vzdump on the HN. Testing during the last few weeks has proven that this problem goes away if we don't run vzdump. (compress/no compress doesn't matter).
The thing that is strange is that when the vzdump process completes, everything stays hung in the VE, it never returns to normal.
There is another thread on this topic here:
http://forum.openvz.org/index.php?t=tree&goto=33745& &srch=IO+activity#msg_33745
|
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
