OpenVZ Forum


Home » General » Support » Performance problems on heavily-loaded VE
Performance problems on heavily-loaded VE [message #33018] Wed, 17 September 2008 19:09 Go to next message
seanfulton is currently offline  seanfulton
Messages: 105
Registered: May 2007
Senior Member
We recently moved our busy, spam-fighting mail server from a dual-XEON machine with 2 G of RAM to a CentOS-4 VE on an Intel Dual Core machine with 4 G of RAM.

At the time of the move, we had two Windows nodes running on the machine with Xen, but after NUMEROUS kernel crashes I ditched the XEN kernel and am now just using the 2.6.18-53.1.19.el5.028stab053.14PAE kernel and have not had any crashes.

In any event, there are only two VEs running on this machine; one is a dev VE with one person using it as an FTP source and next to no activity. The other is the mail VE.

During the move we generated a HUGE mail spool of mostly spam. What I am seeing now is that whenever the system load climbs above say 4 or so within the VE, it effectively stops responding to POP/IMAP/SMTP connetions. The connection opens, but the appropriate service just doesn't come up. Usually this is when sendmail starts trying to process that backed up mail queue.

What I find strange is that on the dual XEON machine with less RAM, we had no problems even as the load climed to 20 or above. We only started having connection problems when the load got up around 40 or so.

I've tuned everything I can think of an the mail VE has effectively all of the resources available. /proc/user_beancounters shows no failures, and the VE shows 4 G of RAM.

Has anyone else experienced anything like this or is able to point me in a direction? I was thinking that *perhaps* if the system load is due to processing the mail spool and that once we clear that out, it will be fine, but I am concerned.

We're currently running more than 50 other VEs on other machines in all sorts of configuration with no problems, but those are mainly web servers with static and dynamic content, and rarely push that kind of load.

Any ideas or pointers would be appreciated.

Oh yes, running apps inside the VE are sendmail, cyrus-imapd, spamassassin, spamass-milter, saslauthd.

sean

Re: Performance problems on heavily-loaded VE [message #33020 is a reply to message #33018] Wed, 17 September 2008 21:10 Go to previous messageGo to next message
tomp is currently offline  tomp
Messages: 64
Registered: August 2007
Member
Have you looked at your /proc/user_beancounters to see if you are hitting any limits in the container?
Re: Performance problems on heavily-loaded VE [message #33021 is a reply to message #33020] Wed, 17 September 2008 21:13 Go to previous messageGo to next message
seanfulton is currently offline  seanfulton
Messages: 105
Registered: May 2007
Senior Member
I did check--none. No messages in /var/log/messages or dmesg either.

sean
Re: Performance problems on heavily-loaded VE [message #33026 is a reply to message #33021] Thu, 18 September 2008 09:35 Go to previous messageGo to next message
piavlo is currently offline  piavlo
Messages: 159
Registered: January 2007
Senior Member
Can the VE be stoped/restarted with vzctl after the problems begin? If the POP/IMAP/SMTP daemons be restarted insive VE do they resume to work ok?
Do you have any nfs mount in VE? What is the state of the
POP/IMAP/SMTP daemon on VE? Maybe they are stuck on IO in D state?
try to strace/lsof them.

[Updated on: Thu, 18 September 2008 18:32]

Report message to a moderator

Re: Performance problems on heavily-loaded VE [message #33746 is a reply to message #33026] Thu, 06 November 2008 16:23 Go to previous message
seanfulton is currently offline  seanfulton
Messages: 105
Registered: May 2007
Senior Member
piavlo wrote on Thu, 18 September 2008 05:35

Can the VE be stoped/restarted with vzctl after the problems begin? If the POP/IMAP/SMTP daemons be restarted insive VE do they resume to work ok?
Do you have any nfs mount in VE? What is the state of the
POP/IMAP/SMTP daemon on VE? Maybe they are stuck on IO in D state?
try to strace/lsof them.


The VE can be restarted, but restarting the daemons inside the VE also works. There is no NFS inside the VE. The only thing I can find is that during the time when they hang, the sendmail processes are stuck but not in the D state, they seem to be running from all appearances. Also, this happens while we are running vzdump on the HN. Testing during the last few weeks has proven that this problem goes away if we don't run vzdump. (compress/no compress doesn't matter).

The thing that is strange is that when the vzdump process completes, everything stays hung in the VE, it never returns to normal.

There is another thread on this topic here:

http://forum.openvz.org/index.php?t=tree&goto=33745& &srch=IO+activity#msg_33745
Previous Topic: Iptables Logging
Next Topic: kernel Problem
Goto Forum:
  


Current Time: Sun Feb 25 15:52:00 GMT 2024

Total time taken to generate the page: 0.03188 seconds