OpenVZ Forum - RDF feed
type: channel
items: TCP Packets get lost under moderate network load
items: Re: TCP Packets get lost under moderate network load

TCP Packets get lost under moderate network load
type: item
description: Hello! I've filed this as a bug (http://bugzilla.openvz.org/show_bug.cgi?id=1156), but I was hoping to get some feedback/suggestions to make sure I'm not doing something idiotic. I apologize in advance for the length of this note... Background: I'm converting a fairly basic legacy webserver to an OpenVZ VPS. I've been using OpenVZ for a few months on some infrastructure stuff and it's been working fine, but this is the first production server I've tried converting. It's running apache and php, gets about 2000 hits/min (mostly images, php is just for some templates). The new OpenVZ machine is a Dell, dual Xeon 3G, 4G ram, this new VPS is the only thing running on it. Issue: The morning after I converted, http monitors started to throw alerts just as traffic was ramping up. Upon logging in, I saw 500+ sockets in SYN_RECV state. Restarting the webserver would clear them out, but within a minute or two they would reach 100+. Also, there were many "Orphaned socket dropped" messages in the kernel logs, but no barriers were reached. After many false leads, I discovered that the problem was very similar to the problem reported by Max Deineko in January of 2008: http://forum.openvz.org/index.php?t=msg&goto=25678 Unfortunately no resolution was posted. I have reverted back to the original server and done some testing, but unfortunately I have to break our live site to do so because the issue only occurs when the VPS is hit with many different IPs. I can't replicate the behavior even by running ab from ~7 machines. (Doing that only makes load climb and the net link saturate, but SYN_RECV states do not pile up.) Possible clues: -Out of desperation I turned off iptables entirely (no rules, no modules loaded) on both the HN & VPS, which appeared to clear up everything. -The issue reappears with iptables turned ON on the HN but turned OFF on the VPS (IPTABLES="" in /etc/vz/vz.conf). -I've tried logging all INVALID state packets in case it was some sort of conntrack issue, but I sa…
creator: klathor
date: 2009-01-22T22:22:56-00:00

Re: TCP Packets get lost under moderate network load
type: item
description: Hello, Quote: -Out of desperation I turned off iptables entirely (no rules, no modules loaded) on both the HN & VPS, which appeared to clear up everything. -The issue reappears with iptables turned ON on the HN but turned OFF on the VPS (IPTABLES="" in /etc/vz/vz.conf). Sorry, I didn't catch you. Why doesn't this mean that the problem is because of the iptables rules on the HN? Quote: tcpdump (same thing seen on HN and VPS) Are you sure that we get exactly the same output inside VPS? Is there anything in logs or in dmesg? could you please attach the full output of /proc/user_beancounters? And please attach the output of the following files: - /proc/net/netstat - /proc/net/softnet_stat - /proc/net/snmp - /proc/net/sockstat but not as is. Could you please collect information in the dynamics i.e. when a problem appears collect information to obtain what changes and what kind of errors increase. P.S. You'd better attach the requested information to your bug report instead of attaching here.
creator: maratrus
date: 2009-01-23T11:28:51-00:00

Re: TCP Packets get lost under moderate network load
type: item
description: Quote: Quote: -Out of desperation I turned off iptables entirely (no rules, no modules loaded) on both the HN & VPS, which appeared to clear up everything. -The issue reappears with iptables turned ON on the HN but turned OFF on the VPS (IPTABLES="" in /etc/vz/vz.conf). Sorry, I didn't catch you. Why doesn't this mean that the problem is because of the iptables rules on the HN? Actually that's a good question... I assumed the rules were ok since they're pretty simple and the problems aren't constant. I will try another test with just the modules loaded and no rules. Quote: Quote: tcpdump (same thing seen on HN and VPS) Are you sure that we get exactly the same output inside VPS? I haven't done dumps simultaneously on the HN and VPS, but I do see the same behavior either way (syn+ack sent multiple times, responding ack ignored). I can do dumps simultaneously if you wish. Quote: Is there anything in logs or in dmesg? The only thing I see are the infamous "Orphaned socket dropped" messages in dmesg. They start about 2-3 mins after the SYN_RECV states start to pile up and repeat a lot. Quote: could you please attach the full output of /proc/user_beancounters? And please attach the output of the following files: - /proc/net/netstat - /proc/net/softnet_stat - /proc/net/snmp - /proc/net/sockstat but not as is. Could you please collect information in the dynamics i.e. when a problem appears collect information to obtain what changes and what kind of errors increase. P.S. You'd better attach the requested information to your bug report instead of attaching here. Will do. Unfortunately I'm out of the office till Monday but I'll get to it asap. Thank you very much for the help. -JayKim
creator: klathor
date: 2009-01-23T13:52:40-00:00

Re: TCP Packets get lost under moderate network load
type: item
description: Ok so I'm an idiot. I tried replicating my iptables failures but was surprised to find out that the problem happened regardless of whether or not iptables rules/modules were loaded. Looks like the times I tested and it succeeded were times of relatively low load. Long story short, it was an apache misconfig. MaxClients was getting overridden by an apache include, made it seem that apache wasn't spawning new children but somehow dropping connections instead. I just happened to miss the error messages because the logs are filled with cruft. In the end I just reset everything external to vz (iptables, sysctls, syncookies), tweaked the ubc limits and apache maxclients/keepalivetimeout and everything works fine now. Sorry for the time wastage, and thanks for the help (at least I learned more in the process).
creator: klathor
date: 2009-01-27T15:03:33-00:00

Re: TCP Packets get lost under moderate network load
type: item
description: Thanks for reporting the end of that story and the reason of that behavior. It's really important because I suppose somebody faced with or will face with similar problem. Thank you.
creator: maratrus
date: 2009-01-28T07:29:25-00:00