| Debug numtcpsock growing without bounds [message #52121] |
Mon, 13 July 2015 13:58  |
stompro
Messages: 3
Registered: July 2015
Location: Moorhead, MN
|
Junior Member |
|
|
Hello, I'm trying to figure out what numtcpsock is counting in my container. The number just keeps climbing, but the number of connections stays steady according to the various methods of showing connections (netstat, ss -s, /proc/sockstat)
How can I reconcile the numtcpsock number with what the container reports?
The following shows that I have 507 connections, 114 in time_wait, but numtcpsock says that 1711 socket connections are in use? How do I track down why openvz thinks that there are 1204 more connections than the container knows about?
# cd /proc/net && netstat -nat | wc && netstat -na |grep TIME_WAIT | wc && cat sockstat && cat sockstat6 && grep numtcpsock /proc/user_beancounters && ss -s && uname -a
507 3044 40534
114 684 9120
sockets: used 7222
TCP: inuse 216 orphan 0 tw 608 alloc 2721 mem 1575
UDP: inuse 8 mem 11
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
TCP6: inuse 175
UDP6: inuse 4
UDPLITE6: inuse 0
RAW6: inuse 0
FRAG6: inuse 0 memory 0
numtcpsock 1711 1717 9048 9048 0
Total: 7222 (kernel 0)
TCP: 3329 (estab 362, closed 2938, orphaned 0, synrecv 0, timewait 608/0), ports 0
Transport Total IP IPv6
* 0 - -
RAW 0 0 0
UDP 12 8 4
TCP 391 216 175
INET 403 224 179
FRAG 0 0 0
Linux virt-egapp1 2.6.32-39-pve #1 SMP Wed Jun 24 06:39:42 CEST 2015 x86_64 GNU/Linux
I'm running proxmox 3.4 using openvz with a Debian 8 template.
Thanks
Josh
|
|
|
|
|
|
| Re: Debug numtcpsock growing without bounds [message #52130 is a reply to message #52124] |
Fri, 24 July 2015 13:49  |
stompro
Messages: 3
Registered: July 2015
Location: Moorhead, MN
|
Junior Member |
|
|
I think I have this figured out.
The problem had nothing to do with OpenVZ, other than being apparent because of how OpenVZ meters resources out.
The problem was in a perl module that apache was loading, it was making a http request and calling shutdown on the connection at the end, but not calling close also on the connection, so the file descriptor was never cleared but it was still counted as a tcp connection by user_beancounters. Connections that are shutdown but not closed are not shown in netstat or ss -s. These can also be caused by socket connection that are allocated, but never connected, when there is a socket call but never a connect that follows it. The socket FD (File Descriptor) just hangs around until the program exits.
To view these connections use the command
or to just get a count
lsof | grep " sock " | wc -l
The results look like this, here are several processes that each have 1-2 orphan sockets. These are counted in the numtcpsock total even though they don't show up as connections.
udevd 151 root 4u sock 0,6 0t0 22177 can't identify protocol
rpcbind 828 root 4u sock 0,6 0t0 24640 can't identify protocol
sudo 28430 root 5u sock 0,6 0t0 74035691 can't identify protocol
/usr/sbin 28886 root 3u sock 0,6 0t0 74055248 can't identify protocol
/usr/sbin 28886 root 5u sock 0,6 0t0 74055252 can't identify protocol
/usr/sbin 28903 opensrf 3u sock 0,6 0t0 74055248 can't identify protocol
/usr/sbin 28903 opensrf 5u sock 0,6 0t0 74055252 can't identify protocol
/usr/sbin 28904 opensrf 3u sock 0,6 0t0 74055248 can't identify protocol
/usr/sbin 28904 opensrf 5u sock 0,6 0t0 74055252 can't identify protocol
/usr/sbin 28906 opensrf 3u sock 0,6 0t0 74055248 can't identify protocol
/usr/sbin 28906 opensrf 5u sock 0,6 0t0 74055252 can't identify protocol
/usr/sbin 28907 opensrf 3u sock 0,6 0t0 74055248 can't identify protocol
/usr/sbin 28907 opensrf 5u sock 0,6 0t0 74055252 can't identify protocol
/usr/sbin 28908 opensrf 3u sock 0,6 0t0 74055248 can't identify protocol
/usr/sbin 28908 opensrf 5u sock 0,6 0t0 74055252 can't identify protocol
/usr/sbin 28910 opensrf 3u sock 0,6 0t0 74055248 can't identify protocol
I used strace on the apache processes to find which sockets were being shutdown but not closed. Then found the perl code that was just calling shutdown and fixed that, and now there is no more problem.
Josh
|
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
