OpenVZ Forum


Home » General » Support » Debug numtcpsock growing without bounds (networking)
Debug numtcpsock growing without bounds [message #52121] Mon, 13 July 2015 13:58 Go to next message
stompro is currently offline  stompro
Messages: 3
Registered: July 2015
Location: Moorhead, MN
Junior Member
From: *larl.org
Hello, I'm trying to figure out what numtcpsock is counting in my container. The number just keeps climbing, but the number of connections stays steady according to the various methods of showing connections (netstat, ss -s, /proc/sockstat)

How can I reconcile the numtcpsock number with what the container reports?

The following shows that I have 507 connections, 114 in time_wait, but numtcpsock says that 1711 socket connections are in use? How do I track down why openvz thinks that there are 1204 more connections than the container knows about?


# cd /proc/net && netstat -nat | wc && netstat -na |grep TIME_WAIT | wc && cat sockstat && cat sockstat6 && grep numtcpsock /proc/user_beancounters && ss -s && uname -a
    507    3044   40534
    114     684    9120
sockets: used 7222
TCP: inuse 216 orphan 0 tw 608 alloc 2721 mem 1575
UDP: inuse 8 mem 11
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
TCP6: inuse 175
UDP6: inuse 4
UDPLITE6: inuse 0
RAW6: inuse 0
FRAG6: inuse 0 memory 0
            numtcpsock                   1711                 1717                 9048                 9048                    0
Total: 7222 (kernel 0)
TCP:   3329 (estab 362, closed 2938, orphaned 0, synrecv 0, timewait 608/0), ports 0

Transport Total     IP        IPv6
*         0         -         -
RAW       0         0         0
UDP       12        8         4
TCP       391       216       175
INET      403       224       179
FRAG      0         0         0

Linux virt-egapp1 2.6.32-39-pve #1 SMP Wed Jun 24 06:39:42 CEST 2015 x86_64 GNU/Linux



I'm running proxmox 3.4 using openvz with a Debian 8 template.

Thanks
Josh
Re: Debug numtcpsock growing without bounds [message #52124 is a reply to message #52121] Fri, 17 July 2015 21:21 Go to previous messageGo to next message
stompro is currently offline  stompro
Messages: 3
Registered: July 2015
Location: Moorhead, MN
Junior Member
From: *larl.org
I've narrowed it down to apache, when I restart apache the numtcpsock goes back to normal. So Apache is somehow holding onto connections that are invisible?

Josh
Re: Debug numtcpsock growing without bounds [message #52130 is a reply to message #52124] Fri, 24 July 2015 13:49 Go to previous message
stompro is currently offline  stompro
Messages: 3
Registered: July 2015
Location: Moorhead, MN
Junior Member
From: *midco.net
I think I have this figured out.

The problem had nothing to do with OpenVZ, other than being apparent because of how OpenVZ meters resources out.

The problem was in a perl module that apache was loading, it was making a http request and calling shutdown on the connection at the end, but not calling close also on the connection, so the file descriptor was never cleared but it was still counted as a tcp connection by user_beancounters. Connections that are shutdown but not closed are not shown in netstat or ss -s. These can also be caused by socket connection that are allocated, but never connected, when there is a socket call but never a connect that follows it. The socket FD (File Descriptor) just hangs around until the program exits.

To view these connections use the command
lsof | grep " sock " 

or to just get a count
lsof | grep " sock " | wc -l 


The results look like this, here are several processes that each have 1-2 orphan sockets. These are counted in the numtcpsock total even though they don't show up as connections.

udevd       151             root    4u     sock                0,6      0t0      22177 can't identify protocol
rpcbind     828             root    4u     sock                0,6      0t0      24640 can't identify protocol
sudo      28430             root    5u     sock                0,6      0t0   74035691 can't identify protocol
/usr/sbin 28886             root    3u     sock                0,6      0t0   74055248 can't identify protocol
/usr/sbin 28886             root    5u     sock                0,6      0t0   74055252 can't identify protocol
/usr/sbin 28903          opensrf    3u     sock                0,6      0t0   74055248 can't identify protocol
/usr/sbin 28903          opensrf    5u     sock                0,6      0t0   74055252 can't identify protocol
/usr/sbin 28904          opensrf    3u     sock                0,6      0t0   74055248 can't identify protocol
/usr/sbin 28904          opensrf    5u     sock                0,6      0t0   74055252 can't identify protocol
/usr/sbin 28906          opensrf    3u     sock                0,6      0t0   74055248 can't identify protocol
/usr/sbin 28906          opensrf    5u     sock                0,6      0t0   74055252 can't identify protocol
/usr/sbin 28907          opensrf    3u     sock                0,6      0t0   74055248 can't identify protocol
/usr/sbin 28907          opensrf    5u     sock                0,6      0t0   74055252 can't identify protocol
/usr/sbin 28908          opensrf    3u     sock                0,6      0t0   74055248 can't identify protocol
/usr/sbin 28908          opensrf    5u     sock                0,6      0t0   74055252 can't identify protocol
/usr/sbin 28910          opensrf    3u     sock                0,6      0t0   74055248 can't identify protocol


I used strace on the apache processes to find which sockets were being shutdown but not closed. Then found the perl code that was just calling shutdown and fixed that, and now there is no more problem.

Josh
Previous Topic: cpulimit - core / CPU?
Next Topic: Is Grsecurity still available or has it been removed?
Goto Forum:
  


Current Time: Sat Dec 14 13:44:43 GMT 2019