OpenVZ Forum


Home » General » Support » TCP hangs up (TCP hangs up for big connexion rates)
TCP hangs up [message #38565] Sat, 02 January 2010 23:08 Go to next message
Sékiltoyai is currently offline  Sékiltoyai
Messages: 7
Registered: August 2009
Location: France
Junior Member
Hello,

Happy new year !

I have some problem with tcp on my VEs. The symptom is that some tcp (not tested with udp) connexions hang up with big files. It is very clear with ftp. And doesn't happen with http, because, i guess, apache can't reach same connexion rates (the problem only occur when i send data from the server).

I first assumed it was a netfilter problem but it occur even when policies are set to access. And it disappear when i set tcpsndbuf to really huge values (obviously it cannot be considered as a good solution Smile ).

I have seen some problems with tcpsndbuf on the bugzilla but it seems to have been solved in past kernel versions (mine is a Debian 2.6.26-2-openvz-amd64 ). Is it really the case ?
Shall i post a bug report or error is of my own ?

I have put in attachment the /proc/user_beancounters of the testing vps and the /etc/vz/conf/240.conf.
With this conf, the ftp transfer stop at approximately 6 or 7 megabytes, for a file of 10 megabytes.

Thanks for your help.

Best regards.

Report message to a moderator

Re: TCP hangs up [message #38627 is a reply to message #38565] Fri, 08 January 2010 12:04 Go to previous messageGo to next message
nuno is currently offline  nuno
Messages: 43
Registered: January 2010
Member
The tcp buffers are in bytes. So you have 8MB for them. It's not much for fast machines on fast networks. Maybe 32MB will do the trick. Don't forget to increase kmemsize (tcpsndbuf memory is accounted in kmemsize too).

If you really need the RAM maybe if you disable tcp window scaling and other tcp performance parameters, you can get away with less than 32MB... Test it and report back.

Regards,

Report message to a moderator

Re: TCP hangs up [message #38630 is a reply to message #38565] Sat, 09 January 2010 01:27 Go to previous messageGo to next message
Sékiltoyai is currently offline  Sékiltoyai
Messages: 7
Registered: August 2009
Location: France
Junior Member
Hello,

Thanks for your answer.

Seems to be ok with your recommend. I have put tcpsndbuf to 48MB:64MB and i can send a 8GB file on my fastest connexion.

Here is the result in ubc :
240: 
kmemsize   2358102  3607954  20971520  31457280  0
...
tcpsndbuf  106952   9038280  50331648  67108864  0
...

So i think a barrier around 10MB will be suitable.


But i have two questions :
- Is it a normal behaviour that programs (or kernel network driver, i don't know where is the problem) hangs up in case of this tcp buffer shortage ? Isn't there a bug in the kernel, which should simply reject new send() system calls when its buffers are full ?
- Why a call to vzsplit (with a division in 6 vps) gives me a tcpsndbuf of approximately 5.8MB:11MB, so a barrier of 5.8MB, which is far below the needed barrier ?

[Updated on: Sat, 09 January 2010 01:29]

Report message to a moderator

Re: TCP hangs up [message #38631 is a reply to message #38630] Sat, 09 January 2010 09:10 Go to previous messageGo to next message
nuno is currently offline  nuno
Messages: 43
Registered: January 2010
Member
Sékiltoyai wrote on Sat, 09 January 2010 01:27
Hello,

Thanks for your answer.

Seems to be ok with your recommend. I have put tcpsndbuf to 48MB:64MB and i can send a 8GB file on my fastest connexion.

Here is the result in ubc :
240: 
kmemsize   2358102  3607954  20971520  31457280  0
...
tcpsndbuf  106952   9038280  50331648  67108864  0
...

So i think a barrier around 10MB will be suitable.



Don't forget that you can have 100 of those connections. Testing is your best bet.

Sékiltoyai wrote on Sat, 09 January 2010 01:27

But i have two questions :
- Is it a normal behaviour that programs (or kernel network driver, i don't know where is the problem) hangs up in case of this tcp buffer shortage ? Isn't there a bug in the kernel, which should simply reject new send() system calls when its buffers are full ?
- Why a call to vzsplit (with a division in 6 vps) gives me a tcpsndbuf of approximately 5.8MB:11MB, so a barrier of 5.8MB, which is far below the needed barrier ?



After faling to allocate memory the TCP connection may be FUBAR (damaged beyond repair).

I never rely on vzsplit. Maybe those numbers will be fine for a 1mbit link? Or for a VPS provider that wants to put 200 containers in a small server? It's a good question for the maintainers.

Regards,

Report message to a moderator

Re: TCP hangs up [message #38636 is a reply to message #38631] Sun, 10 January 2010 17:43 Go to previous messageGo to next message
Sékiltoyai is currently offline  Sékiltoyai
Messages: 7
Registered: August 2009
Location: France
Junior Member
Hello,

Thanks again for your advices.


nuno wrote on Sat, 09 January 2010 10:10

Don't forget that you can have 100 of those connections. Testing is your best bet.


I've made some tests and, as you have wroten, if i make several ftp transfers in the same time, the tcpsndbuf explode.
For only four parallel transfers, i have to save at least 20M of kernel memory for tcp connexions, it's too big for an auxiliary vps (this one is going to be dedicated to the backup, and i have only 1G RAM).

It doesn't seem to be possible to control independantly tcp parameters (via sysctl) on each vps. And I don't want to disable tcp window scale for the whole server. So, i have made another test. I have limited trafic with some simple netfilter rules :

-A FORWARD -s 192.168.2.40/32 -i venet0 -o eth0 -j LIMITER
-A LIMITER -m limit --limit 2/sec -j ACCEPT 
-A LIMITER -m statistic --mode nth --every 10000 -j LOG --log-prefix "[FOR_INTERNET]limited_packet" --log-tcp-options --log-ip-options 
-A LIMITER -j DROP


And for 2 connexions, the tcp buffer falls down to 244K.
I think i'll set up a consistent trafic limiting with tc, but is it a good way of solving my problem ?


nuno wrote on Sat, 09 January 2010 10:10

After faling to allocate memory the TCP connection may be FUBAR (damaged beyond repair).



And how can we check this ? And how can we "repair" a connexion in FUBAR state ? Is it automatic ?

Thank,

Regards.

[Updated on: Sun, 10 January 2010 17:43]

Report message to a moderator

Re: TCP hangs up [message #38637 is a reply to message #38636] Sun, 10 January 2010 18:43 Go to previous messageGo to next message
nuno is currently offline  nuno
Messages: 43
Registered: January 2010
Member
If it's your server and your container just increase the parameters and be happy.

If the tcp connection is FUBAR, the transfer will fail. That's not good, but it isn't too bad: you just restart the transfer.

Regards,

Report message to a moderator

Re: TCP hangs up [message #38639 is a reply to message #38637] Sun, 10 January 2010 19:00 Go to previous message
Sékiltoyai is currently offline  Sékiltoyai
Messages: 7
Registered: August 2009
Location: France
Junior Member
nuno wrote on Sun, 10 January 2010 19:43
If it's your server and your container just increase the parameters and be happy.


mmh, no way !

It's my server (i administrate the host system) but i must deal with each vps on this server. So I can't assign to it 1/10 of the whole RAM just for the kernel memory !

Report message to a moderator

Previous Topic: Total but temporary loss of network
Next Topic: 20 second delay with private ip address
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ] [ PDF ]

Current Time: Mon Nov 18 16:34:34 GMT 2024

Total time taken to generate the page: 0.02891 seconds
Powered by FUDforum Powered by Virtuozzo Containers