| load spikes to 150-290 on HN [message #28060] |
Fri, 07 March 2008 11:52  |
sara3
Messages: 38
Registered: February 2008
|
Member |
|
|
Hello
i have sudden unexplained load spikes on the hardware node
here is my bean counters
cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
136: kmemsize 5838686 10451272 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 94335 120403 2147483647 2147483647 0
shmpages 219 731 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 62 134 267 268 0
physpages 29767 48439 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 43634 53789 6144 2147483647 0
numtcpsock 28 66 2147483647 2147483647 0
numflock 6 13 2147483647 2147483647 0
numpty 0 0 255 255 0
numsiginfo 5 22 1024 1024 0
tcpsndbuf 274256 926936 2147483647 2147483647 0
tcprcvbuf 458752 696492 2147483647 2147483647 0
othersockbuf 17760 605356 2147483647 2147483647 0
dgramrcvbuf 0 11960 47483647 147483647 0
numothersock 18 49 2147483647 2147483647 0
dcachesize 852338 900178 2147483647 2147483647 0
numfile 2982 4502 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 398 400 400 400 245
138: kmemsize 6259105 21805746 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 62541 240161 2147483647 2147483647 0
shmpages 14 14 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 74 328 32567 32567 0
physpages 14785 121721 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 22566 122026 6144 2147483647 0
numtcpsock 38 228 2147483647 2147483647 0
numflock 10 95 2147483647 2147483647 0
numpty 0 1 255 255 0
numsiginfo 0 150 1024 1024 0
tcpsndbuf 459540 2982872 2147483647 2147483647 0
tcprcvbuf 622592 3777732 2147483647 2147483647 0
othersockbuf 13320 1177960 2147483647 2147483647 0
dgramrcvbuf 0 82080 47483647 147483647 0
numothersock 16 281 2147483647 2147483647 0
dcachesize 375978 592910 2147483647 2147483647 0
numfile 2995 7685 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 33 400 400 0
144: kmemsize 7330763 39453623 182802841 201083125 0
lockedpages 0 0 8925 8925 0
privvmpages 40093 654747 305911 336502 33610
shmpages 149 805 30591 30591 0
dummy 0 0 0 0 0
numproc 82 485 8000 8000 0
physpages 19434 228719 0 2147483647 0
vmguarpages 0 0 305911 2147483647 0
oomguarpages 19509 232475 305911 2147483647 0
numtcpsock 103 298 8000 8000 0
numflock 56 377 1000 1100 0
numpty 0 1 512 512 0
numsiginfo 11 387 1024 1024 0
tcpsndbuf 959040 3285600 28166280 60934280 0
tcprcvbuf 1303748 3583756 28166280 60934280 0
othersockbuf 2220 2862648 14083140 46851140 0
dgramrcvbuf 0 10416 14083140 14083140 0
numothersock 2 342 8000 8000 0
dcachesize 279314 1100958 39924069 41121792 0
numfile 2461 14010 71392 71392 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 48 200 200 0
137: kmemsize 2233967 6569195 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 24608 41516 2147483647 2147483647 0
shmpages 0 0 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 22 76 32567 32567 0
physpages 4661 21212 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 11166 23002 6144 2147483647 0
numtcpsock 9 34 2147483647 2147483647 0
numflock 6 63 2147483647 2147483647 0
numpty 0 1 255 255 0
numsiginfo 0 3 1024 1024 0
tcpsndbuf 91020 435120 2147483647 2147483647 0
tcprcvbuf 147456 0 2147483647 2147483647 0
othersockbuf 17760 73944 2147483647 2147483647 0
dgramrcvbuf 0 26640 47483647 147483647 0
numothersock 12 38 2147483647 2147483647 0
dcachesize 110152 196700 2147483647 2147483647 0
numfile 1372 2894 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 14 14 400 400 0
139: kmemsize 4940371 13141172 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 54594 138526 2147483647 2147483647 0
shmpages 223 2159 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 65 177 32567 32567 0
physpages 9785 75007 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 19144 75041 6144 2147483647 0
numtcpsock 29 116 2147483647 2147483647 0
numflock 8 12 2147483647 2147483647 0
numpty 0 5 255 255 0
numsiginfo 0 94 1024 1024 0
tcpsndbuf 404040 2069040 2147483647 2147483647 0
tcprcvbuf 475136 2457580 2147483647 2147483647 0
othersockbuf 39532 223184 2147483647 2147483647 0
dgramrcvbuf 0 26640 47483647 147483647 0
numothersock 30 77 2147483647 2147483647 0
dcachesize 381036 486200 2147483647 2147483647 0
numfile 2436 3530 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 33 400 400 0
149: kmemsize 3370556 3984508 30138368 33152204 0
lockedpages 0 0 1471 1471 0
privvmpages 40001 46809 466667 513333 0
shmpages 91 91 46666 46666 0
dummy 0 0 0 0 0
numproc 51 62 1333 1333 0
physpages 6945 16235 0 2147483647 0
vmguarpages 0 0 466667 2147483647 0
oomguarpages 10555 16235 466667 2147483647 0
numtcpsock 23 32 1333 1333 0
numflock 8 10 1000 1100 0
numpty 0 0 133 133 0
numsiginfo 0 19 1024 1024 0
tcpsndbuf 204240 248804 4586154 10046122 0
tcprcvbuf 376832 567828 4586154 10046122 0
othersockbuf 11100 18276 2293077 7753045 0
dgramrcvbuf 0 8364 2293077 2293077 0
numothersock 18 24 1333 1333 0
dcachesize 183774 212436 6567518 6764544 0
numfile 1355 1579 11744 11744 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 33 200 200 0
148: kmemsize 2212756 5026269 182802841 201083125 0
lockedpages 0 0 8925 8925 0
privvmpages 29877 71793 393216 393216 0
shmpages 4236 4908 30591 30591 0
dummy 0 0 0 0 0
numproc 24 65 8000 8000 0
physpages 3805 41844 0 2147483647 0
vmguarpages 0 0 305911 2147483647 0
oomguarpages 6560 41844 305911 2147483647 0
numtcpsock 21 30 8000 8000 0
numflock 0 9 1000 1100 0
numpty 0 0 512 512 0
numsiginfo 0 25 1024 1024 0
tcpsndbuf 194844 265368 28166280 60934280 0
tcprcvbuf 344064 575848 28166280 60934280 0
othersockbuf 13320 23156 14083140 46851140 0
dgramrcvbuf 0 11960 14083140 14083140 0
numothersock 9 22 8000 8000 0
dcachesize 184968 304112 39924069 41121792 0
numfile 912 1844 71392 71392 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 33 200 200 0
146: kmemsize 10587662 19536296 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 114635 196626 2147483647 2147483647 0
shmpages 4108 12300 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 117 219 32567 32567 0
physpages 40653 148380 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 58506 148902 6144 2147483647 0
numtcpsock 97 139 2147483647 2147483647 0
numflock 9 22 2147483647 2147483647 0
numpty 0 1 255 255 0
numsiginfo 0 105 1024 1024 0
tcpsndbuf 1813912 2567516 2147483647 2147483647 0
tcprcvbuf 1589248 2033836 2147483647 2147483647 0
othersockbuf 50108 1550580 2147483647 2147483647 0
dgramrcvbuf 0 140904 47483647 147483647 0
numothersock 55 122 2147483647 2147483647 0
dcachesize 490836 592136 2147483647 2147483647 0
numfile 4485 6236 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 50 66 400 400 0
145: kmemsize 6381252 14681982 2147483647 2147483647 0
lockedpages 0 2 2147483647 2147483647 0
privvmpages 76249 161947 2147483647 2147483647 0
shmpages 142 798 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 80 172 32567 32567 0
physpages 29268 76630 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 36749 83947 6144 2147483647 0
numtcpsock 47 134 2147483647 2147483647 0
numflock 9 27 2147483647 2147483647 0
numpty 0 1 255 255 0
numsiginfo 0 78 1024 1024 0
tcpsndbuf 717060 2020200 2147483647 2147483647 0
tcprcvbuf 770048 1405960 2147483647 2147483647 0
othersockbuf 17760 677572 2147483647 2147483647 0
dgramrcvbuf 0 27360 47483647 147483647 0
numothersock 26 98 2147483647 2147483647 0
dcachesize 351320 441942 2147483647 2147483647 0
numfile 2535 4124 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 49 97 400 400 0
143: kmemsize 4726414 9096118 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 56878 122330 2147483647 2147483647 0
shmpages 219 555 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 63 150 32567 32567 0
physpages 16626 69353 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 20916 74471 6144 2147483647 0
numtcpsock 37 75 2147483647 2147483647 0
numflock 8 40 2147483647 2147483647 0
numpty 0 1 255 255 0
numsiginfo 0 53 1024 1024 0
tcpsndbuf 610500 943500 2147483647 2147483647 0
tcprcvbuf 606208 969064 2147483647 2147483647 0
othersockbuf 17760 519592 2147483647 2147483647 0
dgramrcvbuf 0 16736 47483647 147483647 0
numothersock 17 117 2147483647 2147483647 0
dcachesize 172534 233792 2147483647 2147483647 0
numfile 2074 3572 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 99 99 400 400 0
142: kmemsize 9923301 13876884 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 70267 110804 2147483647 2147483647 0
shmpages 145 1441 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 97 136 32567 32567 0
physpages 23546 70348 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 37308 70352 6144 2147483647 0
numtcpsock 55 95 2147483647 2147483647 0
numflock 6 12 2147483647 2147483647 0
numpty 0 2 255 255 0
numsiginfo 10 78 1024 1024 0
tcpsndbuf 632700 1445220 2147483647 2147483647 0
tcprcvbuf 901120 1295368 2147483647 2147483647 0
othersockbuf 28860 1077608 2147483647 2147483647 0
dgramrcvbuf 0 62192 47483647 147483647 0
numothersock 27 47 2147483647 2147483647 0
dcachesize 952590 985186 2147483647 2147483647 0
numfile 9084 9292 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 60 60 400 400 0
116: kmemsize 4977804 7061303 2147483647 2147483647 0
lockedpages 0 1 2147483647 2147483647 0
privvmpages 73187 86155 2147483647 2147483647 0
shmpages 14 686 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 67 103 32567 32567 0
physpages 18179 43271 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 32642 43271 6144 2147483647 0
numtcpsock 25 48 2147483647 2147483647 0
numflock 9 15 2147483647 2147483647 0
numpty 0 0 255 255 0
numsiginfo 0 21 1024 1024 0
tcpsndbuf 222000 561660 2147483647 2147483647 0
tcprcvbuf 409600 581804 2147483647 2147483647 0
othersockbuf 26212 407320 2147483647 2147483647 0
dgramrcvbuf 0 4440 47483647 147483647 0
numothersock 26 40 2147483647 2147483647 0
dcachesize 153988 202882 2147483647 2147483647 0
numfile 2314 3659 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 84 84 400 400 0
111: kmemsize 2546173 3604579 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 39615 51096 2147483647 2147483647 0
shmpages 91 763 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 36 60 32567 32567 0
physpages 5971 16769 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 8934 16824 6144 2147483647 0
numtcpsock 23 31 2147483647 2147483647 0
numflock 3 9 2147483647 2147483647 0
numpty 0 1 255 255 0
numsiginfo 0 13 1024 1024 0
tcpsndbuf 204240 256488 2147483647 2147483647 0
tcprcvbuf 376832 484188 2147483647 2147483647 0
othersockbuf 13320 20068 2147483647 2147483647 0
dgramrcvbuf 0 17080 47483647 147483647 0
numothersock 11 23 2147483647 2147483647 0
dcachesize 95540 114086 2147483647 2147483647 0
numfile 1275 1638 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 33 400 400 0
0: kmemsize 35002855 81718407 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 415751 643657 2147483647 2147483647 0
shmpages 1007 2303 2147483647 2147483647 0
dummy 0 0 2147483647 2147483647 0
numproc 452 983 2147483647 2147483647 0
physpages 52369 189876 2147483647 2147483647 0
vmguarpages 0 0 2147483647 2147483647 0
oomguarpages 83380 193903 2147483647 2147483647 0
numtcpsock 79 205 2147483647 2147483647 0
numflock 64 369 2147483647 2147483647 0
numpty 5 10 2147483647 2147483647 0
numsiginfo 6 33 2147483647 2147483647 0
tcpsndbuf 1117480 4311312 2147483647 2147483647 0
tcprcvbuf 1296556 3089424 2147483647 2147483647 0
othersockbuf 57292 1416616 2147483647 2147483647 0
dgramrcvbuf 2220 279072 2147483647 2147483647 0
numothersock 84 690 2147483647 2147483647 0
dcachesize 2841890 3957322 2147483647 2147483647 0
numfile 23701 30977 2147483647 2147483647 0
dummy 0 0 2147483647 2147483647 0
dummy 0 0 2147483647 2147483647 0
dummy 0 0 2147483647 2147483647 0
numiptent 902 1772 2147483647 2147483647 0
# uname -a
Linux xxxx.xxyyyyy.com 2.6.18-ovz028stab053.5-enterprise #1 SMP Wed Feb 13 14:33:25 MSK 2008 i686 i686 i386 GNU/Linux
when i do vzlist
can i do something to prevent one VE from overloading the HN or an i tweak settings of my VEs to prevent this
thanks for your help
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: load spikes to 150-290 on HN [message #28072 is a reply to message #28069] |
Fri, 07 March 2008 13:26  |
ugob
Messages: 271
Registered: March 2007
|
Senior Member |
|
|
Well you should:
- Raise numiptent on 136
- Raise privvmpages on 144 (do you know why it is allocating that much memory?)
You should run 'vzmemcheck -vA' on the HN when high load happens. You'll see how much memory is used by each VE.
How much ram do you have?
If not installed, install sysstat and wait one day, then go into /var/log/sa/ to see the performance reports per date (sarXX).
You're saying that some VE have 300+ more process when it happens. It is important to see what kind of process it is. httpd? sendmail? java?
Please read the manual before asking questions:
http://download.openvz.org/doc/OpenVZ-Users-Guide.pdf
Please have a look at the wiki before asking questions:
http://wiki.openvz.org/Main_Page
|
|
|
|
|
|
|
|
|
|
| Re: load spikes to 150-290 on HN [message #28100 is a reply to message #28094] |
Sat, 08 March 2008 15:08 |
ugob
Messages: 271
Registered: March 2007
|
Senior Member |
|
|
First step is to find if you can increase the privvmpages for this VM. Try vzsplit -n1 | grep PRIVVM and let us know what you get.
If you can't, consider increasing the physical RAM in the server.
At the same time, you should try to find what is causing this. You could start by using tail -f /var/log/httpd/access_log to see what is hitting your server.
Finally, you could try to tweak apache so that you have a max number of apache processes, or use threads instead of forks (doesn't work with php, though, I think), or reduce the memory footprint of the individual httpd processes.
What is 144 running exactly?
Please read the manual before asking questions:
http://download.openvz.org/doc/OpenVZ-Users-Guide.pdf
Please have a look at the wiki before asking questions:
http://wiki.openvz.org/Main_Page
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
