| load spikes to 150-290 on HN [message #28060] |
Fri, 07 March 2008 11:52  |
sara3
Messages: 38
Registered: February 2008
|
Member |
|
|
Hello
i have sudden unexplained load spikes on the hardware node
here is my bean counters
cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
136: kmemsize 5838686 10451272 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 94335 120403 2147483647 2147483647 0
shmpages 219 731 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 62 134 267 268 0
physpages 29767 48439 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 43634 53789 6144 2147483647 0
numtcpsock 28 66 2147483647 2147483647 0
numflock 6 13 2147483647 2147483647 0
numpty 0 0 255 255 0
numsiginfo 5 22 1024 1024 0
tcpsndbuf 274256 926936 2147483647 2147483647 0
tcprcvbuf 458752 696492 2147483647 2147483647 0
othersockbuf 17760 605356 2147483647 2147483647 0
dgramrcvbuf 0 11960 47483647 147483647 0
numothersock 18 49 2147483647 2147483647 0
dcachesize 852338 900178 2147483647 2147483647 0
numfile 2982 4502 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 398 400 400 400 245
138: kmemsize 6259105 21805746 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 62541 240161 2147483647 2147483647 0
shmpages 14 14 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 74 328 32567 32567 0
physpages 14785 121721 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 22566 122026 6144 2147483647 0
numtcpsock 38 228 2147483647 2147483647 0
numflock 10 95 2147483647 2147483647 0
numpty 0 1 255 255 0
numsiginfo 0 150 1024 1024 0
tcpsndbuf 459540 2982872 2147483647 2147483647 0
tcprcvbuf 622592 3777732 2147483647 2147483647 0
othersockbuf 13320 1177960 2147483647 2147483647 0
dgramrcvbuf 0 82080 47483647 147483647 0
numothersock 16 281 2147483647 2147483647 0
dcachesize 375978 592910 2147483647 2147483647 0
numfile 2995 7685 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 33 400 400 0
144: kmemsize 7330763 39453623 182802841 201083125 0
lockedpages 0 0 8925 8925 0
privvmpages 40093 654747 305911 336502 33610
shmpages 149 805 30591 30591 0
dummy 0 0 0 0 0
numproc 82 485 8000 8000 0
physpages 19434 228719 0 2147483647 0
vmguarpages 0 0 305911 2147483647 0
oomguarpages 19509 232475 305911 2147483647 0
numtcpsock 103 298 8000 8000 0
numflock 56 377 1000 1100 0
numpty 0 1 512 512 0
numsiginfo 11 387 1024 1024 0
tcpsndbuf 959040 3285600 28166280 60934280 0
tcprcvbuf 1303748 3583756 28166280 60934280 0
othersockbuf 2220 2862648 14083140 46851140 0
dgramrcvbuf 0 10416 14083140 14083140 0
numothersock 2 342 8000 8000 0
dcachesize 279314 1100958 39924069 41121792 0
numfile 2461 14010 71392 71392 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 48 200 200 0
137: kmemsize 2233967 6569195 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 24608 41516 2147483647 2147483647 0
shmpages 0 0 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 22 76 32567 32567 0
physpages 4661 21212 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 11166 23002 6144 2147483647 0
numtcpsock 9 34 2147483647 2147483647 0
numflock 6 63 2147483647 2147483647 0
numpty 0 1 255 255 0
numsiginfo 0 3 1024 1024 0
tcpsndbuf 91020 435120 2147483647 2147483647 0
tcprcvbuf 147456 0 2147483647 2147483647 0
othersockbuf 17760 73944 2147483647 2147483647 0
dgramrcvbuf 0 26640 47483647 147483647 0
numothersock 12 38 2147483647 2147483647 0
dcachesize 110152 196700 2147483647 2147483647 0
numfile 1372 2894 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 14 14 400 400 0
139: kmemsize 4940371 13141172 2147483647 2147483647 0
lockedpages 0 0 2147483647 2147483647 0
privvmpages 54594 138526 2147483647 2147483647 0
shmpages 223 2159 2147483647 2147483647 0
dummy 0 0 0 0 0
numproc 65 177 32567 32567 0
physpages 9785 75007 0 2147483647 0
vmguarpages 0 0 6144 2147483647 0
oomguarpages 19144 75041 6144 2147483647 0
numtcpsock 29 116 2147483647 2147483647 0
numflock 8 12 2147483647 2147483647 0
numpty 0 5 255 255 0
numsiginfo 0 94 1024 1024 0
tcpsndbuf 404040 2069040 2147483647 2147483647 0
tcprcvbuf 475136 2457580 2147483647 2147483647 0
othersockbuf 39532 223184 2147483647 2147483647 0
dgramrcvbuf 0 26640 47483647 147483647 0
numothersock 30 77 2147483647 2147483647 0
dcachesize 381036 486200 2147483647 2147483647 0
numfile 2436 3530 2147483647 2147483647 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 33 33 400 400 0
149: kmemsize 3370556 3984508 30138368 3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: load spikes to 150-290 on HN [message #28072 is a reply to message #28069] |
Fri, 07 March 2008 13:26  |
ugob
Messages: 271
Registered: March 2007
|
Senior Member |
|
|
Well you should:
- Raise numiptent on 136
- Raise privvmpages on 144 (do you know why it is allocating that much memory?)
You should run 'vzmemcheck -vA' on the HN when high load happens. You'll see how much memory is used by each VE.
How much ram do you have?
If not installed, install sysstat and wait one day, then go into /var/log/sa/ to see the performance reports per date (sarXX).
You're saying that some VE have 300+ more process when it happens. It is important to see what kind of process it is. httpd? sendmail? java?
Please read the manual before asking questions:
http://download.openvz.org/doc/OpenVZ-Users-Guide.pdf
Please have a look at the wiki before asking questions:
http://wiki.openvz.org/Main_Page
|
|
|
|
|
|
|
|
|
|
| Re: load spikes to 150-290 on HN [message #28100 is a reply to message #28094] |
Sat, 08 March 2008 15:08 |
ugob
Messages: 271
Registered: March 2007
|
Senior Member |
|
|
First step is to find if you can increase the privvmpages for this VM. Try vzsplit -n1 | grep PRIVVM and let us know what you get.
If you can't, consider increasing the physical RAM in the server.
At the same time, you should try to find what is causing this. You could start by using tail -f /var/log/httpd/access_log to see what is hitting your server.
Finally, you could try to tweak apache so that you have a max number of apache processes, or use threads instead of forks (doesn't work with php, though, I think), or reduce the memory footprint of the individual httpd processes.
What is 144 running exactly?
Please read the manual before asking questions:
http://download.openvz.org/doc/OpenVZ-Users-Guide.pdf
Please have a look at the wiki before asking questions:
http://wiki.openvz.org/Main_Page
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
