System is experiencing very strange high load - uptime says it near 40-50, vmstat says there are around 40 blocked processes while there almost noone with blocked state (from HN): root@beta:~# vmstat 1 10 && echo "" && ps aux|grep ' D ' && echo && uptime procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 37 0 476252 102404 5950924 0 0 71 36 0 5 2 1 89 8 0 39 0 471012 102452 5955228 0 0 4864 24 2108 3046 3 3 0 95 0 38 0 468028 102460 5958752 0 0 3392 32 1779 3222 3 2 1 93 0 35 0 463636 102488 5963332 0 0 4328 112 1950 2509 2 2 0 96 0 34 0 461548 102516 5967700 0 0 4636 0 1933 2668 2 2 0 96 0 34 0 457956 102524 5973324 0 0 5452 12 1715 2038 2 1 0 97 0 35 0 447856 102524 5976828 0 0 3692 16 1799 1904 4 2 1 93 0 37 0 443300 102560 5981656 0 0 4628 20 1744 2253 7 2 2 90 0 39 0 439156 102592 5986348 0 0 4728 56 1817 2272 2 2 0 96 0 38 0 434012 102616 5991188 0 0 5024 0 2058 2738 3 2 0 96 www-data 8752 0.1 0.1 26876 9800 ? D 15:45 0:05 /usr/sbin/apache2 -k start root 11171 0.0 0.1 116968 8244 ? D 16:42 0:00 svn status www-data 12656 0.5 0.2 34640 17392 ? D 17:04 0:01 /usr/sbin/apache2 -k start root 12717 0.1 0.0 18032 6988 ? D 17:05 0:00 svn status root 13009 0.0 0.0 5368 792 pts/1 S+ 17:09 0:00 grep --color=auto D 17:09:49 up 16 days, 10:34, 7 users, load average: 42.99, 47.23, 48.43 pidstat -d doesn't show a lot of processes doing read/write, usually 4-5, but load is somehow io bound - iowait is near 95-100% iostat showed near 100% disk usage. Looks like this load is caused by one of VEs - while HN has LA 42, it has LA 40. UBC is almost clean, dmesg too. krabrss2:/# grep -v '0$' /proc/user_beancounters Version: 2.5 uid resource held maxheld barrier limit failcnt dgramrcvbuf 0 262096 262144 262144 2983 Now load is decreasing on it's own, but it's very strange to get such situation without any clear reasons. Systeminfo: root@beta:~# cat /proc/version && cat /proc/vz/version Linux version 2.6.18-194.26.1.el5.028stab079.2-nekotaz2-kernel05gcc4.1 (root@nekotaz2.coolcold.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Thu Dec 23 12:52:35 MSK 2010 028stab079.2
coolcold wrote on Sun, 27 February 2011 14:19 root@beta:~# vmstat 1 10 && echo "" && ps aux|grep ' D ' && echo && uptime That grep has and extra space. It doesn't catch states like D+ or Ds. My guess is that one of your HDs was retrying a failed read, and stuck everything else. smartctl -l /dev/sdX Regards,