If I add up my memory usage in each VE, it adds up to a lot less than what is (apparently) actually being consumed.
This is causing me grief on a production server, in that I'm running out of RAM, and going into swap, which is really hurting my web server - it slows to a crawl, then processes back up, and memory usage really spirals up.
Evidence:
# for v in ` cd /proc/bc ; echo * `; do echo $v; vzctl exec $v free -m; done
---- VE: 100
total used free shared buffers cached
Mem: 256 9 246 0 0 0
-/+ buffers/cache: 9 246
Swap: 0 0 0
---- VE: 101
total used free shared buffers cached
Mem: 921 493 427 0 0 0
-/+ buffers/cache: 493 427
Swap: 0 0 0
---- VE: 102
total used free shared buffers cached
Mem: 460 58 402 0 0 0
-/+ buffers/cache: 58 402
Swap: 0 0 0
---- VE: 106
total used free shared buffers cached
Mem: 345 20 324 0 0 0
-/+ buffers/cache: 20 324
Swap: 0 0 0
---- VE: 107
total used free shared buffers cached
Mem: 115 37 77 0 0 0
-/+ buffers/cache: 37 77
Swap: 0 0 0
Add up the "used" columns, and I get around 617MB
On the HN:
# free -m
total used free shared buffers cached
Mem: 1945 1901 43 0 40 100
-/+ buffers/cache: 1761 184
Swap: 6143 281 5862
It is showing 1761MB in use.
If I do some maths on the output of /proc/bc/*, I get similar results:
# ~/beano < /proc/user_beancounters
VE 107: priv = 36MB, kmem = 0MB
VE 106: priv = 20MB, kmem = 1MB
VE 102: priv = 58MB, kmem = 1MB
VE 101: priv = 497MB, kmem = 11MB
VE 100: priv = 9MB, kmem = 0MB
VE 0: priv = 39MB, kmem = 7MB
This totals 679MB (note, this was run a few minutes after the above maths, so numbers differ slightly).
So, why the large discrepancy? Where's my missing 1GB of memory gone?
Caveats: I know I'm probably not doing the maths 100% accurately, but it's near enough to know that something isn't right.
Detail:
Here's /proc/user_beancount:
# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
107: kmemsize 903172 2714806 14372700 14790164 0
lockedpages 0 0 256 256 0
privvmpages 9457 20561 29491 32768 0
shmpages 0 1296 21504 21504 0
dummy 0 0 0 0 0
numproc 9 22 240 240 0
physpages 493 8769 0 9223372036854775807 0
vmguarpages 0 0 33792 9223372036854775807 0
oomguarpages 1165 8957 26112 9223372036854775807 0
numtcpsock 6 12 360 360 0
numflock 1 6 188 206 0
numpty 0 3 16 16 0
numsiginfo 0 2 256 256 0
tcpsndbuf 104640 165944 1720320 2703360 0
tcprcvbuf 98304 401464 1720320 2703360 0
othersockbuf 11560 22328 1126080 2097152 0
dgramrcvbuf 0 34880 262144 262144 0
numothersock 7 16 360 360 0
dcachesize 98595 193317 3409920 3624960 0
numfile 221 471 9312 9312 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 14 14 128 128 0
106: kmemsize 1798197 3523117 14372700 14790164 0
lockedpages 0 0 256 256 0
privvmpages 5189 19476 88473 98304 0
shmpages 18 674 21504 21504 0
dummy 0 0 0 0 0
numproc 8 22 240 240 0
physpages 116 5050 0 9223372036854775807 0
vmguarpages 0 0 29491 32768 0
oomguarpages 1588 6541 26112 9223372036854775807 0
numtcpsock 3 11 360 360 0
numflock 1 2 188 206 0
numpty 0 1 16 16 0
numsiginfo 0 2 256 256 0
tcpsndbuf 52320 338296 1720320 2703360 0
tcprcvbuf 49152 0 1720320 2703360 0
othersockbuf 2312 9808 1126080 2097152 0
dgramrcvbuf 0 8456 262144 262144 0
numothersock 1 9 360 360 0
dcachesize 131460 201016 3409920 3624960 0
numfile 202 430 9312 9312 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 14 14 128 128 0
102: kmemsize 1725889 4340821 14372700 14790164 0
lockedpages 0 0 256 256 0
privvmpages 14717 39041 117964 131072 0
shmpages 0 3216 21504 21504 0
dummy 0 0 0 0 0
numproc 9 26 240 240 0
physpages 9404
...