OpenVZ Forum


Home » General » Support » Where has my memory gone?
Where has my memory gone? [message #32204] Fri, 25 July 2008 14:49 Go to next message
gnutered is currently offline  gnutered
Messages: 12
Registered: April 2008
Junior Member
If I add up my memory usage in each VE, it adds up to a lot less than what is (apparently) actually being consumed.

This is causing me grief on a production server, in that I'm running out of RAM, and going into swap, which is really hurting my web server - it slows to a crawl, then processes back up, and memory usage really spirals up.

Evidence:

# for v in ` cd /proc/bc ; echo * `; do echo $v; vzctl exec $v free -m; done
---- VE: 100
             total       used       free     shared    buffers     cached
Mem:           256          9        246          0          0          0
-/+ buffers/cache:          9        246
Swap:            0          0          0
---- VE: 101
             total       used       free     shared    buffers     cached
Mem:           921        493        427          0          0          0
-/+ buffers/cache:        493        427
Swap:            0          0          0
---- VE: 102
             total       used       free     shared    buffers     cached
Mem:           460         58        402          0          0          0
-/+ buffers/cache:         58        402
Swap:            0          0          0
---- VE: 106
             total       used       free     shared    buffers     cached
Mem:           345         20        324          0          0          0
-/+ buffers/cache:         20        324
Swap:            0          0          0
---- VE: 107
             total       used       free     shared    buffers     cached
Mem:           115         37         77          0          0          0
-/+ buffers/cache:         37         77
Swap:            0          0          0

Add up the "used" columns, and I get around 617MB

On the HN:
# free -m
             total       used       free     shared    buffers     cached
Mem:          1945       1901         43          0         40        100
-/+ buffers/cache:       1761        184
Swap:         6143        281       5862


It is showing 1761MB in use.

If I do some maths on the output of /proc/bc/*, I get similar results:
# ~/beano < /proc/user_beancounters
VE 107: priv = 36MB, kmem = 0MB
VE 106: priv = 20MB, kmem = 1MB
VE 102: priv = 58MB, kmem = 1MB
VE 101: priv = 497MB, kmem = 11MB
VE 100: priv = 9MB, kmem = 0MB
VE 0: priv = 39MB, kmem = 7MB


This totals 679MB (note, this was run a few minutes after the above maths, so numbers differ slightly).

So, why the large discrepancy? Where's my missing 1GB of memory gone?

Caveats: I know I'm probably not doing the maths 100% accurately, but it's near enough to know that something isn't right.

Detail:

Here's /proc/user_beancount:

# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
      107:  kmemsize                   903172              2714806             14372700             14790164                    0
            lockedpages                     0                    0                  256                  256                    0
            privvmpages                  9457                20561                29491                32768                    0
            shmpages                        0                 1296                21504                21504                    0
            dummy                           0                    0                    0                    0                    0
            numproc                         9                   22                  240                  240                    0
            physpages                     493                 8769                    0  9223372036854775807                    0
            vmguarpages                     0                    0                33792  9223372036854775807                    0
            oomguarpages                 1165                 8957                26112  9223372036854775807                    0
            numtcpsock                      6                   12                  360                  360                    0
            numflock                        1                    6                  188                  206                    0
            numpty                          0                    3                   16                   16                    0
            numsiginfo                      0                    2                  256                  256                    0
            tcpsndbuf                  104640               165944              1720320              2703360                    0
            tcprcvbuf                   98304               401464              1720320              2703360                    0
            othersockbuf                11560                22328              1126080              2097152                    0
            dgramrcvbuf                     0                34880               262144               262144                    0
            numothersock                    7                   16                  360                  360                    0
            dcachesize                  98595               193317              3409920              3624960                    0
            numfile                       221                  471                 9312                 9312                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            numiptent                      14                   14                  128                  128                    0
      106:  kmemsize                  1798197              3523117             14372700             14790164                    0
            lockedpages                     0                    0                  256                  256                    0
            privvmpages                  5189                19476                88473                98304                    0
            shmpages                       18                  674                21504                21504                    0
            dummy                           0                    0                    0                    0                    0
            numproc                         8                   22                  240                  240                    0
            physpages                     116                 5050                    0  9223372036854775807                    0
            vmguarpages                     0                    0                29491                32768                    0
            oomguarpages                 1588                 6541                26112  9223372036854775807                    0
            numtcpsock                      3                   11                  360                  360                    0
            numflock                        1                    2                  188                  206                    0
            numpty                          0                    1                   16                   16                    0
            numsiginfo                      0                    2                  256                  256                    0
            tcpsndbuf                   52320               338296              1720320              2703360                    0
            tcprcvbuf                   49152                    0              1720320              2703360                    0
            othersockbuf                 2312                 9808              1126080              2097152                    0
            dgramrcvbuf                     0                 8456               262144               262144                    0
            numothersock                    1                    9                  360                  360                    0
            dcachesize                 131460               201016              3409920              3624960                    0
            numfile                       202                  430                 9312                 9312                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            numiptent                      14                   14                  128                  128                    0
      102:  kmemsize                  1725889              4340821             14372700             14790164                    0
            lockedpages                     0                    0                  256                  256                    0
            privvmpages                 14717                39041               117964               131072                    0
            shmpages                        0                 3216                21504                21504                    0
            dummy                           0                    0                    0                    0                    0
            numproc                         9                   26                  240                  240                    0
            physpages                    9404        
...

Re: Where has my memory gone? [message #32213 is a reply to message #32204] Sat, 26 July 2008 16:23 Go to previous messageGo to next message
kenjy is currently offline  kenjy
Messages: 50
Registered: July 2008
Location: Mexico
Member
Please try using vzmemcheck and vzmemcheck -vA in HN, take care of VMGUARPAGES, OOMGUARPAGES and PRIVVMPAGES

http://kenjy.net |Live net !!!
Re: Where has my memory gone? [message #32228 is a reply to message #32213] Sun, 27 July 2008 12:19 Go to previous messageGo to next message
gnutered is currently offline  gnutered
Messages: 12
Registered: April 2008
Junior Member
Thanks.

My output is thus:
$ sudo cat /proc/bc/101/resources && sudo vzmemcheck -v
            kmemsize                 12755129             46486929            134217728            147639500                    0
            lockedpages                     0                    0                  256                  256                    0
            privvmpages                133586               263359               235929               262144               208779
            shmpages                     9152                15568                21504                21504                    0
            numproc                        71                  159                  400                  400                    0
            physpages                   36617                85441                    0  9223372036854775807                    0
            vmguarpages                     0                    0                58982                65536                    0
            oomguarpages                92833               151046                58982                65536                    0
            numtcpsock                     34                  173                  360                  360                    0
            numflock                        9                   21                  188                  206                    0
            numpty                          0                    2                   16                   16                    0
            numsiginfo                      0                   78                  256                  256                    0
            tcpsndbuf                  378856              1724304              1720320              2703360                  113
            tcprcvbuf                  214840               710136              1720320              2703360                    0
            othersockbuf                27744               179904              1126080              2097152                    0
            dgramrcvbuf                     0                25960               262144               262144                    0
            numothersock                   29                   93                  360                  360                    0
            dcachesize                 458232               620679              3409920              3624960                    0
            numfile                      1688                 2534                 9312                 9312                    0
            numiptent                      14                   14                  128                  128                    0
Output values in %
veid        LowMem  LowMem     RAM MemSwap MemSwap   Alloc   Alloc   Alloc
              util  commit    util    util  commit    util  commit   limit
107           0.14    2.76    0.14    0.07    1.53    0.47    1.90    1.85
106           0.23    2.76    0.11    0.09    1.53    0.27    1.69    5.01
102           0.31    2.76    2.10    1.05    1.53    0.76    1.69    6.59
101           1.64   19.04    8.01    4.64    4.68    6.61    4.68   14.49
100           0.11    2.76    0.06    0.10    1.53    0.12    0.98    3.43
-------------------------------------------------------------------------
Summary:      2.42   30.10   10.42    5.95   10.79    8.23   10.94   31.38

I use this page to interpret these results: http://wiki.openvz.org/UBC_systemwide_configuration

Ratios on 101 (my big server) look dire. Ignore lowmem, as this is an AMD64.

RAM on VE 101 is 8.01. If I add up the numbers physpages*4096 + kmemsize + tcpsndbuf + tcprcvbuf + dgramrcvbuf + othersockbuf, I get 163359801, which is 155.8MB. For the ratio to be 8.01, the system thinks (or treats) my server like it has 19.45MB of RAM. Maths on VE 106 (arbitrarily chosen) shows similar.

Something isn't right.

I mention again that I'm running the FZA kernel (from the debian-systs repository), which I chose for reasons that no longer apply. I can go back to the slightly more official kernel if something is broken inside the kernel, but I'm keen to avoid reboots, as this is a production server on the other side of the world.

Other than that, what can I try?

Re: Where has my memory gone? [message #32232 is a reply to message #32228] Mon, 28 July 2008 05:23 Go to previous message
kenjy is currently offline  kenjy
Messages: 50
Registered: July 2008
Location: Mexico
Member
Well I can see that you have a big problem with privvmpages and a small one with tcpsndbuf, you need to increase this values in your VE 101, please use vzcfgvalidate /PATH/VE.conf to check the config, don't think in this way "I have 10gb of ram and this VE is going to have 2gb of ram ...", you can't set a specific amount of RAM for a VE because the RAM for a computer is not a piece of something.

http://kenjy.net |Live net !!!
Previous Topic: Dell PowerEdge T105 optical drive
Next Topic: Able to install any OS Template except one
Goto Forum:
  


Current Time: Fri Aug 30 09:26:50 GMT 2024

Total time taken to generate the page: 0.04536 seconds