OpenVZ Forum: Support » *SOLVED* Strange memory failcnt problem

Home » General » Support » *SOLVED* Strange memory failcnt problem

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Switch to threaded view of this topic

Create a new topic

Submit Reply

*SOLVED* Strange memory failcnt problem [message #7021]

Sat, 30 September 2006 19:43

gral is currently offline

gral
Messages: 34
Registered: May 2006

Member

Hi ev1

Razz

Im having problem in one of my vps, sometimes received apache errors like 500 or forbidden, and the WHM and cPanel are innacesible at the both time...

doing the cat /proc/user_beancounters i received...

 uid  resource                     held              maxheld              barrier                limit              failcnt
       112: kmemsize                  6748091             49176315  1152921504606846975  1152921504606846975              3895057
            lockedpages                     0                    8  1142921504606846975  1152921504606846975                    0
            privvmpages                 76896              1169982  1142921504606846975  1152921504606846975                    0
            shmpages                    19663               950363  1142921504606846975  1152921504606846975                    0
            dummy                           0                    0                    0                    0                    0
            numproc                        84                  787  1142921504606846975  1152921504606846975

(Sorry for big scroll)...

so Why if the limit of the kmemsize is =1152921504606846975 the maxheld is 49176315 and the memory fails 3895057

What's should i do? and what does it mean exactly?

Regards!!

Gral Very Happy

Very Happy

[ADMIN EDIT: added CODE tag pair to fix UBC formatting]

Argentina
--Vz--

[Updated on: Sat, 14 October 2006 06:29] by Moderator

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7024 is a reply to message #7021]

Sat, 30 September 2006 21:34

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

Quote:

Why if the limit of the kmemsize is =1152921504606846975 the maxheld is 49176315 and the memory fails 3895057

I see the only reason why it happens. Suppose at a moment your held=maxheld=49176315. Then some process tries to use 1152921504557670661.

49176315 + 1152921504557670661 = 1152921504606846976 > 1152921504606846975

Thus allocation fails and maxheld=49176315.

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7026 is a reply to message #7021]

Sun, 01 October 2006 00:05

gral is currently offline

gral
Messages: 34
Registered: May 2006

Member

So there is a process trying to take a lot of memory like 1152921504557670661,

how can i fix it ? or detect the process ?

Thanks again guys!!

Gral Very Happy

Very Happy

Argentina
--Vz--

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7101 is a reply to message #7026]

Wed, 04 October 2006 06:27

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

This is really strange process that wants all this amount of kernel memory Rolling Eyes

Rolling Eyes

Well, I sugest to do everything step by step to detect this "bad" process:
1) Create new clear VPS. Start it. Check user_beancounters, if no failconters go on.
2) Install first application you need, run it, check user_beancounters again.
3) etc...

Doing this way you will be able to detect who's responsible for failcounters and find out limit/barrier parameters that will fit you.

HTH,
vass.

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7102 is a reply to message #7101]

Wed, 04 October 2006 07:47

HaroldB is currently offline

HaroldB
Messages: 61
Registered: June 2006

Member

I too have seen several cases where the kmemsize metric has failcnts but the maxheld is less than the barrier (and less than the limit). I am not sure how we can explore the reason for this? Is there some debug option we can enable perhaps?

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7106 is a reply to message #7102]

Wed, 04 October 2006 08:01

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

Harold, it's normal when kmemsize (and others) metric has failcnts but the maxheld is less than the barrier (and less than the limit).

Look at my answer:
http://forum.openvz.org/index.php?t=tree&th=1261&mid =7024&&rev=&reveal=
We just need to understand who tries to get all this memory. I suppose it can be easily done by hand.

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7107 is a reply to message #7024]

Wed, 04 October 2006 08:03

HaroldB is currently offline

HaroldB
Messages: 61
Registered: June 2006

Member

Hi, yes I read your post but its very unlikely anything is trying to use this amount of memory: 1152921504557670661. Especially in a server where you know all of the applications. I guess its possible and its the best explanation right now though.

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7108 is a reply to message #7107]

Wed, 04 October 2006 08:08

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

I agree, it's really strange...

Gral, Harold is this problem reproducable on your nodes?
If "yes", what do you do to reproduce the problem?
What kernel do you use?
Can you provide an access to the node?

Thanks!

[Updated on: Wed, 04 October 2006 08:09]

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7109 is a reply to message #7106]

Wed, 04 October 2006 08:12

HaroldB is currently offline

HaroldB
Messages: 61
Registered: June 2006

Member

I can't reproduce it right now but I'll keep my eye out. Its just something I have noticed, never posed a big problem because I keep the kmemsize at a big number too, but not nearly 1152921504557670661. Very Happy

Very Happy

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7110 is a reply to message #7109]

Wed, 04 October 2006 08:24

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

Well, what's the kernel, how can I reproduce it locally, can you give access to the node?

Thanks!

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7112 is a reply to message #7110]

Wed, 04 October 2006 08:35

HaroldB is currently offline

HaroldB
Messages: 61
Registered: June 2006

Member

I can't reproduce it. I meant to say, sometimes I check the beans of a VE, and see that kmemsize has failcnts, maxheld<barrier, and the barrier hasn't been modified at all. I'll try to catch one and investigate a method to reproduce..

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7115 is a reply to message #7112]

Wed, 04 October 2006 08:42

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

Thanks!
After that, please, fill the bug!

Goof luck,
vass.

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7155 is a reply to message #7115]

Thu, 05 October 2006 14:52

gral is currently offline

gral
Messages: 34
Registered: May 2006

Member

Hi in my case:

This is really strange process that wants all this amount of kernel memory

There was a domain with joomla hacked, and trougt for it they are attacking another server...

Everything bad...

Well... but is a explanation

Cheers!!!

pd: sorry for my english!

gRal Very Happy

Very Happy

Argentina
--Vz--

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7178 is a reply to message #7021]

Fri, 06 October 2006 07:56

disaster is currently offline

disaster
Messages: 158
Registered: February 2006

Senior Member

Hi!

I've the same problem since => patch 018.1

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7195 is a reply to message #7178]

Fri, 06 October 2006 10:48

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

Seems that there is a problem in kmemsize, since 3 persons have complained on it. Please, fill the bug with as much possible information about environment.

Thanks!

P.S. Also you can try patch
http://git.openvz.org/?p=linux-2.6-openvz;a=commit;h=4165f1fa371b60cb3a225910faebde95650a1f65
I doubt, it can help, but however...

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7197 is a reply to message #7021]

Fri, 06 October 2006 10:58

disaster is currently offline

disaster
Messages: 158
Registered: February 2006

Senior Member

I've already used this patch...

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7198 is a reply to message #7197]

Fri, 06 October 2006 10:59

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

Good, so this is other bug, will you fill it, please? Smile

Smile

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7199 is a reply to message #7021]

Fri, 06 October 2006 11:03

disaster is currently offline

disaster
Messages: 158
Registered: February 2006

Senior Member

Sorry but i don't know what i should post... i can write down there must be a bug with kmemsize... but no more info...

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7200 is a reply to message #7199]

Fri, 06 October 2006 11:07

Vasily Tarasov is currently offline

Vasily Tarasov
Messages: 1345
Registered: January 2006

Senior Member

* Kernel version and architecture (output of uname -a command on the kernel that caused a problem)
* In case you compiled your kernel yourself — your .config file
* In case you have used some additional kernel patches — a link to those patches.
* Description of how to reproduce, if possible

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7202 is a reply to message #7021]

Fri, 06 October 2006 11:15

disaster is currently offline

disaster
Messages: 158
Registered: February 2006

Senior Member

Look here: http://bugzilla.openvz.org/show_bug.cgi?id=288

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7256 is a reply to message #7021]

Mon, 09 October 2006 09:48

xemul is currently offline

xemul
Messages: 248
Registered: November 2005

Senior Member

From gral's first comment I see it happens on x86_64, right?
To HaroldB and disaster - what arch do you use?

http://static.openvz.org/userbars/openvz-developer.png

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7257 is a reply to message #7021]

Mon, 09 October 2006 09:51

disaster is currently offline

disaster
Messages: 158
Registered: February 2006

Senior Member

use x86_64

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7270 is a reply to message #7256]

Mon, 09 October 2006 13:29

HaroldB is currently offline

HaroldB
Messages: 61
Registered: June 2006

Member

using 32 bit.

H

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7273 is a reply to message #7021]

Mon, 09 October 2006 14:22

gral is currently offline

gral
Messages: 34
Registered: May 2006

Member

hey xemul, yeap my Hardware node is x86_64, but the VE templates are i386, but i say in another post in this thread, one of the VirtualHost was a joomla hacked and it generate this error.

Hope it help!

Bye!.

gRal Very Happy

Very Happy

Argentina
--Vz--

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7282 is a reply to message #7270]

Tue, 10 October 2006 08:41

xemul is currently offline

xemul
Messages: 248
Registered: November 2005

Senior Member

HaroldB: I thought I used x86_64... OK, If your machine is 32bit then could you show the config you used.

http://static.openvz.org/userbars/openvz-developer.png

Report message to a moderator

Send a private message to this user

Re: Strange memory failcnt problem [message #7283 is a reply to message #7021]

Tue, 10 October 2006 08:44

disaster is currently offline

disaster
Messages: 158
Registered: February 2006

Senior Member

for me it also this:
Hardware node is x86_64, but the VE templates are i386,

Report message to a moderator

Send a private message to this user

*SOLVED* Re: Strange memory failcnt problem [message #7408 is a reply to message #7021]

Fri, 13 October 2006 07:03

xemul is currently offline

xemul
Messages: 248
Registered: November 2005

Senior Member

Looks like problem is resolved.
http://bugzilla.openvz.org/show_bug.cgi?id=288

http://static.openvz.org/userbars/openvz-developer.png

[Updated on: Fri, 13 October 2006 07:12] by Moderator

Report message to a moderator

Send a private message to this user

Switch to threaded view of this topic

Create a new topic

Submit Reply

Previous Topic:	PSYBNC problem
Next Topic:	Install problems on dual opteron (64 bits), CentOS 4.4

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

PDF

]

Current Time: Sat May 11 06:32:22 GMT 2024

Total time taken to generate the page: 0.01648 seconds