Home » Mailing lists » Users » Hornet Queue (jboss) vs sever resource shortage
| Hornet Queue (jboss) vs sever resource shortage [message #45610] |
Thu, 22 March 2012 12:08  |
Dariush Pietrzak
Messages: 40
Registered: November 2007
|
Member |
|
|
Hello,
we have encountered a very strange behaviour of HornetQueue, running it
inside vz results in:
bash: fork: Cannot allocate memory
or
xmalloc: cannot allocate 4008 bytes (2084864 bytes allocated)
( this is on host node, that has ~20G of free, unused RAM, inside
containers the problem is not visible - hornet is limited to ~4G of ram,
container is limited to ~16G of ram and 0 vSwap, rest is unlimited )
UBC inside container shows no visible resource usage problem, I'd go as
far as to call it frugal - 1600 filedescriptors, very low shm usage,
physpages nowhere near limit, and yet, when daemon runs, HN is unable to
even accept login from console.
how can I track down what is causing this?
regards, Eyck
--
Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9
Total Existance Failure
|
|
|
|
| Re: Hornet Queue (jboss) vs sever resource shortage [message #45611 is a reply to message #45610] |
Thu, 22 March 2012 14:12  |
Andrew Vagin
Messages: 28
Registered: November 2011
|
Junior Member |
|
|
Which kernel do you use?
Could you show content of /proc/bc/CTID/resources?
On 03/22/2012 04:08 PM, Dariush Pietrzak wrote:
> Hello,
> we have encountered a very strange behaviour of HornetQueue, running it
> inside vz results in:
>
> bash: fork: Cannot allocate memory
> or
> xmalloc: cannot allocate 4008 bytes (2084864 bytes allocated)
>
> ( this is on host node, that has ~20G of free, unused RAM, inside
> containers the problem is not visible - hornet is limited to ~4G of ram,
> container is limited to ~16G of ram and 0 vSwap, rest is unlimited )
>
> UBC inside container shows no visible resource usage problem, I'd go as
> far as to call it frugal - 1600 filedescriptors, very low shm usage,
> physpages nowhere near limit, and yet, when daemon runs, HN is unable to
> even accept login from console.
>
> how can I track down what is causing this?
>
> regards, Eyck
|
|
|
|
| Re: Hornet Queue (jboss) vs sever resource shortage [message #45612 is a reply to message #45610] |
Thu, 22 March 2012 14:19 |
Stoyan Stoyanov
Messages: 11
Registered: November 2011
|
Junior Member |
|
|
If I understand you don't have ANy failcnts in /proc/ ubc ? is that
right?
Im using jboss (can say with 'middle load') inside vms and the
problems (cannot allocate memory etc.) was beautifully printed in
failcnt on /proc/user_bea...
On Mar 22, 2012, at 2:08 PM, Dariush Pietrzak wrote:
> Hello,
> we have encountered a very strange behaviour of HornetQueue, running
> it
> inside vz results in:
>
> bash: fork: Cannot allocate memory
> or
> xmalloc: cannot allocate 4008 bytes (2084864 bytes allocated)
>
> ( this is on host node, that has ~20G of free, unused RAM, inside
> containers the problem is not visible - hornet is limited to ~4G of
> ram,
> container is limited to ~16G of ram and 0 vSwap, rest is unlimited )
>
> UBC inside container shows no visible resource usage problem, I'd go
> as
> far as to call it frugal - 1600 filedescriptors, very low shm usage,
> physpages nowhere near limit, and yet, when daemon runs, HN is
> unable to
> even accept login from console.
>
> how can I track down what is causing this?
>
> regards, Eyck
> --
> Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9
> Total Existance Failure
Stoyan Stoyanov
Core System Administrator
CONFIDENTIAL
The information contained in this email and any attachment is
confidential. It is intended only for the named addressee(s). If you
are not the named addressee(s) please notify the sender immediately
and do not disclose, copy or distribute the contents to any other
person other than the intended addressee(s).
|
|
|
|
| Re: Hornet Queue (jboss) vs sever resource shortage [message #45613 is a reply to message #45611] |
Thu, 22 March 2012 14:28 |
Dariush Pietrzak
Messages: 40
Registered: November 2007
|
Member |
|
|
On Thu, 22 Mar 2012, Andrew Vagin wrote:
> Which kernel do you use?
042stab049.6
> Could you show content of /proc/bc/CTID/resources?
Well, yes and no, I can't do anything on HN,
but I can show you cat /proc/user_beancounters:
(this is from the guest that caused all that resource shortage)
Version: 2.5
uid resource held maxheld barrier limit failcnt
178032: kmemsize 34481819 36954112 17179869184 19327352832 0
lockedpages 0 0 9223372036854775807 9223372036854775807 0
privvmpages 3360282 5017649 4194304 4718592 0
shmpages 5451 5467 9223372036854775807 9223372036854775807 0
dummy 0 0 0 0 0
numproc 505 693 9223372036854775807 9223372036854775807 0
physpages 766009 942952 0 3932160 0
vmguarpages 0 0 9223372036854775807 9223372036854775807 0
oomguarpages 620184 772457 9223372036854775807 9223372036854775807 0
numtcpsock 120 214 9223372036854775807 9223372036854775807 0
numflock 10 13 9223372036854775807 9223372036854775807 0
numpty 12 12 9223372036854775807 9223372036854775807 0
numsiginfo 0 45 9223372036854775807 9223372036854775807 0
tcpsndbuf 2498464 3850016 17179869184 19327352832 0
tcprcvbuf 2499360 6331520 17179869184 19327352832 0
othersockbuf 9344 35296 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 8768 9223372036854775807 9223372036854775807 0
numothersock 72 76 9223372036854775807 9223372036854775807 0
dcachesize 10880246 11060512 17179869184 19327352832 0
numfile 1488 1660 9223372036854775807 9223372036854775807 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 10 10 9223372036854775807 9223372036854775807 0
(as you can see I tried limiting some of the resources, when it
was first noticed, everything was unlimited:unlimited
except physpages which is at 16G, and swap which was set to 0.)
When the hornetq is stopped inside container, HN returns back to life,
but untill then, even things like bash completion fails with:
xmalloc: ../bash/make_cmd.c:100: cannot allocate 519 bytes (2076672 bytes allocated)
regards, Eyck
--
Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9
Total Existance Failure
|
|
|
|
| Re: Hornet Queue (jboss) vs sever resource shortage [message #45614 is a reply to message #45613] |
Thu, 22 March 2012 16:12 |
Dariush Pietrzak
Messages: 40
Registered: November 2007
|
Member |
|
|
> xmalloc: ../bash/make_cmd.c:100: cannot allocate 519 bytes (2076672 bytes allocated)
And here's excerpt from top ( running on HN ):
Mem: 32942724k total, 5930724k used, 27012000k free, 528k buffers
Swap: 8388600k total, 0k used, 8388600k free, 1864140k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
76750 nn12 20 0 4621m 1.2g 10m S 1 3.7 2:40.11 java -Ddata.dir=/var/spool/ss/queue/hot/server0-data -Djnp.port=1099 -Djnp.rmiPort=1098 -Djnp.host=myhost.cc -Dhornetq.remoting.netty.host=myhost.cc -Dhornetq.remoting.netty.port=5445 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=6000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticat
76876 nn12 20 0 4632m 1.0g 10m S 1 3.3 2:15.22 java -Ddata.dir=/var/spool/ss/queue/hot/server1-data -Djnp.port=1199 -Djnp.rmiPort=1198 -Djnp.host=myhost.cc -Dhornetq.remoting.netty.hos
to verify that all that memory really is available, i ran:
dd if=/dev/zero of=/dev/null bs=1G
( and then 2G, 4G )
and as can be seen here, it works:
Mem: 32942724k total, 8034640k used, 24908084k free, 528k buffers
Swap: 8388600k total, 0k used, 8388600k free, 1864276k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
118477 root 20 0 4107m 2.0g 504 R 100 6.4 0:10.60 dd if=/dev/zero of=/dev/null bs=4G
76750 nn12 20 0 4621m 1.2g 10m S 1 3.7 2:41.35 java -Ddata.dir=/var/spool
but the HN can't allocate anything. I'm baffled. What kind of memory shortage can cause
such symptoms?
regards, Eyck
--
Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9
Total Existance Failure
|
|
|
|
|
|
| Re: Hornet Queue (jboss) vs sever resource shortage [message #45628 is a reply to message #45624] |
Fri, 23 March 2012 20:31 |
|
|
On 03/23/2012 05:49 PM, Andrew Vagin wrote:
> On 03/22/2012 06:28 PM, Dariush Pietrzak wrote:
>> (as you can see I tried limiting some of the resources, when it was
>> first noticed, everything was unlimited:unlimited except physpages
>> which is at 16G, and swap which was set to 0.) When the hornetq is
>> stopped inside container, HN returns back to life, but untill then,
>> even things like bash completion fails with: xmalloc:
>> ../bash/make_cmd.c:100: cannot allocate 519 bytes (2076672 bytes
>> allocated) regards, Eyck
> Could you file a bug in https://bugzilla.openvz.org/ ?
I'd suggest trying to reproduce the case on the latest rhel6-testing
kernel (042stab053.3) and then filing a bug.
Kir Kolyshkin
|
|
|
|
| Re: Hornet Queue (jboss) vs sever resource shortage [message #45726 is a reply to message #45628] |
Fri, 30 March 2012 17:59 |
Dariush Pietrzak
Messages: 40
Registered: November 2007
|
Member |
|
|
> >>which is at 16G, and swap which was set to 0.) When the hornetq is
> >>stopped inside container, HN returns back to life, but untill then,
> >>even things like bash completion fails with: xmalloc:
> >>../bash/make_cmd.c:100: cannot allocate 519 bytes (2076672 bytes
> >>allocated)
> >Could you file a bug in https://bugzilla.openvz.org/ ?
> I'd suggest trying to reproduce the case on the latest rhel6-testing
> kernel (042stab053.3) and then filing a bug.
Filed a bug, reproduced it on 042stab053.3 and also today on 042stab053.5.
Noticed few more different things - when I set privvmpages to 12G:16G, I
can't start java (hornetq) with -Xmx set to ~7G, and UBC increases failure
count on privvmpages, program fails to start with:
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
(so there is something here, that accounting at least notices).
( when I reduce -Xmx to ~6G, hornetq starts, and then I get 'out of memory'
errors on console/HN ).
Also, it seems that limiting heap with -Xmx to ~512M makes it possible to
run one or two instances with hornetq without ill-effects.
regards, Eyck
--
Key fingerprint = 40D0 9FFB 9939 7320 8294 05E0 BCC7 02C4 75CC 50D9
Total Existance Failure
|
|
|
|
Goto Forum:
Current Time: Tue Jul 16 01:45:20 GMT 2024
Total time taken to generate the page: 0.02606 seconds
|
OpenVZ Forum
Members
Search
Help
Register
Login
Home
