java.lang.OutOfMemoryError: getNewTla

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

java.lang.OutOfMemoryError: getNewTla

DUKR
Hi P-H

In our production env, we are seeing out of memory exception in every 10 days, we enabled GC logs and monitoring gc behaviour and also enabled heapdump on out of memory option in weblogic startup parameters and re-started weblogic servers on 02/16 so when it crashes it will write heapdump.
I observed GC happening but the heap in filling up. Here i have attached GC logs.
we are using Jrokit - R28.2.9-13-157938-1.6.0_65-20130920-0911-windows-x86_64
As we are starting weblogic server using node manager jrcmd is not allowing me to take heapdump, so litrelly we are waiting production to crash.
Is there any other way to troubleshoot the issue here.

I took flight recording on prod for 15 min and took thread dumps, In the monitoring also i can see heap growing but not sure is that memory leak or gc is not working.

In startup parameters info:

-Xrs -Xms4092m -Xmx4092m -Xmanagement:ssl=false,authenticate=false,port=9103,autodiscovery=true -Xverbose:memory -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xverboselog:D:\GCLogs\DS\gc_UI2.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\GCLogs\HeapDump\DS2.log

I can upload the gc logs and filight recording. Can you please help me to troubleshoot the issue. Thanks in advance

Regards,
DUKR
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: java.lang.OutOfMemoryError: getNewTla

DUKR
[2016-02-06T06:19:55.186-06:00] [SSP-BS1] [NOTIFICATION] [DFW-40101] [oracle.dfw.incident] [tid: Timer-6] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30191] [APP: MARAPPDashboard_MARAPP#V9.0.1] An incident has been signalled with the incident facts: [problemKey=DFW-99997 [java.lang.OutOfMemoryError] incidentSource=SYSTEM incidentTime=Sat Feb 06 06:18:57 CST 2016 errorMessage=DFW-99997 executionContextId=null]
[2016-02-06T06:19:55.186-06:00] [SSP-BS1] [NOTIFICATION] [DFW-40101] [oracle.dfw.incident] [tid: Timer-7] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30190] [APP: webcenter#11.1.1.4.0] An incident has been signalled with the incident facts: [problemKey=DFW-99997 [java.lang.OutOfMemoryError] incidentSource=SYSTEM incidentTime=Sat Feb 06 06:19:55 CST 2016 errorMessage=DFW-99997 executionContextId=null]
[2016-02-06T06:19:55.186-06:00] [SSP-BS1] [WARNING] [DFW-40126] [oracle.dfw.incident] [tid: Timer-6] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30191] [APP: MARAPPDashboard_MARAPP#V9.0.1] incident with Problem Key "DFW-99997 [java.lang.OutOfMemoryError]" could not be created due to lack of available disk space for incidents; configured maximum is 524,288,000 bytes and used is 532,675,769 bytes
[2016-02-06T06:19:55.186-06:00] [SSP-BS1] [NOTIFICATION] [J2EE JSP-00008] [oracle.j2ee.jsp] [tid: [ACTIVE].ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: kprob106] [ecid: 00iYQwrS2iqDsXX5PvL6id0000rO003CSS,0:5] [APP: MARAPPDashboard_MARAPP#V9.0.1] [URI: /MARAPP9/MARAPP/dashboard/faces/adf.task-flow] unable to dispatch JSP page: The following exception occurred:.[[
java.lang.OutOfMemoryError

]]
[2016-02-06T06:20:13.780-06:00] [SSP-BS1] [NOTIFICATION] [DFW-40101] [oracle.dfw.incident] [tid: JspTimeoutThread] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30192] [APP: webcenter#11.1.1.4.0] An incident has been signalled with the incident facts: [problemKey=DFW-99997 [java.lang.OutOfMemoryError] incidentSource=SYSTEM incidentTime=Sat Feb 06 06:20:13 CST 2016 errorMessage=DFW-99997 executionContextId=null]
[2016-02-06T06:20:13.780-06:00] [SSP-BS1] [WARNING] [DFW-40126] [oracle.dfw.incident] [tid: JspTimeoutThread] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30192] [APP: webcenter#11.1.1.4.0] incident with Problem Key "DFW-99997 [java.lang.OutOfMemoryError]" could not be created due to lack of available disk space for incidents; configured maximum is 524,288,000 bytes and used is 532,675,769 bytes
[2016-02-06T06:18:57.855-06:00] [SSP-BS1] [NOTIFICATION] [] [oracle.dms.collector] [tid: DmsThread-1] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30181] [APP: DMS Application#11.1.1.1.0] Caught Exception.[[
java.lang.OutOfMemoryError: getNewTla
        at java.util.HashMap.newKeyIterator(HashMap.java:1024)
        at java.util.HashMap$KeySet.iterator(HashMap.java:1062)
        at java.util.HashMap.buildCache(HashMap.java:588)
        at java.util.HashMap.resize(HashMap.java:576)
        at java.util.HashMap.addEntry(HashMap.java:939)
        at java.util.HashMap.put(HashMap.java:477)
        at java.util.HashSet.add(HashSet.java:200)
        at java.util.AbstractCollection.addAll(AbstractCollection.java:305)
        at com.sun.jmx.mbeanserver.Repository.query(Repository.java:539)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNamesImpl(DefaultMBeanServerInterceptor.java:568)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:558)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:589)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$9.run(WLSMBeanServerInterceptorBase.java:273)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.queryNames(WLSMBeanServerInterceptorBase.java:271)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$9.run(WLSMBeanServerInterceptorBase.java:273)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.queryNames(WLSMBeanServerInterceptorBase.java:271)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServer.queryNames(WLSMBeanServer.java:247)
        at oracle.as.jmx.framework.generic.BaseMBeanServerInterceptor.queryNames(BaseMBeanServerInterceptor.java:168)
        at oracle.dms.impl.producer.MetricConvertor.run(MetricConvertor.java:988)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at oracle.dms.aggregator.DmsThreadFactory$1$1.run(DmsThreadFactory.java:56)
        at oracle.dms.aggregator.DmsThreadFactory$1.run(DmsThreadFactory.java:51)
        at java.lang.Thread.run(Thread.java:680)

]]
[2016-02-06T06:20:32.250-06:00] [SSP-BS1] [NOTIFICATION] [DFW-40101] [oracle.dfw.incident] [tid: Timer-5] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30193] [APP: MARAPPDashboard_MARAPP#V9.0.1] An incident has been signalled with the incident facts: [problemKey=DFW-99997 [java.lang.OutOfMemoryError] incidentSource=SYSTEM incidentTime=Sat Feb 06 06:20:32 CST 2016 errorMessage=DFW-99997 executionContextId=null]
[2016-02-06T06:20:32.250-06:00] [SSP-BS1] [WARNING] [DFW-40126] [oracle.dfw.incident] [tid: Timer-5] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30193] [APP: MARAPPDashboard_MARAPP#V9.0.1] incident with Problem Key "DFW-99997 [java.lang.OutOfMemoryError]" could not be created due to lack of available disk space for incidents; configured maximum is 524,288,000 bytes and used is 532,675,769 bytes
[2016-02-06T06:20:41.500-06:00] [SSP-BS1] [ERROR] [] [micros.MARAPP] [tid: [ACTIVE].ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: avega305] [ecid: 00iYQwrTUIGDsXX5PvL6id0000rO003CSW,0:4] [APP: MARAPPDashboard_MARAPP#V9.0.1] [URI: /MARAPP9/MARAPP/dashboard/faces/MARAPP-dashboard/SinglePortlet]  [null]- Error Message: allocLargeObjectOrArray: [B, size 153736
[2016-02-06T06:20:50.406-06:00] [SSP-BS1] [NOTIFICATION] [DFW-40101] [oracle.dfw.incident] [tid: Timer-8] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30194] [APP: webcenter#11.1.1.4.0] An incident has been signalled with the incident facts: [problemKey=DFW-99997 [java.lang.OutOfMemoryError] incidentSource=SYSTEM incidentTime=Sat Feb 06 06:19:38 CST 2016 errorMessage=DFW-99997 executionContextId=null]
[2016-02-06T06:20:50.406-06:00] [SSP-BS1] [WARNING] [DFW-40126] [oracle.dfw.incident] [tid: Timer-8] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30194] [APP: webcenter#11.1.1.4.0] incident with Problem Key "DFW-99997 [java.lang.OutOfMemoryError]" could not be created due to lack of available disk space for incidents; configured maximum is 524,288,000 bytes and used is 532,675,769 bytes
[2016-02-06T06:19:20.872-06:00] [SSP-BS1] [WARNING] [WCS-40025] [oracle.portlet.client.service.pipeline.TimeoutMonitor] [tid: TimeoutMonitorTaskCleanUpTimer] [ecid: 0000L9zWugVDkZX5Pvx0id1Mdwz6000001,1:30189] [APP: webcenter#11.1.1.4.0] Muffling exception in method TimeoutMonitor$1.run, context = Throwable: java.lang.OutOfMemoryError.
[2016-02-06T06:22:04.644-06:00] [SSP-BS1] [WARNING] [] [oracle.adfinternal.view.faces.lifecycle.LifecycleImpl] [tid: [ACTIVE].ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: avega305] [ecid: 00iYQwrTUIGDsXX5PvL6id0000rO003CSW,0:4] [APP: MARAPPDashboard_MARAPP#V9.0.1] [URI: /MARAPP9/MARAPP/dashboard/faces/MARAPP-dashboard/SinglePortlet] ADF_FACES-60098:Faces lifecycle receives unhandled exceptions in phase RESTORE_VIEW 1[[
java.lang.OutOfMemoryError: allocLargeObjectOrArray: [C, size 11832
        at java.util.Arrays.copyOfRange(Arrays.java:3209)
        at java.lang.String.<init>(String.java:215)
        at java.lang.StringBuffer.toString(StringBuffer.java:585)
        at micros.MARAPP.common.generic.logger.MARAPPLogFormatter.format(MARAPPLogFormatter.java:56)
        at java.util.logging.StreamHandler.publish(StreamHandler.java:177)
        at java.util.logging.FileHandler.publish(FileHandler.java:553)
        at java.util.logging.Logger.log(Logger.java:509)
        at micros.MARAPP.common.generic.logger.MARAPPLogger.doLog(MARAPPLogger.java:149)
        at micros.MARAPP.common.generic.logger.MARAPPLogger.log(MARAPPLogger.java:118)
        at micros.MARAPP.common.generic.logger.MARAPPLogger.warning(MARAPPLogger.java:275)
        at micros.MARAPP.ui.view.fw.CustomErrorHandlerHelper._logError(CustomErrorHandlerHelper.java:107)
        at micros.MARAPP.ui.view.fw.CustomErrorHandlerHelper.logFullStack(CustomErrorHandlerHelper.java:31)
        at micros.MARAPP.ui.view.fw.MARAPPADFcCutomExceptionHandler.handleException(MARAPPADFcCutomExceptionHandler.java:35)
        at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._handleException(LifecycleImpl.java:1310)
        at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.portlet.client.adapter.adf.ADFPortletFilter.doFilter(ADFPortletFilter.java:32)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.webcenter.framework.events.dispatcher.EventDispatcherFilter.doFilter(EventDispatcherFilter.java:44)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:205)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:106)
        at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
        at oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:60)
        at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
        at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:271)
        at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:177)
        at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at micros.MARAPP.ui.view.tmpl.lifecycle.IECompatibilityFilter.doFilter(IECompatibilityFilter.java:57)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:291)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.wcps.client.PersonalizationFilter.doFilter(PersonalizationFilter.java:75)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.webcenter.content.integration.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:168)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.webcenter.lifecycle.filter.LifecycleLockFilter.doFilter(LifecycleLockFilter.java:151)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:179)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.adf.view.page.editor.webapp.WebCenterComposerFilter.doFilter(WebCenterComposerFilter.java:117)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.adfinternal.view.faces.caching.filter.AdfFacesCachingFilter.doFilter(AdfFacesCachingFilter.java:137)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:315)
        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:442)
        at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
        at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
        at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:139)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Loading...