EMA :3: Finding Connection/Application Module Leak.

Finding a connection leak is pretty easy using EMA for Fusion Apps. All you need to do is find the AM that is holding/retaining the connection object. Then we can check why the AM is not releasing the connection. Once I came around a scenario where SCM Logistics Server reported that its JDBC data source was overloaded.

There was a connection leak happening here as shown below:

When I opened the heap dump for this container, figured out that oracle.jbo.server.DBTransactionImpl2 was holding up a lot of memory. When I checked its Immediate Dominator, figured out the AM name which was dominating (or referring) most of the transaction objects. The following image shows how to find the dominating objects of a Transaction:

In my case, ApplSessionPageRootAMImpl was holding most of the transactions (and transaction objects manages the connections for Fusion Apps), as shown in the image below:

Later on figured out that there is a bug in the ADF technology stack where while releasing the AM, an exception is being thrown with the following stack trace:

Exception occurred during page root am release.
java.lang.NullPointerException
        at oracle.jbo.client.Configuration.releaseRootApplicationModule(Configuration.java:1435)
        at oracle.apps.fnd.applcore.common.ApplSessionFilter.doFilter(ApplSessionFilter.java:614)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:279)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.wcps.client.PersonalizationFilter.doFilter(PersonalizationFilter.java:75)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.webcenter.content.integration.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:168)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.bpel.services.workflow.client.worklist.util.WorkflowFilter.doFilter(WorkflowFilter.java:174)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:175)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:114)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
        at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:97)
        at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:164)
        at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:114)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
        at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:97)
        at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:164)
        at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

Note: The same issue can happen if the developer forgets to add the Configuration.releaseApplicationModule() after calling Configuration.createRootApplicationModule().

You can also use ‘Open Dominator Tree’ link to figure out connection leak if the leak is very big and is visible in the ‘Leak Suspect Report’ as shown in the post 'EMA : The Leak Suspects Report', as shown below:

In this case, the next step was to log a bug with the exception stack trace to chase the appropriate Development team and telling the name of the AM to figure out what it does, how its used and why its not getting released. The bug has been fixed just recently.