EMA :6: Finding Activation/Passivation Related Issues

Save point is a wonderful feature if used appropriately; otherwise it can bring down a server in no time. That’s what we faced once on SCM PIM. To triage these type of issues, EMA alone is not enough. Here I used thread dump in addition to heap dumps. When the server was going down, I was having no clue why it’s going down, and that’s when thread dumps helped me out, which were showing explicit save point calls, as shown in the following thread stacks snippet taken from the thread dump:

"[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x4b466c00 nid=0x34ba runnable [0x4d000000]
   java.lang.Thread.State: RUNNABLE
        at oracle.jbo.server.ViewDefImpl.getAttributeDefImpls(ViewDefImpl.java:782)
        at oracle.jbo.server.ViewDefImpl.getAttributeDefs(ViewDefImpl.java:800)
        at oracle.jbo.server.ViewRowImpl.getAttributeNames(ViewRowImpl.java:2783)
        at oracle.apps.scm.productCatalogManagement.versions.protectedModel.view.ChangeVersionViewRowImpl.getAttributeInternal(ChangeVersionViewRowImpl.java:61)
        at oracle.apps.scm.productModel.items.protectedUiModel.view.ItemVORowImpl.getRedLineCurrentPhaseId(ItemVORowImpl.java:26485)
        at oracle.apps.scm.productModel.items.protectedUiModel.view.ItemVORowImpl$AttributesEnum$611.get(ItemVORowImpl.java:7993)
        at oracle.apps.scm.productModel.items.protectedUiModel.view.ItemVORowImpl.getAttrInvokeAccessor(ItemVORowImpl.java:21807)
        at oracle.jbo.server.ViewRowImpl.getAttribute(ViewRowImpl.java:830)
        at oracle.jbo.server.ViewRowImpl.renderAttributeInXMLElement(ViewRowImpl.java:4269)
        at oracle.jbo.server.ViewRowImpl.renderAttributeInXMLElement(ViewRowImpl.java:4298)
        at oracle.jbo.server.ViewObjectImpl.passivateTransients(ViewObjectImpl.java:16954)
        at oracle.jbo.server.ViewObjectImpl.passivateTransients(ViewObjectImpl.java:16890)
        at oracle.jbo.server.ViewObjectImpl.passivateState(ViewObjectImpl.java:16686)
        at oracle.jbo.server.ViewObjectImpl.passivateState(ViewObjectImpl.java:16626)
        at oracle.jbo.server.ViewObjectImpl.doPassivateSettings(ViewObjectImpl.java:18545)
        at oracle.jbo.server.ApplicationModuleImpl.passivateVOs(ApplicationModuleImpl.java:7556)
        at oracle.jbo.server.ApplicationModuleImpl.doPassivateState(ApplicationModuleImpl.java:7367)
        at oracle.jbo.server.ApplicationModuleImpl.doPassivateAMState(ApplicationModuleImpl.java:7358)
        at oracle.jbo.server.Serializer.passivate(Serializer.java:210)
        at oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:290)
        at oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:267)
        at oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:5820)
        - locked <0xad48c908> (a oracle.jbo.JboSyncLock)
        at oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:5681)
        at oracle.jbo.server.ApplicationModuleImpl.passivateStateForUndo(ApplicationModuleImpl.java:8641)
        - locked <0xad48c908> (a oracle.jbo.JboSyncLock)
        at oracle.adf.model.bc4j.DCJboDataControl.createSavepoint(DCJboDataControl.java:3143)
        at oracle.adf.model.dcframe.LocalTransactionHandler.createSavepoint(LocalTransactionHandler.java:73)
        at oracle.adf.model.dcframe.DataControlFrameImpl.createSavepoint(DataControlFrameImpl.java:529)
        at oracle.adfinternal.controller.util.model.DCFrameImpl.createSavepoint(DCFrameImpl.java:28)        at oracle.adfinternal.controller.activity.TaskFlowCallActivityLogic.initializeModel(TaskFlowCallActivityLogic.java:999)
        at oracle.adfinternal.controller.activity.TaskFlowCallActivityLogic.enterTaskFlow(TaskFlowCallActivityLogic.java:614)
        at oracle.adfinternal.controller.activity.TaskFlowCallActivityLogic.invokeLocalTaskFlow(TaskFlowCallActivityLogic.java:336)
        at oracle.adfinternal.controller.activity.TaskFlowCallActivityLogic.execute(TaskFlowCallActivityLogic.java:163)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.executeActivity(ControlFlowEngine.java:989)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.doRouting(ControlFlowEngine.java:878)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.doRouting(ControlFlowEngine.java:777)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.routeFromActivity(ControlFlowEngine.java:551)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.performControlFlow(ControlFlowEngine.java:147)
        at oracle.adfinternal.controller.application.NavigationHandlerImpl.handleAdfcNavigation(NavigationHandlerImpl.java:109)
        at oracle.adfinternal.controller.application.NavigationHandlerImpl.handleNavigation(NavigationHandlerImpl.java:78)
. . .
 at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2202)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

"[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x45cc4c00 nid=0x1388 runnable [0x451fc000]

   java.lang.Thread.State: RUNNABLE
        at oracle.jbo.server.ViewDefImpl.getAttributeDefImpls(ViewDefImpl.java:782)
        at oracle.jbo.server.ViewDefImpl.getAttributeDefs(ViewDefImpl.java:800)
        at oracle.jbo.server.ViewRowImpl.getAttributeNames(ViewRowImpl.java:2783)
        at oracle.apps.scm.productCatalogManagement.versions.protectedModel.view.ChangeVersionViewRowImpl.getAttributeInternal(ChangeVersionViewRowImpl.java:61)
        at oracle.apps.scm.productModel.items.protectedUiModel.view.ItemVORowImpl.getTemplateItemFlag(ItemVORowImpl.java:22215)
        at oracle.apps.scm.productModel.items.protectedUiModel.view.ItemVORowImpl.isAttributeUpdateable(ItemVORowImpl.java:21848)
        at oracle.jbo.server.ViewRowImpl.readAttributeFromXMLElement(ViewRowImpl.java:4313)
        at oracle.jbo.server.ViewObjectImpl.activateTransientAttribute(ViewObjectImpl.java:17613)
        at oracle.jbo.server.ViewObjectImpl.activateTransients(ViewObjectImpl.java:17679)
        at oracle.jbo.server.ViewObjectImpl.activateTransients(ViewObjectImpl.java:17627)
        at oracle.jbo.server.ViewObjectImpl.activateState(ViewObjectImpl.java:17850)
        at oracle.jbo.server.ViewObjectImpl.activateState(ViewObjectImpl.java:17745)
        at oracle.jbo.server.ViewRowSetIteratorImpl.activateIteratorState(ViewRowSetIteratorImpl.java:3989)
        - locked <0xb0378e40> (a oracle.jbo.JboSyncLock)
        at oracle.jbo.server.ViewRowSetImpl.activateIteratorState(ViewRowSetImpl.java:6992)
        at oracle.jbo.server.ViewObjectImpl.activateIteratorState(ViewObjectImpl.java:18080)
        at oracle.jbo.server.ApplicationModuleImpl.activateVOs(ApplicationModuleImpl.java:8021)
        at oracle.jbo.server.ApplicationModuleImpl.doActivateState(ApplicationModuleImpl.java:7767)
        at oracle.jbo.server.ApplicationModuleImpl.doActivateAMState(ApplicationModuleImpl.java:7734)
        at oracle.jbo.server.Serializer.activate(Serializer.java:293)
        at oracle.jbo.server.DBSerializer.activateRootAM(DBSerializer.java:330)
        at oracle.jbo.server.ApplicationModuleImpl.activateFromStack(ApplicationModuleImpl.java:6306)
        at oracle.jbo.server.ApplicationModuleImpl.activateState(ApplicationModuleImpl.java:6166)
        - locked <0xb0378e40> (a oracle.jbo.JboSyncLock)
        at oracle.jbo.server.ApplicationModuleImpl.activateStateForUndo(ApplicationModuleImpl.java:8710)
        - locked <0xb0378e40> (a oracle.jbo.JboSyncLock)
        at oracle.adf.model.bc4j.DCJboDataControl.restoreSavepoint(DCJboDataControl.java:3149)
        at oracle.adf.model.dcframe.LocalTransactionHandler.restoreSavepoint(LocalTransactionHandler.java:108)
        at oracle.adf.model.dcframe.DataControlFrameImpl.restoreSavepoint(DataControlFrameImpl.java:545)
        at oracle.adfinternal.controller.util.model.DCFrameImpl.restoreSavepoint(DCFrameImpl.java:40)        at oracle.adfinternal.controller.activity.TaskFlowReturnActivityLogic.resolveTransaction(TaskFlowReturnActivityLogic.java:559)
        at oracle.adfinternal.controller.activity.TaskFlowReturnActivityLogic.execute(TaskFlowReturnActivityLogic.java:114)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.executeActivity(ControlFlowEngine.java:989)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.doRouting(ControlFlowEngine.java:878)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.doRouting(ControlFlowEngine.java:777)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.routeFromActivity(ControlFlowEngine.java:551)
        at oracle.adfinternal.controller.engine.ControlFlowEngine.performControlFlow(ControlFlowEngine.java:147)
        at oracle.adfinternal.controller.application.NavigationHandlerImpl.handleAdfcNavigation(NavigationHandlerImpl.java:109)
        at oracle.adfinternal.controller.application.NavigationHandlerImpl.handleNavigation(NavigationHandlerImpl.java:78)
        at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:130)
 . . .
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2202)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2108)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1432)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

After checking the thread dump, I looked at the heap dump, and in the dominator tree, a lot of XML related objects were visible. This confirmed that because of inappropriate use of save point activity, a lot of activation/passivation was happening and because of which server was going OOM. After discussing this with Development team, came to know that they do have a save point call in their code but they missed setting the Restore Save Point property of task flow return activity to true.

The following images show how to look for XML objects in case of passivation issues. Before any run, this is how the XML Objects were in the heap, as shown in the following dominator tree:

We had 2 suspected flows, so we ran 5 iterations of the first suspected flow and checked the XML Objects growth again, as shown below:

And then we ran the second suspected flow for 5 iteration, and check the XML Objects growth, as shown below:

These images are for only single user scenario so imagine what can happen in case of multi user scenario with a lot of data is involved. In multi user scenario, the container went OOM and crashed multiple times before we deployed the application with correct value of the Restore Save Point property of task flow return activity.

EMA :5: Exception Logging Causing Memory Pressure

Too many exceptions can also cause your application to perform poorly, as exception object creation is an expensive operation and an exception object takes more memory than a normal object as it stores the complete stack trace. And if you have incidents enabled on your environment or server, then too many exception can cause an adverse affect, as shown in my next example. In this case, one of the server was throwing a lot of exceptions and the log writing mechanism was storing the stack trace in a LogRecord[] array before writing it to the incident dump file. The problem was that the frequency of the occurance of exceptions outperformed the server mechanism of incident creation. The result was that the server went OOM and crashed. LogRecord is reported as retaining around 14% of heap, and the relevant classloader was holding onto 18% of heap as shown in the Leak Suspects Report below:

On digging further, figured out that each LogRecord is having a HashMap, which is having multiple keys stored in HashMap.Entry. And within that, the key SRC_LINES stores the complete stack of 1 or multiple exception stack traces, as shown below:

When I was analyzing the issue, I was not aware what was going on. When I saw the stack trace which showed reference to incident, I became certain that its related to incident dumping. The above figure shows it’s related to incident dumping. The above stack usually ended as shown below:

When I checked the incident directories, figured out that the exceptions with the same stack trace are getting listed in the incident dump files. The next step was to work with the Development team to fix these errors.

EMA :4: Finding Statement Leak

Finding statement leak is also very easy using EMA. Just following the following steps:

1. Select Java Basics -> Group By Value, as shown in the following image:

2. Select Object=oracle.jdbc.driver.OracleSql (or any specific class you are looking for like T4CPreparedStatement) and field=actualSql as shown below:

3. You can see distinct statements with their counts when you click Finish. You can export and run awk/sed commands as well on the output to get the desired format.

EMA :3: Finding Connection/Application Module Leak.

Finding a connection leak is pretty easy using EMA for Fusion Apps. All you need to do is find the AM that is holding/retaining the connection object. Then we can check why the AM is not releasing the connection. Once I came around a scenario where SCM Logistics Server reported that its JDBC data source was overloaded.

There was a connection leak happening here as shown below:

When I opened the heap dump for this container, figured out that oracle.jbo.server.DBTransactionImpl2 was holding up a lot of memory. When I checked its Immediate Dominator, figured out the AM name which was dominating (or referring) most of the transaction objects. The following image shows how to find the dominating objects of a Transaction:

In my case, ApplSessionPageRootAMImpl was holding most of the transactions (and transaction objects manages the connections for Fusion Apps), as shown in the image below:

Later on figured out that there is a bug in the ADF technology stack where while releasing the AM, an exception is being thrown with the following stack trace:

Exception occurred during page root am release.
java.lang.NullPointerException
        at oracle.jbo.client.Configuration.releaseRootApplicationModule(Configuration.java:1435)
        at oracle.apps.fnd.applcore.common.ApplSessionFilter.doFilter(ApplSessionFilter.java:614)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:279)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.wcps.client.PersonalizationFilter.doFilter(PersonalizationFilter.java:75)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.webcenter.content.integration.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:168)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.bpel.services.workflow.client.worklist.util.WorkflowFilter.doFilter(WorkflowFilter.java:174)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:175)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:114)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
        at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:97)
        at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:164)
        at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:114)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
        at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:97)
        at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:164)
        at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

Note: The same issue can happen if the developer forgets to add the Configuration.releaseApplicationModule() after calling Configuration.createRootApplicationModule().

You can also use ‘Open Dominator Tree’ link to figure out connection leak if the leak is very big and is visible in the ‘Leak Suspect Report’ as shown in the post 'EMA : The Leak Suspects Report', as shown below:

In this case, the next step was to log a bug with the exception stack trace to chase the appropriate Development team and telling the name of the AM to figure out what it does, how its used and why its not getting released. The bug has been fixed just recently.

EMA :2: Finding View Object Leaks

Once I faced an issue in FIN Payables (AP) where the managed server was going down because of Out Of Memory (OOM) errors in every 3-4 hours. Initially I was clueless about what’s going on but after taking 3 heap dumps for 3 OOM errors and comparing them, I was able to figure out that there is a VO Leak and which AM is causing this. Later on, after having a meeting with Development team, I was able to figure out which peace of code is causing this. Here are the details of how I found the details:

After getting the heap dumps, I was just looking at the histograms of each of the heap dumps. The following image shows how you can open histogram in EMA for a heap dump:

Note:  These images are taken from a different heap dump than the one where VO leak was identified This is done just for the documentation purpose as I found the VO leak issue couple of years back and don’t have that heap dump with me right now.

The first obvious difference that I found was the instance count of OAViewObjectImpl class. It was alarming and was very high in each heap dump, and then I came up with the following figures:

• First Heap Dump : Total OAViewObjectImpl object count is : 10,323, out of which 1760 are of BookControlPVO.
• Second Heap Dump : Total OAViewObjectImpl object count is 16224, 8774 are of BookControlPVO.
• Third Heap Dump : Total OAViewObjectImpl object count is 19062, 13661 are of BookControlPVO.

So it was obvious that BookControlPVO was getting leaked. Now the question was how and from where. The next thing that I found out was which AM this VO was associated with. For that, you need to check the metadata of a View Object instance. The following object shows how to open View Object instances metadata window:

The ‘With Outgoing References’ link lists all the view object instances. Now each VO instance metadata has a mParent field which holds a reference to the AM the current VO instance is associated with, as shown below:

Once I came to know about that, I ran a simple query in OQL to figure out which AM is referenced the most and from which VO:

SELECT toString(s.mViewDef.mFullName), s.mParent FROM oracle.apps.fnd.applcore.oaext.model.OAViewObjectImpl s

Once you run this query, you can export that output to a .txt or .csv file (to parse it using awk or sed), as shown in the figure below:

After exporting, you can run the following awk and sed commands to find out which VO is being created from which AM:

awk -F"|" '{print "VO="$1" Parent="$2 }' ViewObjectList.txt | sort | uniq -c > ViewObject2.txt    

sed 's/^[^0-9]*//g' ViewObject2.txt | sed 's/\([^ ]\)[ ]*$/\1/g' | sed 's/[ ]/:/' | sed 's/^\([1-9][0-9]:\)/0\1/' | sed 's/^\([1-9]:\)/00\1/' | sort -r | sed 's/$/\n/' > ViewObjects3.txt

sed 's/VO[ ]*Parent/VO Parent/; s/Parent=[ ]*oracle/Parent= oracle/' ViewObjects3.txt > ViewObject4.txt

And the following script can be used to figure out which AM is creating how many VOs:

sed 's/=/ /g' ViewObject4.txt > ViewObject5.txt

awk '{print  $4 ", "$2", "$1}' ViewObject5.txt | sort | uniq -c > ViewObject6.txt

In my case, I was able to come up with the following figures:

510:VO=oracle.apps.financials.assets.sharedSetup.systemControls.publicView.SystemControlPVO Parent= oracle.apps.financials.assets.additions.uiModel.applicationModule.AdditionInterfaceUIAMImpl [id=0xbeabe770]

443:VO=oracle.apps.financials.assets.sharedSetup.systemControls.publicView.SystemControlPVO Parent= oracle.apps.financials.assets.additions.uiModel.applicationModule.AdditionInterfaceUIAMImpl [id=0xb1c08c08]

3638:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xbbc743e0]

3111:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xcdb3e508]

2967:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xa6a80c00]

287:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.additions.uiModel.applicationModule.AdditionInterfaceUIAMImpl [id=0xbeabe770]

228:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.additions.uiModel.applicationModule.AdditionInterfaceUIAMImpl [id=0xb1c08c08]

1992:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xa9af0ff0]

1331:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xa9c1c658]

. . .

The format for the above figures is:

<VO instance count>:VO=<VO Name> Parent=<AM Name>

So in the above case, the initial number for example 3638:, 3111: represents the VO instance count, VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO represents the name of the VO, and Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl represents the AM being referred as parent from the VO.

After getting this data, we had a meeting with the appropriate Development team and after investigating the code, figured out the piece of code that is causing this issue. The problem was that the instances of BookControlPVO VO were created dynamically from a utility method (was exposed to other teams as well) but were not removed after use.

EMA :1: The Leak Suspects Report

This blog is about using Eclipse Memory Analyzer (EMA) to fix Out Of Memory (OOM) errors. You can do runtime profiling as well using JProfiler to figure out if a method is retaining a lot of heap , but EMA is far more useful in scenarios when you are having servers running on 2GB heap and you are frequently facing OOM errors. I was taking care of Fusion System Test environments where we have a separate domain for each Product Family like Financials (FIN), Supply Chain Management (SCM) etc, and there are managed servers for each sub family having 2GB heap (-Xmx2048m). For example, in case of FIN, we are having managed servers for Receivables, Payables etc. I am using this blog to share/document my findings.

Using the Leak Suspects Report

As  soon as you open the heap dump, you get the following dialog:

Selecting Leak Suspect Report option is very handy because if there is a big memory leak then Leak Suspects Report automatically lists which object is retaining most of the heap in a new tab, as shown below:

In the above snap, you can see that there is an active thread is referring to a connection pool which is retaining around 17% of heap. In this case, the next step would be to find out why connection pool is retaining so much of memory (or is there a connection leak going on), and what Thread 3 is currently doing.

However, sometimes it’s not that easy as this tool does not list any memory leak. Then you need to figure out yourself by using various other tools/options provided by EMA to find the root cause of memory pressure.

You can find any memory leak issue using EMA, and I have used to find out the following kind of issues in Fusion ADF technology stack:

1. View Object Leak.
2. Connection/Application Module Leak.
3. Statement Leak.
4. Issues Because of Logging Exceptions.
5. Passivation Issues.

Other posts in this blog explains each of the above topic in detail.