EMA :2: Finding View Object Leaks

Once I faced an issue in FIN Payables (AP) where the managed server was going down because of Out Of Memory (OOM) errors in every 3-4 hours. Initially I was clueless about what’s going on but after taking 3 heap dumps for 3 OOM errors and comparing them, I was able to figure out that there is a VO Leak and which AM is causing this. Later on, after having a meeting with Development team, I was able to figure out which peace of code is causing this. Here are the details of how I found the details:

After getting the heap dumps, I was just looking at the histograms of each of the heap dumps. The following image shows how you can open histogram in EMA for a heap dump:

Note:  These images are taken from a different heap dump than the one where VO leak was identified This is done just for the documentation purpose as I found the VO leak issue couple of years back and don’t have that heap dump with me right now.

The first obvious difference that I found was the instance count of OAViewObjectImpl class. It was alarming and was very high in each heap dump, and then I came up with the following figures:

• First Heap Dump : Total OAViewObjectImpl object count is : 10,323, out of which 1760 are of BookControlPVO.
• Second Heap Dump : Total OAViewObjectImpl object count is 16224, 8774 are of BookControlPVO.
• Third Heap Dump : Total OAViewObjectImpl object count is 19062, 13661 are of BookControlPVO.

So it was obvious that BookControlPVO was getting leaked. Now the question was how and from where. The next thing that I found out was which AM this VO was associated with. For that, you need to check the metadata of a View Object instance. The following object shows how to open View Object instances metadata window:

The ‘With Outgoing References’ link lists all the view object instances. Now each VO instance metadata has a mParent field which holds a reference to the AM the current VO instance is associated with, as shown below:

Once I came to know about that, I ran a simple query in OQL to figure out which AM is referenced the most and from which VO:

SELECT toString(s.mViewDef.mFullName), s.mParent FROM oracle.apps.fnd.applcore.oaext.model.OAViewObjectImpl s

Once you run this query, you can export that output to a .txt or .csv file (to parse it using awk or sed), as shown in the figure below:

After exporting, you can run the following awk and sed commands to find out which VO is being created from which AM:

awk -F"|" '{print "VO="$1" Parent="$2 }' ViewObjectList.txt | sort | uniq -c > ViewObject2.txt    

sed 's/^[^0-9]*//g' ViewObject2.txt | sed 's/\([^ ]\)[ ]*$/\1/g' | sed 's/[ ]/:/' | sed 's/^\([1-9][0-9]:\)/0\1/' | sed 's/^\([1-9]:\)/00\1/' | sort -r | sed 's/$/\n/' > ViewObjects3.txt

sed 's/VO[ ]*Parent/VO Parent/; s/Parent=[ ]*oracle/Parent= oracle/' ViewObjects3.txt > ViewObject4.txt

And the following script can be used to figure out which AM is creating how many VOs:

sed 's/=/ /g' ViewObject4.txt > ViewObject5.txt

awk '{print  $4 ", "$2", "$1}' ViewObject5.txt | sort | uniq -c > ViewObject6.txt

In my case, I was able to come up with the following figures:

510:VO=oracle.apps.financials.assets.sharedSetup.systemControls.publicView.SystemControlPVO Parent= oracle.apps.financials.assets.additions.uiModel.applicationModule.AdditionInterfaceUIAMImpl [id=0xbeabe770]

443:VO=oracle.apps.financials.assets.sharedSetup.systemControls.publicView.SystemControlPVO Parent= oracle.apps.financials.assets.additions.uiModel.applicationModule.AdditionInterfaceUIAMImpl [id=0xb1c08c08]

3638:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xbbc743e0]

3111:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xcdb3e508]

2967:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xa6a80c00]

287:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.additions.uiModel.applicationModule.AdditionInterfaceUIAMImpl [id=0xbeabe770]

228:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.additions.uiModel.applicationModule.AdditionInterfaceUIAMImpl [id=0xb1c08c08]

1992:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xa9af0ff0]

1331:VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl [id=0xa9c1c658]

. . .

The format for the above figures is:

<VO instance count>:VO=<VO Name> Parent=<AM Name>

So in the above case, the initial number for example 3638:, 3111: represents the VO instance count, VO=oracle.apps.financials.assets.sharedSetup.bookControls.publicView.BookControlPVO represents the name of the VO, and Parent= oracle.apps.financials.assets.adjustments.search.uiModel.applicationModule.SearchUIAMImpl represents the AM being referred as parent from the VO.

After getting this data, we had a meeting with the appropriate Development team and after investigating the code, figured out the piece of code that is causing this issue. The problem was that the instances of BookControlPVO VO were created dynamically from a utility method (was exposed to other teams as well) but were not removed after use.