This post has been contributed by Timur Vafin. This post deals about What, Why and How of Memory Leaks in Siebel and he has also shared a set of SQL and Perl Script and a tool that could help you uncover possible causes of memory leaks due to not nullifying the objects after use.
I think that many Siebel Administrators know about problem with Siebel process memory leakage when Siebel OS process memory grows constantly and process can not fit to memory boundary (be it LDR_CNTRL environment variable or 2GB process memory limit on 32 bit platform or server resource boundary).
As a result of this problem Object Manager process crashes and generates core dump or FDR file. Siebel Server (siebsvc process) restarts this process again but the session of the users becomes invalid and all their unsaved data is lost. They are required to re-login in application. If this is repeated then UNIX OS start swapping process memory to swap device or file and Siebel performance become too slow and the only way to solve this is to restart Siebel Server.
What causes Memory Leaks?
Configuration: Recursive self joins and complex or wrong expression in calculated fields can result in extensive memory usage and subsequently crashing Siebel Object Manager process (Although this is a rare scenario).
Scripting: One main cause of Memory leaks in my experience is badly written scripts that omit Garbage Collection (GC), let me explain what I mean.
When we assign something to a variable in eScript Siebel reserves some memory to hold the reference and values of that variable. The amount of memory that is reserved for it depends on the type the variable. The only way to get this memory back is to nullify the object used in the script at the end of the script. Assigning a variable as null notifies Siebel GC to destroy the variable release the memory at the next GC run.
var bc = this.GetBusComp(); //siebel reserves some memory for variable bc.
bc = null; //now siebel can release the memory assigned to this variable.
If I don’t nullify an object and the script is executed too frequently Siebel will keep using more and more memory till the point it runs out of it causing the crash. So, if you have lot badly written scripts without appropriate GC implemented you could run in Memory Leaks. The real embarrassing thing for me is that experienced people know about this problem but sometimes they trying to forget about it.
How to find cause of Memory Leaks?
When Siebel Object Manager process crashes, at that time system will generate FDR file in Siebel Server bin directory. You can analyze these FDR files and find out the last steps before crash.
Log files contain very important information about the crash. Monitor OS process that consumes too much memory and then find the component name from Siebel Log using PID. You can search the log files for lines like
“Starting <Component Name> with PID <PID>”.
You can find the Log files at the following location
UNIX: [Siebel Root]/siebsrvr/enterprises/[Enterprise Name]/[Server Name]/log
Windows: [Siebel Root]\siebsrvr\log
To find out what objects are used frequently by Object Manager you can enable 2nd level SARM logs for some time and analyze it. I will omit the details on this but you can find the most useful information by typing command line “sarmquery –tips” and in Siebel Documentation:
Siebel Performance Tuning Guide > Monitoring Siebel Application Performance with Siebel ARM
Siebel Performance Tuning Guide > Analyzing Siebel ARM Data
How to prevent Siebel Crash from Memory Leaks?
For multithreaded components Oracle has introduced “Memory-Based Multithread Component Recycling” feature to prevent Siebel Server from crashing due to memory leaks. This feature can be enabled by setting MemoryBasedRecycle parameter as true.
Once this feature is enabled (by setting parameter MemoryBasedRecycle=true) for a component, Siebel Server stops opening new sessions when Object Manager process reach memory limit defined by parameter MemoryLimit (in mega bytes, default 1500) for this process and starts another process to handle requests. It waits until all tasks are completed on problem process and shuts it down. Although this approach doesn’t fixes the root cause of the memory leak.
More information about this feature you can find in Siebel Documentation:Siebel System Administration Guide > Configuring Siebel Servers > Advanced Configuration Tasks > Configuring Memory-Based Server Component Recycling.
Garbage Collection: To overcome the problem of Memory leaks caused because of objects not nullified in script I have come up with set of SQL’s and Perl Script that can easily help you to:
- Analyzes eScript variables that has been opened but not closed (“null” are not assigned to variable) in the script
- Searches for objects that has been opened using the following functions such as GetAssocBusComp, GetPicklistBusComp, this.BusObject, ActiveBusObject, GetBusObject, GetService, NewPropertySet but not nullified
- Skipped eScript comments in code
- Takes under consideration multiple variables assignments like : var1 = var2 = var3 = <Something>
Furthermore, is you used Oracle Database for Siebel, you can use my tool that I call “Siebel Open Variable Analyzer”. I’ve combined Oracle Instant Client 10g with SQL*Plus, Strawberry perl 5.12, and my perl script.
There are steps to use it:
- Extract SiebelOVAnalyzer.rar to appropriate directory
- Edit connection information for Oracle Database Server (Siebel Database):
- tnsnames.ora file in root directory – change TNS
- run.bat file in root directory – change LOGIN environment variable: Siebel Database User that has sse_role \ tblo_role, password, TNS
- Run “run.bat”, it will produce “out.csv” file in root directory
- Open “out.csv” in Excel and analyze data for your purposes
So now you have a tool to produce list of variables that you should review in your code. You can download the SiebelOVAAnalyzer tool from the link below
Here is the link to download the SQL and Perl script in case if you don’t want to download the utility
I hope my article will help. Below are some good articles about Memory Leaks on Oracle My Support
- How To Troubleshoot Memory Leaks on UNIX [ID 477522.1]
- How To Troubleshoot Memory Leaks on Microsoft Windows [ID 477521.1]
- How To Analyze the FDR Output in Siebel Versions 7.7.x, 7.8.x and 8. [ID 473939.1]
I have tried the utility and it works. Although I had to tweak a few setting to make it work. I will posting an article soon with details of the utility and my experience of using it. Please rate the article and provide your comments.