Memory-related issues occur in the Manager when the amount of the heap space allocated by the Operating System, based on JVM options (-Xms, -Xmx) specified in tms.bat for Windows-based system and tms.sh for Linux-based system, is not enough for the application to continue to behave an expected manner.
Typical symptoms include the following:
Application not being responsive — CPU usage of the Java process being high
Application crashing — terminating
Communication channel(s) flap between the Manager and the device — channel connections being reset frequently
Application not being able to start
The following logs are required for analysis:
Infocollector logs (mainly ems, emsmem, acqount, slowquery, and DB err file).
Threads stack trace and CPU usage using stack trace and collect live objects in heap memory space using the heap dump tool.
Note
These logs are required before restarting the application, which is usually done to restore the application, unless it is recurring issue. Heap dump tool or stack trace doesn't require a restart as, in most cases, memory leak might not be reproduced. And without these logs, an RCA would be extremely challenging.
Steps:
Establish that JVM has experienced memory overload. This can be determined by searching the info collector log with string OutOfMemoryError. The most preferred way is to perform a global search in all the files of InfoCollector whose file name starts with ems* - with wildcard, and this can be done using text editors like TextPad. If there are no search results, it signifies that JVM does not experience any memory issue because of the Manager application, but it could be caused by other applications or some operating System dlls - check JVM crash files.
If there is an exception as mentioned above, check the emsmem logs to know the time of memory and frequency; usually, most cases exhibit either slow memory over a period of days or months, or sudden decrease in memory.
After establishing the time of memory leak, check alert rate in aqcount logs. The recommended value is maximum 60alert/sec; any value above this value over a period of time can cause memory issues. Alert Rate can be calculated from aqcount logs using the following method:
Look for an entry similar to 2012-07-31 13:27:52,012 AltQ:EPR-RCD: 6178500 0 112.Three important information, as mentioned below, need to be extracted:
(t1)timestamp(2012-07-31 13:27:52,012)
Alert received string(AltQ:EPR-RCD)
alert count(6178500).
Now, look for next immediately occurring entry which contains "AltQ:EPR-RCD". This entry will have an alert count greater by 300. So, if the above example is considered, the alert count will be 6178800. Note the (t2)timestamp of this entry.
Alert Rate = 300/(t2-t1)
Check the MariaDB errors logs to find if there are any errors messages.
Check Slowquery logs to find out if there are any queries that are being called repeatedly and taking considerable amount time to execute (that is, more than 5-10 minutes).
Search for all the error messages in ems logs using string "error" as similar to the first step. Observe for the error messages that have occurred during the time interval of memory leak.
If a heap dump — .bin file with prefixes 850heap, 1500heap — is available, it can be used in the heap dump analyzer tools such as MAT and VisualVM, which will identify the suspects causing memory leak.