TM1 web 10.2.2 memory leaks

[lotsaram]

I have an 10.2.2 instance (up to and including FP2 and the latest interim fixes) where tomcat is regularly going over 30GB. The server has sufficient memory as it has 200 GB and the TM1 server itself is using about 90.

But_this_can_not_be_normal. There has to be memory leaking somewhere for the webserver to be getting anything north of a handful of GB I would have thought. Has anyone else seen this and has IBM said anything about it?

[pandinus]

Have you made any changes to the tomcat configuration, especially in the memory and garbage collection method?

[Duncan P]

It is possible that there is not actually a memory leak, but instead that a large amount of memory has been allocated to Tomcat and the Java run-time is not doing garbage collection because it doesn't need to. Is there any way you can find and post the Java start-up options?

[karkea]

I'm facing similar issues. Symptoms are that memory consumption seems to be growing pretty linear to amount of concurrent users. If we put lower limits for Tomcat memory we face Out of Memory issues so it seems that garbage collector is not able to do its job.

The only idea I have at the moment is that somehow it is generating and loading into memory individual copy of Perspectives templates to each user/user group and therefore can't use garbage collector because of the concurrent usage. But that's just a guess.

[lotsaram]

It is possible that there is not actually a memory leak, but instead that a large amount of memory has been allocated to Tomcat and the Java run-time is not doing garbage collection because it doesn't need to. Is there any way you can find and post the Java start-up options?

I am a real tomcat novice and much more comfortable with IIS but with the help of this technote here are what I think is the full configuration settings for tomcat and the java vm.

Here are the settings for tomcat. Max memory has been set to 65 GB ...

tomcatsettings.png (9.02 KiB) Viewed 10428 times

This is the full list of the advanced java options as plain text:
-Dfile.encoding=UTF-8
-Dcatalina.base=D:\APPS\cognos\tm1_64\tomcat
-Dcatalina.home=D:\APPS\cognos\tm1_64\tomcat
-Djava.endorsed.dirs=D:\APPS\cognos\tm1_64\tomcat\common\endorsed
-Djava.io.tmpdir=D:\APPS\cognos\tm1_64\tomcat\temp
-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
-XX:MaxNewSize=384m
-XX:NewSize=192m
-XX:MaxPermSize=128m
-Xmn32768m
-Dcom.sun.management.jmxremote.port=7999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.password=false
-Djmx.remote.x.server.connection.timeout=1000
-Xgcpolicy:gencon

As you can see the tomcat server is consuming a tad over 35 GB!

tomcatmemoryconsumption.png (9.95 KiB) Viewed 10428 times

The max memory setting and the rest of the tomcat config was actually done by IBM SG. The best explanation I can get as to why it is so high is "there should be at least a 50% buffer between actual usage and the maximum memory allocation setting to avoid slowdown in performance, web freezes and the need to frequently restart tomcat." But as to WHY tomcat is needing so much memory no one has yet provided an explanation. This is a dev environment with 20 developers and testers using it. IMO there's no way tomcat should be needing that much memory and I'm concerned about what's going to happen in production when we have 200-300 users hammering away during the planning cycle.

[Duncan P]

Here is an interesting link http://www-01.ibm.com/support/knowledge ... m_v61.html

The particularly interesting bit is here

-Xmn
This setting controls how much space the young generation is allowed to consume on the heap. Properly tuning this parameter can reduce the overhead of garbage collection, improving server response time and throughput. The default setting for this is typically too low, resulting in a high number of minor garbage collections. Setting this setting too high can cause the JVM to only perform major (or full) garbage collections. These usually take several seconds and are extremely detrimental to the overall performance of your server. You must keep this setting below half of the overall heap size to avoid this situation.

On your server this is set to half the overall heap size.

[lotsaram]

I have set Xmn to 1/4 of the maximum size, will see if this makes a difference.

[lotsaram]

I have set Xmn to 1/4 of the maximum size, will see if this makes a difference.

In the interests of good corporate citizenship I'm reporting back that adjusting the Xmn setting to the recommended 1/4 of maximum memory size does seem to be working and memory used by tomcat6.exe is now in a much more manageable and normal range of btw 2 - 18 Gb depending on usage. As far as anyone can tell, no impact on performance.

[Duncan P]

Excellent. Looks as if the new job has taught me something useful after all.