Turbo Integrator process provokes memory leak on the TM1 ser

[jaudouig]

Hi all,

Please find bellow the description of an issue I am struggling with at the moment:

Environment:

Cognos BI 10.1.1
TM1 9.5.2

A Framework Manager DMR package is used as a source (via Package Connector) in Turbo Integrator process.
The TI process builds a cube that is then exposed to Cognos BI for reporting.
2 cubes are created/refreshed via this process. The 2 cubes are secured at the cell level. The instance only has these 2 cubes.
The cubes don’t contain any rule.

Description of the issue:

When the TM1 instance is started and the cubes are loaded, a maximum of 4gb are allocated to the instance. This amount doesn’t vary much when the cube is accessed.

However, when the cubes are refreshed via the TI processes described before, the memory allocated to the instance increases by 3-4 gb to a total of 7-8gb. This memory never gets released to the OS.
It gets worse the following time the TI process is ran as the memory allocation increases again by 3-4gb to a total of 11-12gb.
The memory usage increases mainly when data is calculated.

Monitoring the memory usage, we realized that most of the memory used is Garbage Memory, except for the 4gb it needed to load on start.

The only way to free up the memory is by restarting the instance.

I have searched the forum for answers and I did find posts that helped me in my investigations, but nothing that actually helped me solve the issue.
Is there something I could investigate to figure out why so much Garbage Memory? Also why is TI process not using the memory already allocated to the service, but allocates a further 3-4 gb each time?

Thanks for your help

[lotsaram]

What you are seeing is expected behaviour. This is not a memory leak, it is the way TM1 works as memory is not released back to the OS but held by TM1 in internal garbage recycle. I'll leave a more detailed explanation to someone else if anyone feels inclined.

[jaudouig]

Hi,

Thanks for your answer.

I understand that TM1 never releases memory, but why isn't it using the memory already allocated to the instance?

That memory is garbage memory and could potentialy be re-used, right?

Thanks

[David Usherwood]

It is.
I don't think you are that familiar with TM1's calculation cache. When you access cells which are ruled and/or consolidated, TM1 calculated the values you need and retains the values in the calculation cache. This can (and should) get quite large on a heavily modelled cube. When you change a cell which flows through to that cube, the cache is invalidated and the memory released to garbage.
You do say that there are no rules on this cube. I suggest you describe the dimensionality of this cube as it is really hard to push the calc cache to extremes without rules (I do it with rules all the time ). I'm thinking possibly that you have some very large dimensions which you are rebuilding in the Metadata tab using the 'standard' commands (dimensionelementinsert, dimensionelementcomponentadd) rather than the new 'direct' alternatives. Possibly this is hitting memory hard during the metadata process - try turning TM1's Performance Monitor on and using TM1Top to see what's happening.
I am also suspicious about your use of the Package Connector. This isn't used much by the TM1 community as far as I know, and it may, therefore, have issues. Worth trying whether straight SQL worked better.

[jaudouig]

Thanks David for your reply.

The cube is very heavy on the dimension side, both in number of dimensions and size of them. The worst part of it is that I cannot change the internal order of the dimensions to improve performance. I explain why in that other post:
http://www.tm1forum.com/viewtopic.php?f=3&t=8613

Can you point me where I could find information on new "direct" way of rebuilding the metedata?

Thanks

[David Usherwood]

The online help....
or even ibm's own website - google dimensionelementinsertdirect to get
http://pic.dhe.ibm.com/infocenter/ctm1/ ... %3D4_3_6_6

I think it would be worth setting out how many dimensions you have and number of elements in (say) the largest three dimensions. I'm thinking you have been left with a very poorly designed set of structures.