9.4 MR1 - memory leakage issue?

[Martin Ingram]

I've just been doing some testing in preparation for a potential upgrade from 9.0 SP3 to 9.4 MR1 (32bit running on WS 2003)

We update TM1 from numerous datasources via TI - approach used is to execute one process including SQL which then writes data out to ascii file. Subsequent processes update dimension structures and load data.

What I'm finding under 9.4 MR1 is that the amount of memory being used by the application ramps up during the data loading stage of my update chores;

Under 9.0 SP3 the server is using a pretty steady 600mb of RAM (this doesn't change when refreshing the data)
In 9.4 MR1 the memory jumped up to 1.6 gb after running TI updates

I did experience something similar at another site running 8.x, but it wasn't to such a great extent and an overnight restart was used to free the memory up again. Anyone else noticed anything similar / able to put this down to anything?

Cheers Guys

Martin

[Martin Ingram]

Logged on Cognos Insight - SR #: 11-108922781

[Ken Vuong]

Hi Martin,

Memory leakage is probably not the right term for this, but I have experienced a similar issue here.
According to IBM\Cognos support, this is just due to the new feeder algorithm that was introduced in TM1 9.1.

Our TM1 application here (64bit) under TM1 9.0Sp3 loaded at 5GB. With TM1 9.4MR1 this has more than doubled to 10.7GB. I can only thank the heavens that we decided to buy 32GB of RAM instead of 16GB when we purchased the hardware.

So sadly to say, it does appear that if you have a medium\large TM1 system, if you go to 9.4 you'll probably need 64 bit.

If you do use TM1 9.4, do use the MaximumCubeLoadThread command as this cut down the load time considerably. This was taken out of TM1 9.0SP3...which was really annoying...

Cheers,
Ken

[Steve Rowe]

Hi Ken,
How are things in the grim North??

this is just due to the new feeder algorithm that was introduced in TM1 9.1

Have you had any detail from IBM on this?

If they are saying that they have doubled the size of the systems by design under 9.4 that seems a bit rich. As far am I am aware this has not bought a performance advantage? Have you tried to push back at all?

Is there any acceptance from them that this is a problem? Any idea if there is something peculiar to your system that is causing this?

I'm getting more and more concerned about moving to the 9.1+ series....

(Oh and big gratz btw I'm suprised you have time to post)!

Cheers,

Steve

[Michel Zijlema]

Our TM1 application here (64bit) under TM1 9.0Sp3 loaded at 5GB. With TM1 9.4MR1 this has more than doubled to 10.7GB. I can only thank the heavens that we decided to buy 32GB of RAM instead of 16GB when we purchased the hardware.

Hi Ken,

I would think the fact that memory usage has doubled coming from TM1 9.1 is due to the fact that TM1 9.4 is on unicode.

Michel

[Mike Cowie]

I would think the fact that memory usage has doubled coming from TM1 9.1 is due to the fact that TM1 9.4 is on unicode.

Michel/Ken:

I'd have to respectfully disagree with the above point, unless Ken's cubes store text almost exclusively - also, I think Ken was talking about 9.0 vs 9.4, not 9.1 vs 9.4. In any event, the unicode aspect might add some additional overhead for the storage of any unicode text (element names, attributes, values in cubes) on the TM1 Server, but in the end any storage of numeric values in a cube is still some structure of arrays of pointers that are used to "store" the 8 byte double values in the cube. Because 64-bit pointers are twice the size of 32-bit pointers (TM1 supposedly does some compression, where possible), you would expect to see something close to double the RAM usage moving from 32-bit to 64-bit TM1, but Ken's always been running 64-bit TM1. The fact that unicode is supported shouldn't have any bearing on this aspect of memory usage.

I have definitely seen significantly increased RAM consumption going from 9.0 to 9.1/9.4, though I haven't done any scientific testing to quantify it across different models. I know there were a lot of changes to the core "engine" in 9.1 that might explain this increase (whether it be feeder changes, additional monitoring hooks, etc), but I've never really pursued that in much detail with IBM/Cognos. What I have pursued was the drop in performance from 9.0 to 9.1+, and while it continues to improve with each release we are still nowhere near 9.0 levels in many TM1 models.

Regards,
Mike

[Michel Zijlema]

Hi Mike,

I agree I was a bit short-cornered here . The amount in which the memory usage increase from (f.i.) 9.0 to 9.4 depends on the model - the amount of text cells (attributes, security) and the size of dimensions (nr. of elements).
I haven't done any testing on 9.4 yet - I'm working mostly with PALO nowadays.

Michel

[ScottW]

Our TM1 application here (64bit) under TM1 9.0Sp3 loaded at 5GB. With TM1 9.4MR1 this has more than doubled to 10.7GB

Hi Ken,

What's your MaximumCubeLoadThreads parameter in the config file? If you start TM1 PerfMon after the server loads how much of that 10Gb is memory in garbage? It might be worthwhile testing loading the server with MaximumCubeLoadThreads=1 and see if this has a positive impact.

We found that with several models migrated from 9.0 to 9.1 that loading with only one CPU resulted in significantly reduced memory consumption and in some cases loading with only a single CPU was significantly faster (load time reduced by 20 - 50% on a case by case basis) than loading on up to 7 CPUs.

The way it was explained by Cognos engineering was that with the changes to core engine when loading over multiple cores when reading a cube and processing feeders each CPU doesn't know if a cell has already been fed, so all cells are fed as it processes feeders. Versus if loading from a single CPU it knows if a cell has already been fed so each cell only gets fed once. This overfeeding can chew through significant amounts of memory and processing time.

If your model is mainly data for reporting then its unlikely to be your issue, but if you have a detailed rule heavy planning model with allocations that involve many => one feeders then this could well be the cause of the blow out in memory consumption on load of the model.

[Steve Rowe]

The way it was explained by Cognos engineering was that with the changes to core engine when loading over multiple cores when reading a cube and processing feeders each CPU doesn't know if a cell has already been fed, so all cells are fed as it processes feeders. Versus if loading from a single CPU it knows if a cell has already been fed so each cell only gets fed once. This overfeeding can chew through significant amounts of memory and processing time.

Hmmm, interesting, it just goes to show what a different beast 9.1+ is to the earlier releases.

My understanding and I'm more than happy to be wrong, is that feeding a cell multiple times will slow start up times but should not increase memory? A cell is only capable of holding a single "feeder pointer", even if that feeder pointer is created multiple times it only exists once.

IBM seem to be saying that a cell is capable of holding multiple feeder pointers? This seems to imply that on a mutli CPU load you have multiple mirror/versions of the server coming up that are some how consolidated when the server is loaded leading to large amounts of garbage memory.

The multi-cpu load thing was I thought introduced to improve load times, wasn't it introduced before9.1+? Your post seems to suggest that it's not worth using in 9.1+

All guess work from our end of course since we don't really have any information on how the engine works....

[ScottW]

Hi Steve,

A cell can still be fed only once. It's just that in a multi-threaded load each CPU is an independent process and doesn't know what the others are doing. The redundant multiple feeding still consumes memory but this goes straight to memory in garbage once the cube is finished loading. Multi-threaded loading wasn't new in 9.1 but I believe this behaviour was.

I wouldn't say that multi-threaded loading isn't worth using on 9.1+. Multi-threaded loading can and does dramatically speed load times for DATA. It is just that for some rule heavy models, especially where it is necessary to feed from a consolidation that if there are potentially many cells that could feed a single cell then the effect of evaluating multiple feeders rather than stopping after the first one can outweigh the performance gain of loading with multiple cores. It's pretty hard to predict what number of cores will result in the best load performance (and this could change as the model changes) so it is often a worthwhile experiment to do.

I haven't done any detailed testing of this in 9.4, it could be that this has been fixed but I am inclined to think not. (Some, Any!) Documentation from Cognos on areas like this would certainly be helpful.

[Ken Vuong]

Hi,

Firstly, Thanks Steve, the little man (Dylan) is hard work but it's all worth it. I can't remember what my golf clubs looks like anymore. All I see is nappies....arrrghhhh...

We have an intensive rules\feeders application here, so I was not too suprised that it went to 10.7GB.
I will test the MaximumCubeLoadThreads = 1 Scott, thanks. I doubt we can use this though as loading in 25 instead of 45 minutes (on TM1 9.0SP3) is very important to us.

Just put the MaximumCubeLoadThreads = 1 on and loaded the server. Hope it finishes before I have to go home tonight

Cheers,
Ken

BTW: Scott, if you wish to find out more about our system, you could read up on the Health Check report Paul Simon did for Equity Group.

[Ken Vuong]

A suprising result. It does appear that the MaximumCubeLoadThreads significantly affects the Ram consumption. Nice one Scott.

On the server that had MaximumCubeLoadThreads=1, loads at 45 mins taking 5GB.
Original server with MaximumCubeLoadThreads=7, loads at 25 mins taking 10GB.
Tested on same hardware, same TM1 database.

Performance wise, they are identical on a few tests I did on Ti import and Rule save. We will still use the MaximumCubeLoadThreads=7 as we have the RAM available.
So I think 9.4 is a different beast to 9.1. Hopefully better as we did not like 9.1 and went to 9.0SP3 instead. The other reason is that Mr Usherwood kept refering to it as TM1 Vista...enough to scare anybody

I might report these findings to IBM\Cognos to see what they say about it.

Cheers,
Ken

[Michel Zijlema]

A suprising result. It does appear that the MaximumCubeLoadThreads significantly affects the Ram consumption. Nice one Scott.

On the server that had MaximumCubeLoadThreads=1, loads at 45 mins taking 5GB.
Original server with MaximumCubeLoadThreads=7, loads at 25 mins taking 10GB.
Tested on same hardware, same TM1 database.
...
I might report these findings to IBM\Cognos to see what they say about it.

Hi Ken,

I reported this to Applix in august 2007. This is the response I got:

This is expected. The memory isn't leaked it is being used during the multi-threaded load. This process requires more memory but once the server is loaded it is released back to the servers memory manager for reuse.

To see this, start the server with >0 threads
login and turn on the performance monitor
(after a minute or two) browse the }statsbyserver cube

if you subtract the memory in garbage from total memory you should come up with something close to the amount of memory needed to start the server when the cube load threads are =0

Michel

[paulsimon]

Ken

The advice I got from Cognos, was to start with MaximumCubeLoadThreads = 1, and gradually increase this. You might find a happy medium somewhere between 1 and 7 that gives you a good load time, but significantly less peak memory usage. However, because the problem is caused by multiple feeders to the same cell, that means that it is very dependent on the particular model, so it is impossible to predict where the happy medium will lie. From memory you have 8 processors so 7 is the most you should go to, but you probably already know that.

If you have conditional feeders that feed from a value that is itself fed, then you should keep MaximumCubeLoadThreads = 0, as if parallel processing is used, TM1 is not able to determine the correct order in which to process the feeders and that may result in values being unfed. Having said that, this is only a problem at load time, and I would guess that running a ProcessFeeders command on the relevant cubes would cure that. However, I haven't tried it. At present I am too busy completely rebuilding an entire system on a new chart of accounts to have time to experiment. (By the way, ReevaluateConditionalFeeders = T only helps with this issue after load time, and not during the load).

Congratulations on your new boy.

Regards

Paul Simon

[Cam]

Ken

The advice I got from Cognos, was to start with MaximumCubeLoadThreads = 1, and gradually increase this.

Now we have 8 CPU on two quad cores using MaximumCubeLoadThreads = 7, I found that 800 MB are on garbage and 700 MB are used (Making 1.5 GB on perfmon cubes server total) our start time is around 1 minute, what i believe is very fast. I am interested in optimizing memory (less garbage memory), if I understood correctly, we can go increasing this parameter? For example start with 0, 1, 2 and see results having less garbage?

I hope my appreciation is correct of this expert information, any advice will be great...

Thanks!

[Steve Rowe]

Hi Cam,
That's correct. Your system is fairly small, especially if you are on a 64 bit OS so you may find that you get a good time and RAM with a much lower setting.

Cheers