Memory Used for input data

Post Reply
shanley
Posts: 3
Joined: Mon May 26, 2014 2:39 pm
OLAP Product: TM1
Version: 10.1
Excel Version: 2013

Memory Used for input data

Post by shanley »

I am working on a TM1 (Version 10.2.2) database with two large reporting cubes. In the data load TI's it is necessary to use CellIncrementN rather than CellPutN, and it has always been the case that a significant amount of garbage memory is accumulated during a full cube load. I have always assumed this is due to the CellIncrementN function issuing a read query in a background and these results being cached. What I have also noticed is there is a significant difference in the memory used for stored input data, if I check immediately after a restart versus checking after executing the load TI's. Can anyone help explain how the value for memory used for input data in }StatsByCube is captured by TM1 as I did not expect executing a data load would change when the number of populated cells has not changed.
Attachments
Post Restart.png
Post Restart.png (22.47 KiB) Viewed 5053 times
Post Data Load.png
Post Data Load.png (30.41 KiB) Viewed 5053 times
Cube Stats.JPG
Cube Stats.JPG (40.9 KiB) Viewed 5053 times
lotsaram
MVP
Posts: 3703
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Memory Used for input data

Post by lotsaram »

I agree that this does seem odd.

Firstly a "P&L cube" with 24 dimensions! Never seen that before, but let's move on.

More questions than answers in terms of what might be going on.
- is performance monitor on? (I assume yes as memory is changed even if # populated cells hasn't)
- have you checked that the DBRW formulas in the excel report to }StatsByCube are pointing to the correct measures?
- do you get the same result if you replace CellIncrementN with CellGetN + NewValue into CellPutN?
- the number of populated cells is the same but is the location of all leaf data the same?

Due to the "Trie" structure in which TM1 stores data the memory used per populated cell will depend on the dimensionality of the data and the indexing of the dimensions in the cube. This is why sparsity and dimension order is so important with regards to memory consumption and speed of retrieval. It is quite possible that for the same number of populated cells if the location of those cells changes then the memory required to store the data will also change. I would guess this is probably unlikely in your described scenario but you never know ....

Otherwise it it could be that what is reported in }StatsByCube is incorrect and there is memory used by feeders or memory used for calculations which is being counted in the memory for input data measure. Only someone from IBM with knowledge of TM1's internals would be able to give you a proper answer.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
User avatar
Steve Rowe
Site Admin
Posts: 2456
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: Memory Used for input data

Post by Steve Rowe »

It might be worth posting your cfg as well?

I'm wondering if something like this "ForceReevaluationOfFeedersForFedCellsOnDataChange=T" is in place and causing all your feeders re-fire, though the stats cube would have to be mis-reporting as lotsaram said.

Are you doing a full re-load of all data?
How are clearing the data out? I'm wondering if the RAM during the data load is from the large view that would be built if using the ViewZeroOut, if doing a full reload then do a CubeClearData instead (assuming no external feeders from the cube).

You could test a bit to see what's going on by exporting all the data from the cube, watch out for disc space! Then importing it without the cellincrement and with different (or just no) ViewZeroOut.

Your cube has so many dims that it would be considered pretty unusual so I wouldn't be surprised if you are encountering behaviour that the forum found difficult to explain.

HTH
Technical Director
www.infocat.co.uk
shanley
Posts: 3
Joined: Mon May 26, 2014 2:39 pm
OLAP Product: TM1
Version: 10.1
Excel Version: 2013

Re: Memory Used for input data

Post by shanley »

Thanks for taking the time to respond, some further info, I took a smaller example this time a cube with 9 dims and populated 2 million cells from a TI.
I tried various combinations of CellIncrementN, CellGetN + CellPutN, CellPutN on it's own.
Step 1 TI A. is executed, Cube is cleared using CubeClearData
Step 2 TI B. is executed, Data is loaded the memory used for input data is X, my source data does not have >1 value for any cube cell
Step3 TI B is executed again, overwriting all values in the cube, memory used for input data is increased to circa X * 1.8
Step 4 Executing TI B again further times does not increase the memory used for input data once the cell is initially overwritten
BrianL
MVP
Posts: 264
Joined: Mon Nov 03, 2014 8:23 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2 PA2
Excel Version: 2016

Re: Memory Used for input data

Post by BrianL »

From the docs:
Parallel Interaction impact on Cognos TM1 server memory

Internal testing comparing Cognos TM1 9.5.1 to Cognos TM1 9.5.2 with Parallel Interaction have shown the Cognos TM1 9.5.2 Server to allocate between 10% to 30% more RAM than the Cognos TM1 9.5.1 server. Some of this increased memory use is due to the internal structures created by Parallel Interaction, however the additional throughput of transactions and their impact on View Cache is also a contributing factor. Plan for 30% more memory usage with Cognos TM1 9.5.2 than with prior versions of Cognos TM1 .
http://www-01.ibm.com/support/knowledge ... ction.html


PI essentially allows multiple versions of cube data to reside in memory. To do this, the initial startup memory consumption would be minimal. However, a large data load would need to allocate enough extra memory to hold the data changes/additions as part of a new version. This memory would be tracked as input data memory, even after any old versions become unnecessary. The extra, unused memory doesn't go into general garbage memory; it instead stays allocated for future reuse on the actual cube data.
User avatar
Steve Rowe
Site Admin
Posts: 2456
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: Memory Used for input data

Post by Steve Rowe »

Ok that makes sense however the scale of the RAM used seems much higher than the tech docs stated 30% vs 80% in the test.

That said the wording of the doc and your post suggest the extra RAM is proportional to the volume of the data load, so it doesn't seem sensible for the tech doc to put hard limits on the RAM growth from PI.
Technical Director
www.infocat.co.uk
shanley
Posts: 3
Joined: Mon May 26, 2014 2:39 pm
OLAP Product: TM1
Version: 10.1
Excel Version: 2013

Re: Memory Used for input data

Post by shanley »

Hi Brian
Thank you for sharing that knowledge, I have tested with PI off in my test environment and it behaves as you describe.
Post Reply