Memory Used for input data
-
- Posts: 3
- Joined: Mon May 26, 2014 2:39 pm
- OLAP Product: TM1
- Version: 10.1
- Excel Version: 2013
Memory Used for input data
I am working on a TM1 (Version 10.2.2) database with two large reporting cubes. In the data load TI's it is necessary to use CellIncrementN rather than CellPutN, and it has always been the case that a significant amount of garbage memory is accumulated during a full cube load. I have always assumed this is due to the CellIncrementN function issuing a read query in a background and these results being cached. What I have also noticed is there is a significant difference in the memory used for stored input data, if I check immediately after a restart versus checking after executing the load TI's. Can anyone help explain how the value for memory used for input data in }StatsByCube is captured by TM1 as I did not expect executing a data load would change when the number of populated cells has not changed.
- Attachments
-
- Post Restart.png (22.47 KiB) Viewed 5053 times
-
- Post Data Load.png (30.41 KiB) Viewed 5053 times
-
- Cube Stats.JPG (40.9 KiB) Viewed 5053 times
-
- MVP
- Posts: 3703
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: Memory Used for input data
I agree that this does seem odd.
Firstly a "P&L cube" with 24 dimensions! Never seen that before, but let's move on.
More questions than answers in terms of what might be going on.
- is performance monitor on? (I assume yes as memory is changed even if # populated cells hasn't)
- have you checked that the DBRW formulas in the excel report to }StatsByCube are pointing to the correct measures?
- do you get the same result if you replace CellIncrementN with CellGetN + NewValue into CellPutN?
- the number of populated cells is the same but is the location of all leaf data the same?
Due to the "Trie" structure in which TM1 stores data the memory used per populated cell will depend on the dimensionality of the data and the indexing of the dimensions in the cube. This is why sparsity and dimension order is so important with regards to memory consumption and speed of retrieval. It is quite possible that for the same number of populated cells if the location of those cells changes then the memory required to store the data will also change. I would guess this is probably unlikely in your described scenario but you never know ....
Otherwise it it could be that what is reported in }StatsByCube is incorrect and there is memory used by feeders or memory used for calculations which is being counted in the memory for input data measure. Only someone from IBM with knowledge of TM1's internals would be able to give you a proper answer.
Firstly a "P&L cube" with 24 dimensions! Never seen that before, but let's move on.
More questions than answers in terms of what might be going on.
- is performance monitor on? (I assume yes as memory is changed even if # populated cells hasn't)
- have you checked that the DBRW formulas in the excel report to }StatsByCube are pointing to the correct measures?
- do you get the same result if you replace CellIncrementN with CellGetN + NewValue into CellPutN?
- the number of populated cells is the same but is the location of all leaf data the same?
Due to the "Trie" structure in which TM1 stores data the memory used per populated cell will depend on the dimensionality of the data and the indexing of the dimensions in the cube. This is why sparsity and dimension order is so important with regards to memory consumption and speed of retrieval. It is quite possible that for the same number of populated cells if the location of those cells changes then the memory required to store the data will also change. I would guess this is probably unlikely in your described scenario but you never know ....
Otherwise it it could be that what is reported in }StatsByCube is incorrect and there is memory used by feeders or memory used for calculations which is being counted in the memory for input data measure. Only someone from IBM with knowledge of TM1's internals would be able to give you a proper answer.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
- Steve Rowe
- Site Admin
- Posts: 2456
- Joined: Wed May 14, 2008 4:25 pm
- OLAP Product: TM1
- Version: TM1 v6,v7,v8,v9,v10,v11+PAW
- Excel Version: Nearly all of them
Re: Memory Used for input data
It might be worth posting your cfg as well?
I'm wondering if something like this "ForceReevaluationOfFeedersForFedCellsOnDataChange=T" is in place and causing all your feeders re-fire, though the stats cube would have to be mis-reporting as lotsaram said.
Are you doing a full re-load of all data?
How are clearing the data out? I'm wondering if the RAM during the data load is from the large view that would be built if using the ViewZeroOut, if doing a full reload then do a CubeClearData instead (assuming no external feeders from the cube).
You could test a bit to see what's going on by exporting all the data from the cube, watch out for disc space! Then importing it without the cellincrement and with different (or just no) ViewZeroOut.
Your cube has so many dims that it would be considered pretty unusual so I wouldn't be surprised if you are encountering behaviour that the forum found difficult to explain.
HTH
I'm wondering if something like this "ForceReevaluationOfFeedersForFedCellsOnDataChange=T" is in place and causing all your feeders re-fire, though the stats cube would have to be mis-reporting as lotsaram said.
Are you doing a full re-load of all data?
How are clearing the data out? I'm wondering if the RAM during the data load is from the large view that would be built if using the ViewZeroOut, if doing a full reload then do a CubeClearData instead (assuming no external feeders from the cube).
You could test a bit to see what's going on by exporting all the data from the cube, watch out for disc space! Then importing it without the cellincrement and with different (or just no) ViewZeroOut.
Your cube has so many dims that it would be considered pretty unusual so I wouldn't be surprised if you are encountering behaviour that the forum found difficult to explain.
HTH
Technical Director
www.infocat.co.uk
www.infocat.co.uk
-
- Posts: 3
- Joined: Mon May 26, 2014 2:39 pm
- OLAP Product: TM1
- Version: 10.1
- Excel Version: 2013
Re: Memory Used for input data
Thanks for taking the time to respond, some further info, I took a smaller example this time a cube with 9 dims and populated 2 million cells from a TI.
I tried various combinations of CellIncrementN, CellGetN + CellPutN, CellPutN on it's own.
Step 1 TI A. is executed, Cube is cleared using CubeClearData
Step 2 TI B. is executed, Data is loaded the memory used for input data is X, my source data does not have >1 value for any cube cell
Step3 TI B is executed again, overwriting all values in the cube, memory used for input data is increased to circa X * 1.8
Step 4 Executing TI B again further times does not increase the memory used for input data once the cell is initially overwritten
I tried various combinations of CellIncrementN, CellGetN + CellPutN, CellPutN on it's own.
Step 1 TI A. is executed, Cube is cleared using CubeClearData
Step 2 TI B. is executed, Data is loaded the memory used for input data is X, my source data does not have >1 value for any cube cell
Step3 TI B is executed again, overwriting all values in the cube, memory used for input data is increased to circa X * 1.8
Step 4 Executing TI B again further times does not increase the memory used for input data once the cell is initially overwritten
-
- MVP
- Posts: 264
- Joined: Mon Nov 03, 2014 8:23 pm
- OLAP Product: TM1
- Version: 9.5.2 10.1 10.2 PA2
- Excel Version: 2016
Re: Memory Used for input data
From the docs:
PI essentially allows multiple versions of cube data to reside in memory. To do this, the initial startup memory consumption would be minimal. However, a large data load would need to allocate enough extra memory to hold the data changes/additions as part of a new version. This memory would be tracked as input data memory, even after any old versions become unnecessary. The extra, unused memory doesn't go into general garbage memory; it instead stays allocated for future reuse on the actual cube data.
http://www-01.ibm.com/support/knowledge ... ction.htmlParallel Interaction impact on Cognos TM1 server memory
Internal testing comparing Cognos TM1 9.5.1 to Cognos TM1 9.5.2 with Parallel Interaction have shown the Cognos TM1 9.5.2 Server to allocate between 10% to 30% more RAM than the Cognos TM1 9.5.1 server. Some of this increased memory use is due to the internal structures created by Parallel Interaction, however the additional throughput of transactions and their impact on View Cache is also a contributing factor. Plan for 30% more memory usage with Cognos TM1 9.5.2 than with prior versions of Cognos TM1 .
PI essentially allows multiple versions of cube data to reside in memory. To do this, the initial startup memory consumption would be minimal. However, a large data load would need to allocate enough extra memory to hold the data changes/additions as part of a new version. This memory would be tracked as input data memory, even after any old versions become unnecessary. The extra, unused memory doesn't go into general garbage memory; it instead stays allocated for future reuse on the actual cube data.
- Steve Rowe
- Site Admin
- Posts: 2456
- Joined: Wed May 14, 2008 4:25 pm
- OLAP Product: TM1
- Version: TM1 v6,v7,v8,v9,v10,v11+PAW
- Excel Version: Nearly all of them
Re: Memory Used for input data
Ok that makes sense however the scale of the RAM used seems much higher than the tech docs stated 30% vs 80% in the test.
That said the wording of the doc and your post suggest the extra RAM is proportional to the volume of the data load, so it doesn't seem sensible for the tech doc to put hard limits on the RAM growth from PI.
That said the wording of the doc and your post suggest the extra RAM is proportional to the volume of the data load, so it doesn't seem sensible for the tech doc to put hard limits on the RAM growth from PI.
Technical Director
www.infocat.co.uk
www.infocat.co.uk
-
- Posts: 3
- Joined: Mon May 26, 2014 2:39 pm
- OLAP Product: TM1
- Version: 10.1
- Excel Version: 2013
Re: Memory Used for input data
Hi Brian
Thank you for sharing that knowledge, I have tested with PI off in my test environment and it behaves as you describe.
Thank you for sharing that knowledge, I have tested with PI off in my test environment and it behaves as you describe.