[10.2] Sparse consolidation

[hittony]

Hi,

Sparcity from IBM Knowledage Center

When consolidating data in cubes that have rules defined, TM1 turns off this sparse consolidation algorithm because one or more empty cells may be calculated by a rule. Skipping rules-calculated cells will cause consolidated totals to be incorrect.

If the skipped cells are empty, even they are rules-calculated, how can it be possible that the consolidation is incorrect ?

Thanks for your help

[qml]

If the skipped cells are empty, even they are rules-calculated, how can it be possible that the consolidation is incorrect ?

If the cells are empty then they will be ignored by the consolidation engine. The consolidation results can be incorrect because only non-empty cells will be taken into account, where 'non-empty' means 'containing a real value'. In most cases you'd want all rule-calculated values to consolidate up, so if not all of them do then that result can be considered 'incorrect', wouldn't you agree?

There are many good posts on the Forum explaining skipcheck, feeders and the sparse consolidation algorithm. Probably not much point repeating it all.

[hittony]

The consolidation results can be incorrect because only non-empty cells will be taken into account, where 'non-empty' means 'containing a real value'.

Hi qml

“Empty cells” can contain “fictive values” which should be consolidated ? How can it be ?

Could you kindly give a concret example ?

Thanks again

[qml]

“Empty cells” can contain “fictive values” which should be consolidated ? How can it be ?

I'm not quite sure who/where you are quoting from, so I'm not really sure what I should clarify. These phrases in scare quotes are not very precise and could mean different things. But let me explain this in another way.

There are three possible states of a cube cell:

A) It is empty (holds no data) and unfed. This means it takes up no memory because the data structure (which from the technical perspective is a branch of a digital tree type called a trie) does not exist.

B) It is non-empty (holds data). This means the relevant branch of the digital tree exists and there is an 'apple' (data) on it. The total memory used is however much the branch is taking (varies with dimension number, size and order) plus the size of the 'apple' which is 64 bits for numeric data (double-precision floating point number format).

C) It is empty (holds no data) but fed. This means the relevant branch of the digital tree exists but there is no 'apple' (data) on it. The total memory used is however much the branch is taking plus 1 bit for the feeder flag (let's say it's a small 'flower'). Please note that such a cell takes up more memory than in scenario A, but less than in scenario B.

On top of the above three scenarios a cell can have a rule applied to it, but, interestingly, that is irrelevant. Regardless of whether there is a rule applied to the cell, TM1's sparse consolidation algorithm will skip any cells in state A and only look at cells in state B or C (so it will move through all branches of the tree relevant to the consolidation requested). If the sparse consolidation algorithm is switched off (i.e. there is a rule attached to the cube, but no SKIPCHECK) TM1 will include any cell in its consolidation, whether they are A, B or C.

One additional thing to mention - the process of building branches in the digital tree is a costly one. It happens during feeding, data writing or loading cube into memory. And there is no pruning of branches that are no longer used. In other words, a specific branch of a tree gets created when you want to put a 'flower' or an 'apple' on it. And when you later pick the 'apple' the branch does not get removed - it stays there for the entire duration of the server session, taking up space and being included in any consolidations performed by the server engine.