TM1 intermittently stops caching rule calculations

[Michel Zijlema]

We're currently running into a nasty situation where the performance of a TM1 cube with a lot of rule calculations intermittently drops significantly.
Checking the statistics for the problem cube I can see in the }StatsByCube cube that the variables 'Memory Used for Views' and 'Number of Stored Calculated Cells' are both zero (which is not the case in our UAT environment running the exactly same setup - technically and data wise). TM1 apparently stops caching rule calculated values.

We're working on AIX running TM1 10.1. There is plenty of memory available.

The issue started to occur a about 2/3 weeks ago (I can't think of anything that changed at that point in time) and seems to be related (but I'm not sure) to data updates in the model and/or data save (commit RAM to disk) operations. I briefly thought that persistent feeders had something to do with this behavior (as purging those turned rule calc caching on again at one point), but apparently it's not, as we've disabled persistent feeders and the problem now occurred again.

I'm a bit at loss at what's happening. We'll probably open another PMR (the list is growing - is the AIX version that instable?). But is there anyone at the forum who has an idea of cause and resolution?

MIchel

[David Usherwood]

Michel, this could be related to
PMR 8.7.87555,019,866
where SaveDataAll causes the server to lose trust in the cache. OK in 10.2, problem in earlier releases (on X64 anyway). Once you update something the caches for the cubes affected are reset. We've taken it up with IBM who have accepted it as a bug but we don't know if they will be working on a patch for pre 10.2 versions.

[Michel Zijlema]

Michel, this could be related to
PMR 8.7.87555,019,866
where SaveDataAll causes the server to lose trust in the cache. OK in 10.2, problem in earlier releases (on X64 anyway). Once you update something the caches for the cubes affected are reset. We've taken it up with IBM who have accepted it as a bug but we don't know if they will be working on a patch for pre 10.2 versions.

Thanks for that David. We'll check this with IBM.
In the meantime I got the caching working again by 'fiddling around'. I changed the VMM settings on the problem cube several times and resaved the rules sheets of the most important cubes within the model, while requesting all kinds of views on the problem cube and at a certain point the rule calculations started to cache again...

Michel

[David Usherwood]

Refiring feeders will clear the cache and reestablish the engine's warm, fuzzy feeling - again, in the cube which the feeders have hit (from, to and downstream).

[declanr]

Refiring feeders will clear the cache and reestablish the engine's warm, fuzzy feeling - again, in the cube which the feeders have hit (from, to and downstream).

I have been hacked off a number of times by this and as standard stick a "p_process_all_feeders" in every single application I build now which just loops through the }cubes dimension and does a CubeProcessFeeders on them... I will be very happy to see this getting resolved.

[Michel Zijlema]

Refiring feeders will clear the cache and reestablish the engine's warm, fuzzy feeling - again, in the cube which the feeders have hit (from, to and downstream).

I know, but yesterday - when the same issue occurred (indeed after a SaveDataAll) - even re-saving the rules sheets didn't work. After an unload/reload of the relevant cubes there even was no rule calculation performed anymore at all - and re-saving the rules sheets again (after forcing a change to the rules sheet) didn't make a difference at all. In the end a server restart (which is a 10 min process, but a 2 hr procedure over here...) was the only way to get everything working again. Annoying.

Michel

[Duncan P]

If a CubeProcessFeeders and a ViewConstruct are in the same transaction, and the ViewConstruct causes a cube dependency to be added then this can cause invalidation.

Also if a user opens a view containing a private subset used as a user-defined consolidation and then logs out, then on logging out the cache of all cubes using the dimension will get cleared.

Is it either of these?

[Michel Zijlema]

If a CubeProcessFeeders and a ViewConstruct are in the same transaction, and the ViewConstruct causes a cube dependency to be added then this can cause invalidation.

Also if a user opens a view containing a private subset used as a user-defined consolidation and then logs out, then on logging out the cache of all cubes using the dimension will get cleared.

Is it either of these?

Hi Duncan,

No, neither of those. There is no automatic feeder processing and view building going on in the update and all users access the model through Cognos BI reports.
This is not just a cache invalidation thing, the model stops caching at all.
What happens is that the model gets updated with fresh data. Next I (in Perspectives) manually open a large view on the reporting cube to validate the results. Opening this view takes about 1.5 min. After that drill-down/-up and slice/dice operations take seconds. But at a certain point I get a call that the performance is gone. When I check in the performance statistics cubes I can see no (rule/view) caching is happening at all. I have some indication now that this 'certain point' is the issue of a Save Data All.
The thing is that I can't experiment with this as this is a well used production environment and I need to go through a 2 hr hassle to get the TM1 Server restarted when something goes wrong...

Michel

[whitej_d]

It's not just cubes which have rules and feeders, it's all cubes in the system which have the ability to cause the mistrust of cache with this issue, so if you have some rates cubes or otherwise which are part of a chain of dependencies, but don't have rules/feeders, they will also need to be invalidated.

The cache has to be invalidated upstream of all the data changes before the savedataall occurred in each dependent branch of the system. This would probably include attributes and control cubes, although I haven't actually tested that.

It's worth noting that the cache will break again as soon as a SaveDataAll is performed again, so ideally you need to engineer some way of invalidating the relevant cube caches immediately after a savedataall.

IBM have accepted it as a bug, and should hopefully fix it. We submitted it with quite high priority and refused to let them downgrade it!

[David Usherwood]

Can forumers who have encountered this issue specify which platform/version/fp level they are using? IBM have 'accepted' it for 10.1.1 but have asked the developers to find out whether it's a problem with 9.5.2. We think it is but haven't checked yet with the latest fp.