Slow to open views a few hours after restart of services

[BigG]

hi there, we are version 9.5.2 Tm1, We have a few views (not too large) which are over rule driven cubes. The behaviour we have is on restart the view opens like magic (read very quick), then as the day progresses we start seeing a delay, this will increase until you practically cant open the view. So to resolve we restart the service, and all is back to normal. Has anyone had a similar experience or suggestions to fix our little problem?

thanks in advance

[tomok]

Add more RAM???????

[BigG]

We have 12GB so should be fine, doesnt explain why it is quick to open at start, then slows to a creeping pace over time. Have increased VMM on the cubes to check its not a caching issue, but the views are very small.

[BigG]

been searching a bit on the web:

LockPagesInMemory=T in the Tm1s.cfg file? Only really applicable to operating system is idle for a long period of time, so not helpful.
AllRuleCalcStargateOptimization deosnt exist in our Tm1s.cfg so not an issue

dont know if UseStargateForRules would assist. Default is T anyway.

So still unsure how to resolve

[Duncan P]

This is just a thought and will probably not be the case with you, but ...

If the system contains conditional feeders and if there is significant data change throughout the day (and if you have ReevaluateConditionalFeeders=T) then, as the conditional feeders are re-evaluated and more cells are fed, the cells that should no longer evaluate to fed are still marked as fed in the cubes. This could mean that the view construction is having to calculate a number of cells that evaluate to zero and that this number is increasing through the day. You could check this by turning on monitoring and checking the value of Number of Fed Cells in }StatsByCube against your view construction times.

[tomok]

We have 12GB so should be fine, doesnt explain why it is quick to open at start, then slows to a creeping pace over time. Have increased VMM on the cubes to check its not a caching issue, but the views are very small.

Why do you assume that 12GB "should be fine"? How big is your model? 12GB actually sounds rather small to me. Have you checked the RAM usage late in the day? Have you run Performance Monitor to see what's happening when things are slow? What are your virtual memory settings?

[BigG]

Thanks for the replies.

VMM is set for 524288 for the main cubes. The server and cubes use about 0.5GB of RAM so not a large model

Performance monitor has been switched on and getting users to enter data intest over next few days to see the affect

ReevaluateConditionalFeeders = T is not inthe tm1s.cfg

have also turned onLockPagesInMemory=T to test this has any affect.

Will post updates later

[tomok]

Check your paging file settings as well. If the paging file is on a drive with not much space left on it (like the C: drive on a server) this can play havoc with TM1. Make sure you have at least 12GB (because that's the RAM size) on the drive the paging file is on.

[Andy Key]

As you are on 9.5.2 you should probably take note of this...

LockPagesInMemory doesn't work before 10.1

and if your servers start crashing...

LockPagesInMemory=T can crash servers

Neither of which help with your original problem!

[BigG]

Thanks for the heads up on the lockpagesinmemory, lookslike this function is not much use in 9.5.2

I have also checked the space on drives and all are well over 12 GB,

[BigG]

I thought yesterday the problem was resolved , but it has raised its head again.

With performance monitor turned on I could not see any real impact / increase of memory use. it was more in task manager the CPU usage was very high, to a point it was difficult to actually log on to the server. the users were opening a view that would normally only take a maximum of 30 second to open (if that), but it took 20 minutes

Once again if we restart the Tm1 service everything opens up quickly and works well. It seems like you have to wait more than 24 hours to get the performancee degradation.

Any one have any suggestions here?

[David Usherwood]

I'm _really_ puzzled by this whole thread. TM1 normally speeds up after startup, because of the calculation cache. Have you run TM1Top to see what users are doing?
It might be worth seeing if users are using Sandboxes or Personal Workspaces - I do recall that in earlier versions they didn't cache.
Otherwise I'd go after your infrastructure people - have they misconfigured the server in some way? Is the TM1 instance swapping out to disk too soon?
Strange...

[BigG]

Hi thanks for the reply, since we are in development we have a small number of users testing and they sit within earshot... so I can confirm they do not use sandbox function

Tm1top has been observed and you can see the processing of a view (all cell point calculations), but all they are doing is opening a view

" have they misconfigured the server in some way? Is the TM1 instance swapping out to disk too soon?"

thanks, will chase this, as it seems to be the only obvious reason...

[tomok]

" have they misconfigured the server in some way? Is the TM1 instance swapping out to disk too soon?"

thanks, will chase this, as it seems to be the only obvious reason...

This is why I asked you about your paging file settings. TM1 makes extensive use of paging files, even when you have plenty of RAM. Problems with the paging file can wreak havoc on TM1.

[BigG]

apologies for the apathy on my part. If I go to virtual memry in performance options of system properties I see page filing is "Automatically managed" by the server. With current settings Min 16MB, recomended 18430 and currently allocated 12287 MB on C Drive . Should we be customising? - if so what should the settings be for initial and maximum size for a 12GB TM1 server?

[BigG]

Hi, just an update, was none of the above, instead it is a TM1 product code fix coming through on a hot fix for 9.5.2. An error in the cache mechanism of TM1.

[lotsaram]

Hi, just an update, was none of the above, instead it is a TM1 product code fix coming through on a hot fix for 9.5.2. An error in the cache mechanism of TM1.

Can you provide some extra details? Your problem sounded fairly unusual, more or less unique. If TM1 freezing after a period of the server being up was a general problem then all customers would be up in arms. Makes me think there must also be something fairly unusual going on in this particular model to cause the issue or expose the flaw in the view caching.

[BigG]

Yes ok, long story short. It is unusual but not as unique as I initially thought. Whether it is something in the model that exposed the flaw it is still a flaw and required a code fix to resolve. The cube rules are nothing special, there is a lot of aggregation is occurring from another more detailed cube but only contains 3 reasonably simple rules.

We have spent the last month searching for a trigger or configuration in the environment, tm1 or model. The issue only occurs overnight and only after data entry the day before, so troubleshooting and testing is very slow moving. Only overnight process is a Save Data All, if issue occurs you restart service and all is good again. The issue is a dramatic increase in wait time for specific cube views to open. Another piece of behaviour is the cube view opens just as slow the second time you open it , even if no data is changing in model (cache not working), this behaviour you can replicate during the day (not the dramatic increase in wait time) if you save data all after data entry.

IBM development reproduced after sending debug logs and they followed the steps of data entry, saved data all and left idle overnight. In both logs there was a cache fault error specific to the cube in question, this is a known error and has a fix in 9.5.2 SP2 Hot fix 1. This hot fix has removed the error and issue.

hope this helps

[BigG]

The root cause could be described further as the bug in the TM1 code caused the new version of the cube in memory to distrust the version in cache and therefore ignore it. When the cube was started in the morning it updated the cube and all its dependencies thereby causing a delay. The action that caused the delay was most probably the Save Data All, resetting the "dirty" flag on an upstream cube that occurred overnight.

When a "versioned" cube property is updated, a new version of that cube will be created. Downstream (dependent) cubes are not updated to reflect the new version. When the cache on a downstream cube is being validated, it is likely that the old version referenced will not be found, causing the cache to be ignored. This cache distrust will persist until both cubes (dependency and dependent) are updated in the same transactions (e.g. invalidation).

A new version of the cube is created when data or metadata is changed. When a new cube version is created, its cache is empty. Readers of that version of the cube may calculate and cache values. Each publication to that cache will create a new version of the cache. Old versions of cubes/caches are cleaned up when they go out of scope (no one is referencing them anymore).If a downstream cube refers to the old version that is not correct - this is a bug. Creation of a new version of an upstream cube, regardless of the reason, should cause the creation of a new version of a downstream cube, so that the downstream cube has the correct upstream version referenced. This was not happening in this instance.