Avoiding cube locks

Post Reply
holger_b
Posts: 131
Joined: Tue May 17, 2011 10:04 am
OLAP Product: TM1
Version: Planning Analytics 2.0
Excel Version: 2016
Location: Freiburg, Germany

Avoiding cube locks

Post by holger_b »

There is quite a number of threads around on how to avoid cube locks. I am trying to put this in a nutshell - can you contribute? This is what I found so far:

Provided we use 9.5.2 FP3 or higher, we can use ParallelInteraction which makes life a lot easier. I found that the IBM say the following actions will still cause cube locking:
  • SaveDataAll
    SetCubeDependencies
    ViewConstruct
    Dimension Updates
    Alias updates

    Security Refresh
    Views with dynamic subsets
From your experience, would you agree so far?

In the environment which I currently work on, we can run most of these things during night time where cube locks do not matter too much. The tricky thing is, we can not restrict data import to night hours, since we need to import data as soon as the source system provides them, and this can be any time during the entire day. So we have to regularly both update dimensions and zero out data, hence manipulating the views which we need for ZeroOut.

Seems to me the best way to deal with this is to separate these things from the actual data imports in the respective chores, using RunTI. I do not find it very elegant to re-use subsets, and often CreateSubsetByMDX is the easiest way to populate a subset properly so I use them after I made them static. Seems to me like this should avoid the "Views with dynamic subsets" topic, but I still have to test this in detail.

Did I miss anything? Which are your best practices?

Thank you
Holger
User avatar
qml
MVP
Posts: 1097
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.9 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: Avoiding cube locks

Post by qml »

holger_b wrote:From your experience, would you agree so far?
Well, it is a bit simplistic approach to say that these operations cause cube locking without going into more detail. It is especially quite simplistic in the context of the locking mechanism being changed almost with every release, sometimes even with Fix Packs and not just major releases. For example, versions 10.1.1 FP2 IF1 and 10.2 FP2 introduced a new mechanism of locking during data save (to start with item 1 on your list), which basically means that from these versions on there is no cube locking for readers or writers during a data save.

Dimension updates are one that always irks me as it's done in a very sub-optimal way. When you edit a dimension via Dimension Editor there is no lock on the dimension until you commit the change. However, when you change a dimension via TI there is a lock on it from the moment you start manipulating the dimension in question (although, actually, you're only manipulating a duplicate of this dimension) until you commit the change at the end of the Metadata tab. This in my opinion is inconsistent and entirely unnecessary. There is no need to lock the dimension (and all associated cubes) until all the changes to the copy of the dimension are done and are to be commited to the original dimension (effectively swapping the changed copy of the dimension for the original one). This is one of the areas IBM can definitely improve on as it could reduce the time when readers and writers have to wait, which in large enterprise models can take quite long.

I'm not quite sure why I keep hearing about dynamic subsets being such a big cause of locking. I don't think that's been the case in a while now. I use them whenever I can, often implicitly turning them to static ones right after creation. I haven't seen any issues caused by dynamic subsets in years, to be honest. I would still not use views based on dynamic subsets as data sources; I would first turn them into static subsets by doing a SubsetElementInsert/Delete, but other than that I'm more of a fan of dynamic subsets than most guys here seem to be.
Kamil Arendt
declanr
MVP
Posts: 1830
Joined: Mon Dec 05, 2011 11:51 am
OLAP Product: Cognos TM1
Version: PA2.0 and most of the old ones
Excel Version: All of em
Location: Manchester, United Kingdom
Contact:

Re: Avoiding cube locks

Post by declanr »

qml wrote: I'm not quite sure why I keep hearing about dynamic subsets being such a big cause of locking. I don't think that's been the case in a while now. I use them whenever I can, often implicitly turning them to static ones right after creation. I haven't seen any issues caused by dynamic subsets in years, to be honest. I would still not use views based on dynamic subsets as data sources; I would first turn them into static subsets by doing a SubsetElementInsert/Delete, but other than that I'm more of a fan of dynamic subsets than most guys here seem to be.
I agree entirely, the only reason I used to avoid creating dynamic subsets in TI was because of the fact it errored when it returned no elements but now IBM have resolved that fact with their (still undocumented) third parameter for the subsetcreatebymdx function. I agree on making them static before using them in a TI datasource view. Never noticed any issues with MDX subsets in views used with user interfaces though (i.e. cube views, excel workbooks or websheets.)


Dimension updates have been made arguably less lock inducing with the use of the "direct" functions but that is only beneficial in certain circumstances - normally I resort to trial and error to see whether it will add benefit or not.


Edit - I should point out that if using 9.5.2 you won't get these benefits.
Declan Rodger
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: Avoiding cube locks

Post by TableManagerOne »

IBM may not have communicated it as well as it could have, but things are getting better. And, of course, as declanr points out, you often need to be up-to-date with releases to take advantage.

Locking during SaveDataAll and CubeSaveData was recently reduced as qml points out.
SetCubeDependencies will lock if it's establishing a new dependency - however dependencies are more effectively extracted from rules at server load/rule update and kept around while relevant, so the function is less necessary.
ViewConstruct also no longer locks the cube. Flew under the radar with SaveDataAll/CubeSaveData changes.
CubeSetLogChanges was an annoying one... Fixed in 10.1.
Views and subsets (dynamic or not) shouldn't be an issue these days.

I certainly agree that the situation could be improved around dimension update (especially wrt cube locking,) alias, and security updates.
holger_b
Posts: 131
Joined: Tue May 17, 2011 10:04 am
OLAP Product: TM1
Version: Planning Analytics 2.0
Excel Version: 2016
Location: Freiburg, Germany

Re: Avoiding cube locks

Post by holger_b »

Thank you very much for your elaborate answers! Your replies make me think we are basically on the right way.

declanr, two things if I may:
Dimension updates have been made arguably less lock inducing with the use of the "direct" functions
What are direct functions? Did I miss something?
the only reason I used to avoid creating dynamic subsets in TI was because of the fact it errored when it returned no elements
Here is my way around that:

Code: Select all

sMDX = <Some weird statement that may result in an empty array>
sMDX = '{UNION({[' | sDim | '].[' | DIMNM(sDim,1) | ']},' | sMDX | ', All)}';
SubsetCreateByMDX(sSubset, sMDX);
SubsetElementDelete(sDim, sSubset,1);
When it comes to dimension updates, we use a standard process which builds an entirely new dimension in the shape in which we want to have the existing one in the end. Only after we did that we make a DimensionDeleteAllElements on the "real" one and rebuild it based on the previously created one, thus avoiding to drop elements that carry values without meaning to. The main reason why we did this was we wanted to re-order existing elements - interesting to read here that this may also help with reducing cube lock time.

I will try and find some time for extended testing, especially regarding how view and subset creation may cause cube locking or not.

Thank you
Holger
declanr
MVP
Posts: 1830
Joined: Mon Dec 05, 2011 11:51 am
OLAP Product: Cognos TM1
Version: PA2.0 and most of the old ones
Excel Version: All of em
Location: Manchester, United Kingdom
Contact:

Re: Avoiding cube locks

Post by declanr »

holger_b wrote:
the only reason I used to avoid creating dynamic subsets in TI was because of the fact it errored when it returned no elements
Here is my way around that:

Code: Select all

sMDX = <Some weird statement that may result in an empty array>
sMDX = '{UNION({[' | sDim | '].[' | DIMNM(sDim,1) | ']},' | sMDX | ', All)}';
SubsetCreateByMDX(sSubset, sMDX);
SubsetElementDelete(sDim, sSubset,1);
The easier way that got introduced as of v10 is to just do:

Code: Select all

 SubsetCreateByMDX ( sSubsetName, sMDX, sDimName ); 
Declan Rodger
hariri
Posts: 1
Joined: Tue Sep 07, 2010 7:53 am
OLAP Product: IBM Cognos TM1
Version: All
Excel Version: All

Re: Avoiding cube locks

Post by hariri »

I experienced lock issues with user actions like "roll-up" and the subset-in-subset-capability.
Undisputable usefull functions, but highly locking because they are implemented as internal dimension changes.

I have sometimes the impression that the locks of such action are server wide, because other users are even not able to logon.
We are so far to set the appropriate parameter to false in high available periods like month closing :( .
lotsaram
MVP
Posts: 3704
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Avoiding cube locks

Post by lotsaram »

declanr wrote:The easier way that got introduced as of v10 is to just do:

Code: Select all

 SubsetCreateByMDX ( sSubsetName, sMDX, sDimName ); 
Do you know if this is documented anywhere Declan? I just checked both the 10.1 and 10.2 documentation libraries and no change to the SubsetCreateByMDX documentation to acknowledge either the new parameter or the change in functionality if the MDX produces an empty set.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
declanr
MVP
Posts: 1830
Joined: Mon Dec 05, 2011 11:51 am
OLAP Product: Cognos TM1
Version: PA2.0 and most of the old ones
Excel Version: All of em
Location: Manchester, United Kingdom
Contact:

Re: Avoiding cube locks

Post by declanr »

lotsaram wrote:
declanr wrote:The easier way that got introduced as of v10 is to just do:

Code: Select all

 SubsetCreateByMDX ( sSubsetName, sMDX, sDimName ); 
Do you know if this is documented anywhere Declan? I just checked both the 10.1 and 10.2 documentation libraries and no change to the SubsetCreateByMDX documentation to acknowledge either the new parameter or the change in functionality if the MDX produces an empty set.
I've looked a couple of times and haven't even been able to find a technote on it but I have tested it a LOT of times and it works perfectly every time in 10.1 and 10.2.


Edit - Just to elaborate since it's not documented - it just allows the function to create an empty dynamic subset without error; if the MDX code is wrong it will still error.
Declan Rodger
lotsaram
MVP
Posts: 3704
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Avoiding cube locks

Post by lotsaram »

I just tested SubsetCreateByMDX( sSubsetName, sMDX, sDimName ) on TM1 10.1 and it works. Interestingly does not work in CX 10.1
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: Avoiding cube locks

Post by paulsimon »

Hi Holger
When it comes to dimension updates, we use a standard process which builds an entirely new dimension in the shape in which we want to have the existing one in the end. Only after we did that we make a DimensionDeleteAllElements on the "real" one and rebuild it based on the previously created one, thus avoiding to drop elements that carry values without meaning to. The main reason why we did this was we wanted to re-order existing elements - interesting to read here that this may also help with reducing cube lock time.
Your method can still potentially delete elements that have data. However, I presume that what you do is to check that your temp version of the dimension has all the base level elements of the real dimension before you use DimensionDeleteAllElements on the real dimension.

You said that you did that to control Dimension Order. I usually just use the }DimensionProperties cube and put in BYHIERARCHY etc - Same parameters as DimensionSortOrder but putting them into }DimensionProperties seems to work more reliably.

For things like Account dimensions I allocate a prefixed number for the consolidations to ensure that eg Revenue comes before Expenses so you have we N_3 with an Alias of Revenue and N_4 with an Alias of Expenses. As the Dimension Sort Order is based on the Element Names this gives the right sort order.

Carrying out Dimension Updates on a Temp Copy is one way of reducing locking. However, if I were you I would avoid the use of DimensionDeleteAllElements. It is too easy to lose data if something goes wrong. I use a standard process that just breaks the consol links leaving the base level elements and consols in place, which ensures that any static subsets etc are not upset by a failure. I have recently been working on some very large dimensions eg over 400,000 elements, where my standard process was too slow. Instead I am using one that deletes consols, where the consols are selected by an MDX subset - this proved to be faster than looping over all elements and using DTYPE to select the consols.

When it comes to reducing locking from minor Dimension Updates that are needed just to add new elements that are encountered while loading data into a cube, I have found that the new DimensionInsertDirect statements still cause locking. The only benefit is that the locking only occurs during the Data Tab step as the MetaData Tab pass over the data is no longer needed. However, it seems that the locking happens as soon as the IF( DIMIX( vDIm, vElem ) = 0 ) test finds that a new element is needed and the DimensionInsertDirect statement is executed. That might be on the first record in the Data Tab step or the last, so there can still be a lot of locking.

The only approach I have seen to minimise this is as follows:

Use the IF( DIMIX test to identify new elements.

If a new element is needed, write the whole row to a file, set a flag and and do an ItemSkip.

In the Epilog if the flag is set then call a sub-process to read from the file, insert the elements, and put in the data.

Locking still happens but if lets say you are loading 1,000,000 records and only 100 records are affected by New elements, then the sub-process only needs to process 100 records, and therefore the duration of the locking is much shorter than it would be for the 1,000,000 records.

Regards

Paul Simon
Wim Gielis
MVP
Posts: 3235
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1, Jedox
Version: PAL 2.1.5
Excel Version: Microsoft 365
Location: Brussels, Belgium
Contact:

Re: Avoiding cube locks

Post by Wim Gielis »

qml wrote:I'm not quite sure why I keep hearing about dynamic subsets being such a big cause of locking. I don't think that's been the case in a while now. I use them whenever I can, often implicitly turning them to static ones right after creation. I haven't seen any issues caused by dynamic subsets in years, to be honest. I would still not use views based on dynamic subsets as data sources; I would first turn them into static subsets by doing a SubsetElementInsert/Delete, but other than that I'm more of a fan of dynamic subsets than most guys here seem to be.
For me too, dynamic data source cube views (and dimension subsets) always contain MDX statements.
I know that I can make them static but I don't want to do that. I have never had issues with it.

@Holger: DIMNM(dimension, 1) is possible, but there is a direct function: DFRST(dimension)
Best regards,

Wim Gielis

IBM Champion 2024-2025
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: Avoiding cube locks

Post by paulsimon »

Hi Holger

To avoid locking caused by TM1 adding Cube Dependencies, I have a utility I wrote that scans the TM1Server.log for any cases where Cube Dependencies have been added and then generates the AddCubeDependency statements. Then I just paste these into a process, and call this from a Startup chore (You can add a parameter to the TM1S.CFG to make it run a Chore when the server starts up).

So long as you do this, TM1 should never have a need to add a cube dependency on the fly, and therefore you should not get a case where locking occurs during normal operation.

Regards

Paul Simon
David Usherwood
Site Admin
Posts: 1458
Joined: Wed May 28, 2008 9:09 am

Re: Avoiding cube locks

Post by David Usherwood »

Nice idea Paul - but wouldn't it make sense if the engine could do it without being told to????
lotsaram
MVP
Posts: 3704
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Avoiding cube locks

Post by lotsaram »

paulsimon wrote:Hi Holger

To avoid locking caused by TM1 adding Cube Dependencies, I have a utility I wrote that scans the TM1Server.log for any cases where Cube Dependencies have been added and then generates the AddCubeDependency statements. Then I just paste these into a process, and call this from a Startup chore (You can add a parameter to the TM1S.CFG to make it run a Chore when the server starts up).

So long as you do this, TM1 should never have a need to add a cube dependency on the fly, and therefore you should not get a case where locking occurs during normal operation.

Regards

Paul Simon
I have something similar that trawls the tm1server.log file and writes a 1 to the intersection of a cube that has dimensions "SYS Cubes" and "SYS Dependent Cubes" so the mapping of cube dependencies is maintained dynamically and a zero suppressed view of this cube is used as the data source for a process to set cube dependencies. This process is run on server startup. Achieves the same but without needing to copy and paste anything. Although there are some issues with the dependency relationship for Element Attributes being recorded in the wrong order in the log file (this might be fixed in 10.2 haven't checked).

This little utility was very important in 9.5 to pre-set dependencies and avoid locking but 10.1 did a much better job of parsing rule files and pre-setting dependencies making it almost redundant. With 10.2 the management of dependencies has apparently improved again so doing this might be totally redundant in the latest version. Definitely useful in 9.5 though if that is indeed the OP's current version.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
Duncan P
MVP
Posts: 600
Joined: Wed Aug 17, 2011 1:19 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2
Excel Version: 2003 2007
Location: York, UK

Re: Avoiding cube locks

Post by Duncan P »

In answer to David's observation it would be nice if all the cube dependencies were worked out up front, and more are done now than in previous versions due to static analysis of the rules when they are loaded.

However the first argument to a DB or ATTR statement can be a string which can be the result of an expression depending on cube data and so there may always be cases where a new cube dependency arises. It would be possible to create a putative dependency on all cubes in the model that match the number of dimensions specified by the rest of the DB statement, and in the respective dimensions of which the statically defined elements of the remaining arguments appear. However this would possibly be a bit over the top.

There is another consideration though. Every cube dependency that is created by static analysis of the rules will add to the cache invalidation work that has to be done every time a cell is written. If the dependency is for cells that have not been read (or not been cached) then this work is wasted. By delaying the adding of the dependency until it is actually needed this overhead is mitigated.

Swings and roundabouts.
declanr
MVP
Posts: 1830
Joined: Mon Dec 05, 2011 11:51 am
OLAP Product: Cognos TM1
Version: PA2.0 and most of the old ones
Excel Version: All of em
Location: Manchester, United Kingdom
Contact:

Re: Avoiding cube locks

Post by declanr »

lotsaram wrote:I just tested SubsetCreateByMDX( sSubsetName, sMDX, sDimName ) on TM1 10.1 and it works. Interestingly does not work in CX 10.1
Noticed this a bit late and it's possibly getting a bit off topic in here but did you test on the base version of CX? I have used it in CX 10 before but can't recall what versions it would have been on.
Declan Rodger
Wim Gielis
MVP
Posts: 3235
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1, Jedox
Version: PAL 2.1.5
Excel Version: Microsoft 365
Location: Brussels, Belgium
Contact:

Re: Avoiding cube locks

Post by Wim Gielis »

Lotsaram wrote:I have something similar that trawls the tm1server.log file and writes a 1 to the intersection of a cube that has dimensions "SYS Cubes" and "SYS Dependent Cubes" so the mapping of cube dependencies is maintained dynamically and a zero suppressed view of this cube is used as the data source for a process to set cube dependencies. This process is run on server startup. Achieves the same but without needing to copy and paste anything. Although there are some issues with the dependency relationship for Element Attributes being recorded in the wrong order in the log file (this might be fixed in 10.2 haven't checked).
Duncan P wrote:However the first argument to a DB or ATTR statement can be a string which can be the result of an expression depending on cube data and so there may always be cases where a new cube dependency arises. It would be possible to create a putative dependency on all cubes in the model that match the number of dimensions specified by the rest of the DB statement, and in the respective dimensions of which the statically defined elements of the remaining arguments appear. However this would possibly be a bit over the top.
Rather than reading the TM1server.log file, I have a utility that checks the RUX files for DB and ATTR and =>DB statements.
This TI process is also scheduled in a startup chore.
Best regards,

Wim Gielis

IBM Champion 2024-2025
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
holger_b
Posts: 131
Joined: Tue May 17, 2011 10:04 am
OLAP Product: TM1
Version: Planning Analytics 2.0
Excel Version: 2016
Location: Freiburg, Germany

Re: Avoiding cube locks

Post by holger_b »

Hi everyone,

thanks a lot for the overwhelming number of answers. Sorry I did not find time to visit the forum for a couple of days so I could not respond to the later ones, but your advice is very much appreciated.

I did some elaborate testing, using the free stress tool which I found in the forum (http://ykud.com/blog/cognos/tm1-cognos/ ... tm1-models), very helpful and absolutely easy to work with. My findings with 9.5.2 FP3 which I am currently using is, apart from the list of things mentioned in the very beginning, I did not manage to produce any cube locks at all:
  • Save Data All
    Establishing Cube Dependencies (Cross cube rule statements (DB, ATTRN/S, etc))
    ViewConstruct(cubename, viewname) in a TI Statement
    Element Maintenance
    Updating an Alias
    Security Refresh
Even Views containing Subsets created by MDX did not produce locks either in my tests, so to me it looks like in this version it is safe to always create "fresh" views with a user name and timestamp in their names plus subsets, using MDX where it makes sense, making them static probably for performance reasons, throwing all that away in the epilog.

Regards
Holger
Post Reply