Epilog: The Limitation that went Unnoticed

[Steve Vincent]

Ever since i started using TM1 i was always under the impression that TI code worked in any tab, it just did the job differently depending on which one. Today i have found out that is not the case for at least one function, and probably more.

I had a process where i wanted to list the names of cubes and dimensions that met certain criteria in 2 temporary dimensions. The plan was to then use this list in another process to destroy those cubes / dims. A simple piece of code went in to prolog to list the cubes, and an identical piece in epilog to list the dims. i only did this to keep things tidy and easy to read, i could do it all in prolog but this is where i found the limitation.

DimensionElementInsert does not work in epilog. Even using SaveDataAll at the end makes no difference, it never does the task. Even an asciioutput confirmed the code was being run and the parameters were correct, but TI refuses to do the task. I then made a simple check using this code;

PROLOG

Code: Select all

DimensionCreate( '_TempKillCubes' );
DimensionElementInsert ( '_TempKillCubes' , '' , 'test1' , 'N' );

EPILOG

Code: Select all

DimensionCreate( '_TempKillDims' );
DimensionElementInsert ( '_TempKillDims' , '' , 'test2' , 'N' );

The epilog will not add test2 as an element. If all the code is in prolog it will run fine.

I have tested this in 9.0 and 9.5.1 with the same results. At the very least nuances like this should be mentioned in the documentation for the function, and who knows what other functions are limited this way. Does anyone have any other examples of functions where this happens, or a version of TM1 where it doesn't?

[lotsaram]

I think it is mentioned in some documentation ... somewhere.

At least it has always been my understanding that meta data changes are not possible in the Epilog.

[Steve Vincent]

have just searched every PDF on 9.0 and there is no mention at all. is it possible you dreamt the fact that documentation was complete?

[tomok]

At least it has always been my understanding that meta data changes are not possible in the Epilog.

My experience has been that metadata changes only work in the Prolog or Metadata tabs. Data changes work everywhere EXCEPT the Metadata tab.

[Alan Kirk]

At least it has always been my understanding that meta data changes are not possible in the Epilog.

My experience has been that metadata changes only work in the Prolog or Metadata tabs. Data changes work everywhere EXCEPT the Metadata tab.

Not sure where you got the last bit from; I've never seen a technical problem with updating data in the metadata tab, though IMHO it's a bad practice for self-documentation reasons if no other. I've seen it in processes that I've inherited, though and although it's worked, I've changed it.

In the Metadata tab of a 9.5.1 process, this worked just fine and dandy using the default view of my 2 dimensional Test cube as the source:

Code: Select all

dbl = Value*2;

CellPutN(dbl,'Test', plan_source, plan_version);

[Alan Kirk]

Ever since i started using TM1 i was always under the impression that TI code worked in any tab, it just did the job differently depending on which one. Today i have found out that is not the case for at least one function, and probably more.

I had a process where i wanted to list the names of cubes and dimensions that met certain criteria in 2 temporary dimensions. The plan was to then use this list in another process to destroy those cubes / dims. A simple piece of code went in to prolog to list the cubes, and an identical piece in epilog to list the dims. i only did this to keep things tidy and easy to read, i could do it all in prolog but this is where i found the limitation.

DimensionElementInsert does not work in epilog. Even using SaveDataAll at the end makes no difference, it never does the task. Even an asciioutput confirmed the code was being run and the parameters were correct, but TI refuses to do the task. I then made a simple check using this code;

PROLOG

Code: Select all

DimensionCreate( '_TempKillCubes' );
DimensionElementInsert ( '_TempKillCubes' , '' , 'test1' , 'N' );

EPILOG

Code: Select all

DimensionCreate( '_TempKillDims' );
DimensionElementInsert ( '_TempKillDims' , '' , 'test2' , 'N' );

The epilog will not add test2 as an element. If all the code is in prolog it will run fine.

I have tested this in 9.0 and 9.5.1 with the same results. At the very least nuances like this should be mentioned in the documentation for the function, and who knows what other functions are limited this way. Does anyone have any other examples of functions where this happens, or a version of TM1 where it doesn't?

To be honest this doesn't surprise me. It's always been part of TM1 folklore (i.e., generally believed by many longer-term admins but not found explicitly in the documentation) that the Metadata tab does "something" that refreshes the metadata structures when it completes, but that this something isn't done on any of the other tabs. If this is indeed the case, it would make sense that any metadata changes that were done up to and prior to that tab (i.e. in the Prolog as well) would be implemented, though the same change won't work on the Data / Epilog tabs. Indeed I recall a few questions like this over the years and the usual advice given is to move the code back a tab or two.

Without breaking into Iboglix headquarters and stealing the code for TI (in which case I'd hack it to include search and replace features in the editor; oh, wait, I just gave away the plot for the next Mission Impossible movie) there's no way to be sure what the "something" is, but it's possible to speculate.

When you make metadata changes to a dimension (for example) using the API, you do the following (page 46 of the 9.5.1 API manua)l:

1. Call TM1ObjectDuplicateto make a copy of the dimension you want to update. The copy is an unregistered dimension.
2. Use the calls TM1DimensionElementInsertand TM1DimensionElementComponentAdd to add new elements and components to the duplicated, unregistered dimension.
3. Use the calls TM1DimensionElementDeleteand TM1DimensionElementComponentDeleteto delete unneeded elements and components.
4. Call TM1DimensionCheck to verify the integrity of the new dimension.
5. Call TM1DimensionUpdate to overwrite the old dimension with the new one.

If I had to bet, it would be that TI is only calling the TM1DimensionUpdate function at the end of the Metadata tab and that that function call is the "something" that TI is doing to update the metadata. If I'm correct about that, it would indeed explain why metadata changes need to be done on the Metadata tab or earlier.

And yes, you're correct, the limitation that this imposes on DimensionElementInsert should indeed be documented, as should a bunch of other things.

In relation to your general question of function limitations, aside from the above one of limiting metadata functions to the Metadata tab or earlier I've seen a bunch of instances where people have attempted to use functions that access variables in the Prolog / Epilog tabs where the variables, by definition, have no context. (Notwithstanding that if the variables are populated in the Metadata or Data tabs, they'll still hold the last value populated when the code hits the Epilog tab). Obviously that doesn't work. On the other hand functions like ItemReject will work when you may believe (as I do) that they shouldn't. There's sometimes an element of "suck it and see" to TI coding but I'd regard this particular instance as an exception rather than a rule; most of the functions, as long as they have a valid context, should work equally in any tab.

[Martin Ryan]

To be honest this doesn't surprise me. It's always been part of TM1 folklore (i.e., generally believed by many longer-term admins but not found explicitly in the documentation) that the Metadata tab does "something" that refreshes the metadata structures when it completes, but that this something isn't done on any of the other tabs. If this is indeed the case, it would make sense that any metadata changes that were done up to and prior to that tab (i.e. in the Prolog as well) would be implemented, though the same change won't work on the Data / Epilog tabs. Indeed I recall a few questions like this over the years and the usual advice given is to move the code back a tab or two.

The folklore I heard (by my old boss way back when I first started with TM1) is pretty much as outlined by Alan's API steps - that is, TM1 creates a temporary copy of the dimension(s) you are editing and works with this temporary dimension in the Prolog and Metadata tab. At the completion of the Metadata tab the original dimension is replaced with the edited dimension. The logic is that if you end up with an abortive error in either the Prolog or Metadata then you don't lose all your data.

I've never actually tested this out but I can't remember coming across anything that disproves the theory.

[mce]

It is good to see other experiences here. I also experienced that metadata changes during prolog and metadata tabs are committed at the end of the metadata process. If you need to get some metadata operations to be done after the metadata tab, then you can put them in prolog or metadata tab in another TI process and call that process wherever you want in the main process using ExecuteProcess function.

TI process that involve metadata operations are assumed to be locking the TM1 server. Another strange behaviour we found is that they do not actually lock the server until they actually hit the first metadata operation. For example if you do not have any metadata operation in Prolog, while it is evaluating the source before it starts processing metadata operations in metadata tab, it does not lock the server. But when it hits the first metadata operation, then it locks the server throughout the entireTI process till its end even if it is not doing any further metadata changes for minutes. It would have been much better if it does not lock the server after the metadata tab while it is not doing any further metadata updates. Any more precise or clarifying comments on this locking behaviour would be appreciated.

[LoadzaGrunt]

I am pretty sure Alan's explanation is correct.

Apologies in advance for being the one who copies and pastes from the manual (with 9.4.1 FP3)...

DimensionElementInsert
This is a TM1 TurboIntegrator function, valid only in TurboIntegrator processes.

This function adds an element to a dimension. You can use this function to add numeric, string, or consolidated elements. Note that you cannot use this function on the Data or Epilog tabs of the TurboIntegrator window.

Syntax
DimensionElementInsert(DimName, InsertionPoint, ElName,
ElType);Arguments

Argument
Description

DimName
The dimension to which you want to add a new element.

InsertionPoint
An existing dimension element. The element being added to the dimension will be inserted immediately before this existing element. If this parameter is empty, the new element is added to the end of the dimension.

ElName
The name you want to assign to the new element.

ElType
The element type. There are three possible ElType values:

N - Signifies a numeric element.

S - Signifies a string element.

C - Signifies a consolidated element.


Example
DimensionElementInsert('Region', 'Belgium', 'Netherlands', 'N');

This example adds the numeric element Netherlands to the Region dimension. Netherland displays immediately before Belgium in the dimension definition.

Then there is AddClient which is a bit more informative about where to use particular functions:

AddClient
This is a TM1 TurboIntegrator function, valid only in TurboIntegrator processes.

This function creates a new client on the TM1 server.

Changes applied through the AddClient functions do not take effect until the Metadata procedure in a process is completed.

This function, like all functions that update metadata, should not be used in the Data or Epilog tabs of a process

Syntax
AddClient(ClientName);Arguments

Argument
Description

ClientName
The name of the client you want to add to the TM1 server.

The client name is limited to 255 characters/bytes.


Example
AddClient('Brian');

This example adds the client Brian to the TM1 server.

[lotsaram]

It is good to see other experiences here. I also experienced that metadata changes during prolog and metadata tabs are committed at the end of the metadata process. If you need to get some metadata operations to be done after the metadata tab, then you can put them in prolog or metadata tab in another TI process and call that process wherever you want in the main process using ExecuteProcess function.

TI process that involve metadata operations are assumed to be locking the TM1 server. Another strange behaviour we found is that they do not actually lock the server until they actually hit the first metadata operation. For example if you do not have any metadata operation in Prolog, while it is evaluating the source before it starts processing metadata operations in metadata tab, it does not lock the server. But when it hits the first metadata operation, then it locks the server throughout the entireTI process till its end even if it is not doing any further metadata changes for minutes. It would have been much better if it does not lock the server after the metadata tab while it is not doing any further metadata updates. Any more precise or clarifying comments on this locking behaviour would be appreciated.

Regarding where I have bolded to try and clarify this.
This is not strange behaviour, this is expected behaviour as a thread does not acquire a lock until it needs to upon evaluating the action that it needs to to and whether this will require a lock (possibly not very well documented in the manuals I give you that, but expected behaviour nonetheless...)
A single TI process (or for that matter a chain of TI processes called by ProcessExecute from a master process or a Chore) is handled as a single "transaction" by the TM1 lock manager. If any major errors or lock contention is encountered then the entire transaction will be rolled back. Once a lock is established it is held until the transaction has finished committing.

If you want to use multi-threading then you need to avoid locking and therefore avoid any meta data updates ....

[Steve Vincent]

Thanks guys

I didn't have the 9.5 docs to hand but it's good to see where improvements are being made to them. The explaination via the API makes perfect sense - i've always avoided the API so never had a need to read the guides for it.