Question about the dependency between Metadata and Data

[NewtoTM1P]

Hello everyone,

I am new to TM1, I am testing the dependency between Metadata and Data.

As I set up two new TI, one only with Metadata, and one only with Data.

In the Metadata TI,

Code: Select all

Sleep(9000);
DimensionElementInsert('Dependency test', ' ','60,'N');

In the Data TI,

Code: Select all

CellGetN('Dependency test',' 60','b');

There is a cube called 'Dependency test' as well.

Then, I run the Metadata TI first, and Data TI second and the result was Data TI has an error because the '60' is not set up in the dimension yet.

Therefore, I wonder why the Data TI doesn't wait for the metadata TI to set up the element first?
Also, I am curious about how to let the data TI wait for metadata until the metadata is done.

Any help will be appreciated.

[Alan Kirk]

Hello everyone,

I am new to TM1, I am testing the dependency between Metadata and Data.

As I set up two new TI, one only with Metadata, and one only with Data.

There is nothing to be gained by doing that. The Metadata and Data tabs are designed to work together hand in hand.

These are the roles of the tabs in a TI process:

Prolog
This runs ONCE, before the data source (if any) is connected to, read or even looked at by the process.
You can use it to:

• Create a data source by defining a view and its subsets;
• Check for the existence of a file and run a batch file to rename or move it;
• Change the data source from a view to a text file to a dimension subset or whatever you need it to be;
• Dynamically assign a different data source (a different file name, a new view name or whatever) to the process at run time;
• Define any constants;
• Write information to a log file or to the server log;
• Insert new elements to a dimension if you have ones that do not come from your data source;
• Pretty much anything that you need to do before the process even looks at the data source, if any.
• The fact that it runs before you connect to the data source is what allows you to change the data source on this tab as mentioned above using functions like DataSourceType and DatasourceNameForServer.

There is one proviso with this; one thing that you can't do within a TI itself is to change its variable list or the data types. Normally you would define those by using an example file or view at the time that you write the process.

Metadata
This loops through the data source (if any) ONCE. That is, for every record in the data source, the values in each column will be read into the corresponding process variables, then the code on the metadata tab will be executed, then the process will move onto the next row of the data source and the whole process repeats.

The purpose of the tab is to create any metadata (cubes, dimensions etc) that you will need to store the data that you upload on the Data tab.

When you use functions like DimensionElementInsert, changes are made to a copy of the dimension.

After the last record has been processed on the Metadata tab, the real dimension will be overwritten by the copy. If you did any insertions on the Prolog tab, these will also be added at that point.

Typically you will be using element names from your data source's variables to do the element insertion. If you have a hard coded element name as you have in your example code, the TI will add the element on the first pass if it needs to, and spend every other pass saying "Nope, it's already there, Nope, it's already there, Nope, it's already there, etc". This is not what we call "optimum code efficiency". That's why insertions like that are generally done on the Prolog tab.

There are also newer functions like DimensionElementInsertDirect which will push the elements straight into the dimension without creating a copy. Information about such functions will be found in the Reference Guide.

IMPORTANT NOTE FOR NEW PLAYERS: If you don't have a data source, or if you have a data source which has no records, then nothing that you have written in the Metadata tab will ever be executed. Ever.

Data
This will again loop through each row in the data source one at a time, assigning the values in each column to variables, and doing whatever you tell it to. This may be loading values into a TM1 cube, or it may be writing values from a TM1 cube to a text file, or to another database. If you are loading values into a cube it's assumed that you have created any necessary elements in the Metadata tab.

Note that any attributes of elements (alias names, etc) are regarded as being data for the purposes of this exercise and need to be written on the Data tab (unless you used the Direct functions mentioned above).

Epilog
This is run AFTER the last Data record is processed. It is usually used to clean up whatever needs cleaning up, and maybe writing results into a control cube, according to taste.

In the Metadata TI,

Code: Select all

Sleep(9000);

That doesn't achieve anything constructive. It just means that your metadata tab will execute...

Code: Select all

v e r r r r r y       s  l  o  w  l y.

Code: Select all

DimensionElementInsert('Dependency test', ' ','60,'N');

As mentioned above, all you are doing is trying to add the same element, '60', over, and over, and over again, for every record in your data source.

Of course as also noted above if you have an empty data source this code will never execute, so the element will never be inserted.

In the Data TI,

Code: Select all

CellGetN('Dependency test',' 60','b');

There is a cube called 'Dependency test' as well.

That's all fine and dandy, but we don't know for certain that the dimension "Dependency test" is the first dimension in that cube. If it's not, then you're getting a different error to the one that you think you're getting. However more likely...

Then, I run the Metadata TI first, and Data TI second and the result was Data TI has an error because the '60' is not set up in the dimension yet.

Therefore, I wonder why the Data TI doesn't wait for the metadata TI to set up the element first?

If you really did run the two processes in sequence, then obviously there was nothing for the "Data TI" to "wait for"; the "Metadata TI" had already been done. What is the most likely explanation, then, assuming that "Dependency test" is the first dimension in your cube?

It's that the element was never added.

And what is the most likely cause of that?

That you had no data in the data source of your Metadata process, meaning that the line of code that adds the element was never executed.

To check that, I suggest running the metadata process first, then checking to see whether the new element is there. If it's not, then you know that you have a problem.

Then I suggest that you scrap your two processes and combine them into a single process, the way the thing was designed to work.

[NewtoTM1P]

Hello everyone,

I am new to TM1, I am testing the dependency between Metadata and Data.

As I set up two new TI, one only with Metadata, and one only with Data.

There is nothing to be gained by doing that. The Metadata and Data tabs are designed to work together hand in hand.

These are the roles of the tabs in a TI process:

...

Then I suggest that you scrap your two processes and combine them into a single process, the way the thing was designed to work.

Thank you very much for such a comprehensive and detailed explanation about the tabs and the TI process.

[jim wood]

I think this should be pinned for a short while. The way tabs work is something fundamental to TI and there are plenty of people out that may not understand it fully, Especially how dimension inserts work on a ghost copy etc.