difference between prolog, metadata, data, epilog

[plkkw]

Hi,

I get the definition of prolog, metadata, data, epilog:
==============================
Prolog
A series of actions to be executed before the data source is processed

Metadata
A series of actions that update or create cube, dimensions, and other metadata structures during processing.

Data
A series of data actions to be executed for each record in the data source.

Epilog
A series of actions to be executed after the data source is processed.
==================

My question is:
1) what is "data source"?
2) Is prolog unable to create cube, dimension (and metadata can do it)?
3) Can i take "prolog, epilog" as feature similar to (database table) before-trigger, after-trigger?
4) what is the processing order of "prolog, metadata, data, epilog"?
5) do you have practical example?

thanks.

[winsonlee]

1) data source definition is quite self explanatory. It is a data that you obtain from somewhere eg CSV file, or using SQL statement to get the data from sql server.

2) prolog (before) - you can use prolog to create cube and dimension eg CubeCreate or DimensionCreate.
metadata (during) - mainly populating the data to the dimension. It will actually goes through every single record from the datasource and write the data to the dimension.
data(during) - mainly writing data onto the cube.
epilog(after) - anything that you want to do after the process is completed, eg run a batch file, save the data on to the hard disk.

3) prolog, metadata and data, epilog are in sequence.

[plkkw]

then, metadata and data is quite similar. do you mean that the process will run coding in metadata first, and coding in data?

[lotsaram]

then, metadata and data is quite similar. do you mean that the process will run coding in metadata first, and coding in data?

Metadata and data tabs are only similar in that all code on these tabs is an implicit loop and is repeated for each record in the data source. The metadata tab is (as the name implies) for updating dimensions. At the conclusion of metadata processing, that is after the last record of the source file is read and the connection to the source is closed then and only then are changes to the dimension(s) that have been updated saved.

If there are data updates only then there should be no code on the metadata tab, and vice versa if you only want to update dimensions. Otherwise you process the data source an additional time unnecessarily.

[ParisHilton]

In my head, I'd add these 'explanations' to help newbies like me understand how it works.

The prolog tab hasn't seen the data yet. So it's good for operations like 'Create Dimension' , 'Delete all elements in a Dimension' , add in a hardcoded element to the dimension , such as 'ALL NL codes'.

The metadata tab, will run through the data source and it's here you can add elements to a dimension based on the data. For example if your data source has an NL code which is not already in the cube you can add it here . I guess (clarification from an expert would be nice), that you need to add it here, so that data can be loaded against it.

The data tab again loops over all the data. Primarily you are loading 'fact' data here in BI terms.

The epilog is for tidying up afterwards, I haven't used it for much other than writing out logs, but I am sure someone can add in a decent example!

Hopefully I've got this right , and if not it's a useful post to be picked apart to share the knowledge!

P

[MarenC]

Hi,

I was told that in some situations processes needed to be split up because certain things created in one process would not be recognised until that process had fully run and therefore a second process would be required to fully complete the tasks. For example dimension builds.

Is there any definitive list which details what is and isn't recognised on the tabs? And when processes need to be split up?

regards, Maren

[orlando]

Is there any definitive list which details what is and isn't recognised on the tabs? And when processes need to be split up?

hi maren,

maybe you should read the documentation - this will give you a good overview of what TM1 can do and how it works.

https://www.ibm.com/support/knowledgece ... -gen2.html
https://www.ibm.com/support/knowledgece ... -gen2.html
https://www.ibm.com/support/knowledgece ... -gen5.html
https://www.ibm.com/support/knowledgece ... -gen6.html

Even if this links are for TM1 10.2.2 and not for PA 2.0 - it'll help.

Best regards
orlando

[MarenC]

Hi Orlando,

How do you know that I haven't read the documentation? Maybe I missed the bit about when you needed to split up processes!

But I take it you don't know and hopefully someone else can provide something useful!

[Emixam]

Hi,

Splitting processes is more a "design technique" instead of a necessity.

I suggest you this white paper on the best practice to develop TM1 Turbo Integrator processes

https://static1.squarespace.com/static/ ... grator.pdf

Also, a good example of splitting process is the Bedrock library (https://github.com/cubewise-code/bedrock). For example, instead of creating a dimension directly in your TI, you can easily use an existing process from Bedrock (Bedrock.Dim.Create.pro).

Enjoy !

[tomok]

Maybe I missed the bit about when you needed to split up processes!

Maybe because there isn't anything in the documentation about splitting up processes and maybe because I've never heard of any such requirement. You can most certainly do dimension builds in one process. There is no technical requirement to break it up. Yes, there may be times, because of certain conditions, where you may split a dimension build into different processes. An example would be a process to unwind the existing hierarchy, another to add new leaf nodes and then another to re-build the hierarchy. However, this would be at the developers discretion, maybe because of code sharing or source data limitations, and not because TM1 forces you to. Maybe you should go back to the person who told you processes need to be split and ask them what they are talking about.

[MarenC]

Hi,

ok thanks for the advice, something must have got lost in translation somewhere along the line!

Incidentally I didn't bring up the guides, Orlando did!

regards, Maren

[Wim Gielis]

For example in the prolog tab you can add an element for allocations. In the Data tab you can process data on the element and in the Epilog tab you cannot delete the element. Therefore you need to have the Prolog tab of a second process to do it.

[lotsaram]

Love a bit of thread necromancy.

For example in the prolog tab you can add an element for allocations. In the Data tab you can process data on the element and in the Epilog tab you cannot delete the element. Therefore you need to have the Prolog tab of a second process to do it.

Well you should be able to and in most cases you can but there have been cases of versions with bugs which would crash the server when the source of a process was deleted on the epilog of the same process. (In the case of deleting a cube. In Wim's example you certainly can delete an element on the epilog with the DeleteDirect function, but admittedly you haven't always been able to do this.)

[Catherine]

I had to split a process in 2 steps several times. But I have just one exemple in my mind at the moment.

This concerns security management.
In order to be sure not to lose private objects, here is the way I manage groups/clients in two processes:
Process 1
1. Prolog
-I delete the links between clients and groups
-I delete all groups (except admin)
-I define the source where I can read all goups and clients relevant for my instance
2. Metadata
-I add groups
-I add clients
3. Data
-I assign clients to groups
4. Epilog
Nothing because the last step is to delete clients which are not related to any group but I CAN NOT use DeleteClient function in an epilog
Process 2
I delete all clients which are not belonging to any group (using DeleteClient function)