difference between prolog, metadata, data, epilog
-
- Posts: 23
- Joined: Wed Sep 21, 2011 6:18 am
- OLAP Product: TM1
- Version: 9.4.1
- Excel Version: 2003
difference between prolog, metadata, data, epilog
Hi,
I get the definition of prolog, metadata, data, epilog:
==============================
Prolog
A series of actions to be executed before the data source is processed
Metadata
A series of actions that update or create cube, dimensions, and other metadata structures during processing.
Data
A series of data actions to be executed for each record in the data source.
Epilog
A series of actions to be executed after the data source is processed.
==================
My question is:
1) what is "data source"?
2) Is prolog unable to create cube, dimension (and metadata can do it)?
3) Can i take "prolog, epilog" as feature similar to (database table) before-trigger, after-trigger?
4) what is the processing order of "prolog, metadata, data, epilog"?
5) do you have practical example?
thanks.
I get the definition of prolog, metadata, data, epilog:
==============================
Prolog
A series of actions to be executed before the data source is processed
Metadata
A series of actions that update or create cube, dimensions, and other metadata structures during processing.
Data
A series of data actions to be executed for each record in the data source.
Epilog
A series of actions to be executed after the data source is processed.
==================
My question is:
1) what is "data source"?
2) Is prolog unable to create cube, dimension (and metadata can do it)?
3) Can i take "prolog, epilog" as feature similar to (database table) before-trigger, after-trigger?
4) what is the processing order of "prolog, metadata, data, epilog"?
5) do you have practical example?
thanks.
-
- Regular Participant
- Posts: 180
- Joined: Thu Jul 01, 2010 3:06 am
- OLAP Product: Cognos Express
- Version: 9.5
- Excel Version: 2007
- Location: Melbourne, Australia
Re: difference between prolog, metadata, data, epilog
1) data source definition is quite self explanatory. It is a data that you obtain from somewhere eg CSV file, or using SQL statement to get the data from sql server.
2) prolog (before) - you can use prolog to create cube and dimension eg CubeCreate or DimensionCreate.
metadata (during) - mainly populating the data to the dimension. It will actually goes through every single record from the datasource and write the data to the dimension.
data(during) - mainly writing data onto the cube.
epilog(after) - anything that you want to do after the process is completed, eg run a batch file, save the data on to the hard disk.
3) prolog, metadata and data, epilog are in sequence.
2) prolog (before) - you can use prolog to create cube and dimension eg CubeCreate or DimensionCreate.
metadata (during) - mainly populating the data to the dimension. It will actually goes through every single record from the datasource and write the data to the dimension.
data(during) - mainly writing data onto the cube.
epilog(after) - anything that you want to do after the process is completed, eg run a batch file, save the data on to the hard disk.
3) prolog, metadata and data, epilog are in sequence.
-
- Posts: 23
- Joined: Wed Sep 21, 2011 6:18 am
- OLAP Product: TM1
- Version: 9.4.1
- Excel Version: 2003
Re: difference between prolog, metadata, data, epilog
then, metadata and data is quite similar. do you mean that the process will run coding in metadata first, and coding in data?
-
- MVP
- Posts: 3664
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: difference between prolog, metadata, data, epilog
Metadata and data tabs are only similar in that all code on these tabs is an implicit loop and is repeated for each record in the data source. The metadata tab is (as the name implies) for updating dimensions. At the conclusion of metadata processing, that is after the last record of the source file is read and the connection to the source is closed then and only then are changes to the dimension(s) that have been updated saved.plkkw wrote:then, metadata and data is quite similar. do you mean that the process will run coding in metadata first, and coding in data?
If there are data updates only then there should be no code on the metadata tab, and vice versa if you only want to update dimensions. Otherwise you process the data source an additional time unnecessarily.
-
- Posts: 73
- Joined: Fri Apr 23, 2010 11:35 am
- OLAP Product: Tm1
- Version: 9.5
- Excel Version: 2007 2010
Re: difference between prolog, metadata, data, epilog
In my head, I'd add these 'explanations' to help newbies like me understand how it works.
The prolog tab hasn't seen the data yet. So it's good for operations like 'Create Dimension' , 'Delete all elements in a Dimension' , add in a hardcoded element to the dimension , such as 'ALL NL codes'.
The metadata tab, will run through the data source and it's here you can add elements to a dimension based on the data. For example if your data source has an NL code which is not already in the cube you can add it here . I guess (clarification from an expert would be nice), that you need to add it here, so that data can be loaded against it.
The data tab again loops over all the data. Primarily you are loading 'fact' data here in BI terms.
The epilog is for tidying up afterwards, I haven't used it for much other than writing out logs, but I am sure someone can add in a decent example!
Hopefully I've got this right , and if not it's a useful post to be picked apart to share the knowledge!
P
The prolog tab hasn't seen the data yet. So it's good for operations like 'Create Dimension' , 'Delete all elements in a Dimension' , add in a hardcoded element to the dimension , such as 'ALL NL codes'.
The metadata tab, will run through the data source and it's here you can add elements to a dimension based on the data. For example if your data source has an NL code which is not already in the cube you can add it here . I guess (clarification from an expert would be nice), that you need to add it here, so that data can be loaded against it.
The data tab again loops over all the data. Primarily you are loading 'fact' data here in BI terms.
The epilog is for tidying up afterwards, I haven't used it for much other than writing out logs, but I am sure someone can add in a decent example!
Hopefully I've got this right , and if not it's a useful post to be picked apart to share the knowledge!
P
“The way I see it, you should live everyday like its your birthday”
TM1 9.5 Cognos BI 8.4 Excel 2007. 128GB Ram. E7450@2.40Ghz -24 core. Fluffy white dog.
TM1 9.5 Cognos BI 8.4 Excel 2007. 128GB Ram. E7450@2.40Ghz -24 core. Fluffy white dog.
-
- Regular Participant
- Posts: 357
- Joined: Sat Jun 08, 2019 9:55 am
- OLAP Product: Planning Analytics
- Version: Planning Analytics 2.0
- Excel Version: Excel 2016
Re: difference between prolog, metadata, data, epilog
Hi,
I was told that in some situations processes needed to be split up because certain things created in one process would not be recognised until that process had fully run and therefore a second process would be required to fully complete the tasks. For example dimension builds.
Is there any definitive list which details what is and isn't recognised on the tabs? And when processes need to be split up?
regards, Maren
I was told that in some situations processes needed to be split up because certain things created in one process would not be recognised until that process had fully run and therefore a second process would be required to fully complete the tasks. For example dimension builds.
Is there any definitive list which details what is and isn't recognised on the tabs? And when processes need to be split up?
regards, Maren
- orlando
- Community Contributor
- Posts: 167
- Joined: Fri Aug 04, 2017 8:27 am
- OLAP Product: TM1
- Version: PAL 2.0.8
- Excel Version: Office 365
Re: difference between prolog, metadata, data, epilog
hi maren,
maybe you should read the documentation - this will give you a good overview of what TM1 can do and how it works.
https://www.ibm.com/support/knowledgece ... -gen2.html
https://www.ibm.com/support/knowledgece ... -gen2.html
https://www.ibm.com/support/knowledgece ... -gen5.html
https://www.ibm.com/support/knowledgece ... -gen6.html
Even if this links are for TM1 10.2.2 and not for PA 2.0 - it'll help.
Best regards
orlando
-
- Regular Participant
- Posts: 357
- Joined: Sat Jun 08, 2019 9:55 am
- OLAP Product: Planning Analytics
- Version: Planning Analytics 2.0
- Excel Version: Excel 2016
Re: difference between prolog, metadata, data, epilog
Hi Orlando,
How do you know that I haven't read the documentation? Maybe I missed the bit about when you needed to split up processes!
But I take it you don't know and hopefully someone else can provide something useful!
How do you know that I haven't read the documentation? Maybe I missed the bit about when you needed to split up processes!
But I take it you don't know and hopefully someone else can provide something useful!
-
- Posts: 139
- Joined: Tue May 21, 2019 3:33 pm
- OLAP Product: TM1
- Version: PA 2.0.x
- Excel Version: 2016
- Location: The Internet
Re: difference between prolog, metadata, data, epilog
Hi,
Splitting processes is more a "design technique" instead of a necessity.
I suggest you this white paper on the best practice to develop TM1 Turbo Integrator processes
https://static1.squarespace.com/static/ ... grator.pdf
Also, a good example of splitting process is the Bedrock library (https://github.com/cubewise-code/bedrock). For example, instead of creating a dimension directly in your TI, you can easily use an existing process from Bedrock (Bedrock.Dim.Create.pro).
Enjoy !
Splitting processes is more a "design technique" instead of a necessity.
I suggest you this white paper on the best practice to develop TM1 Turbo Integrator processes
https://static1.squarespace.com/static/ ... grator.pdf
Also, a good example of splitting process is the Bedrock library (https://github.com/cubewise-code/bedrock). For example, instead of creating a dimension directly in your TI, you can easily use an existing process from Bedrock (Bedrock.Dim.Create.pro).
Enjoy !
Last edited by Emixam on Wed Jul 03, 2019 1:22 pm, edited 2 times in total.
-
- MVP
- Posts: 2832
- Joined: Tue Feb 16, 2010 2:39 pm
- OLAP Product: TM1, Palo
- Version: Beginning of time thru 10.2
- Excel Version: 2003-2007-2010-2013
- Location: Atlanta, GA
- Contact:
Re: difference between prolog, metadata, data, epilog
Maybe because there isn't anything in the documentation about splitting up processes and maybe because I've never heard of any such requirement. You can most certainly do dimension builds in one process. There is no technical requirement to break it up. Yes, there may be times, because of certain conditions, where you may split a dimension build into different processes. An example would be a process to unwind the existing hierarchy, another to add new leaf nodes and then another to re-build the hierarchy. However, this would be at the developers discretion, maybe because of code sharing or source data limitations, and not because TM1 forces you to. Maybe you should go back to the person who told you processes need to be split and ask them what they are talking about.
-
- Regular Participant
- Posts: 357
- Joined: Sat Jun 08, 2019 9:55 am
- OLAP Product: Planning Analytics
- Version: Planning Analytics 2.0
- Excel Version: Excel 2016
Re: difference between prolog, metadata, data, epilog
Hi,
ok thanks for the advice, something must have got lost in translation somewhere along the line!
Incidentally I didn't bring up the guides, Orlando did!
regards, Maren
ok thanks for the advice, something must have got lost in translation somewhere along the line!
Incidentally I didn't bring up the guides, Orlando did!
regards, Maren
-
- MVP
- Posts: 3126
- Joined: Mon Dec 29, 2008 6:26 pm
- OLAP Product: TM1, Jedox
- Version: PAL 2.0.9.18
- Excel Version: Microsoft 365
- Location: Brussels, Belgium
- Contact:
Re: difference between prolog, metadata, data, epilog
For example in the prolog tab you can add an element for allocations. In the Data tab you can process data on the element and in the Epilog tab you cannot delete the element. Therefore you need to have the Prolog tab of a second process to do it.
Best regards,
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
-
- MVP
- Posts: 3664
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: difference between prolog, metadata, data, epilog
Love a bit of thread necromancy.
Well you should be able to and in most cases you can but there have been cases of versions with bugs which would crash the server when the source of a process was deleted on the epilog of the same process. (In the case of deleting a cube. In Wim's example you certainly can delete an element on the epilog with the DeleteDirect function, but admittedly you haven't always been able to do this.)Wim Gielis wrote: ↑Wed Jul 03, 2019 6:19 pm For example in the prolog tab you can add an element for allocations. In the Data tab you can process data on the element and in the Epilog tab you cannot delete the element. Therefore you need to have the Prolog tab of a second process to do it.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
-
- Posts: 110
- Joined: Wed May 20, 2009 7:30 am
- OLAP Product: TM1
- Version: 10.2.2 - PA
- Excel Version: 2010
- Location: Rennes, France
Re: difference between prolog, metadata, data, epilog
I had to split a process in 2 steps several times. But I have just one exemple in my mind at the moment.
This concerns security management.
In order to be sure not to lose private objects, here is the way I manage groups/clients in two processes:
Process 1
1. Prolog
-I delete the links between clients and groups
-I delete all groups (except admin)
-I define the source where I can read all goups and clients relevant for my instance
2. Metadata
-I add groups
-I add clients
3. Data
-I assign clients to groups
4. Epilog
Nothing because the last step is to delete clients which are not related to any group but I CAN NOT use DeleteClient function in an epilog
Process 2
I delete all clients which are not belonging to any group (using DeleteClient function)
This concerns security management.
In order to be sure not to lose private objects, here is the way I manage groups/clients in two processes:
Process 1
1. Prolog
-I delete the links between clients and groups
-I delete all groups (except admin)
-I define the source where I can read all goups and clients relevant for my instance
2. Metadata
-I add groups
-I add clients
3. Data
-I assign clients to groups
4. Epilog
Nothing because the last step is to delete clients which are not related to any group but I CAN NOT use DeleteClient function in an epilog
Process 2
I delete all clients which are not belonging to any group (using DeleteClient function)