TM1 Forum

Hi Experts,

Can some one please guide, how to execute multiple TI processes parallely from TM1RUNTI configuration file? I am able to execute one TI process from both command prompt and using a configuration file. But how to specify multiple process names in the configuration files? How to ensure they run parallely - using separate cores of the server. Pasting the sample configuration file for ready reference -

~~~~~ Configuration file content ~~~~~
[TM1RunTI]
process=TEST_RunTI
connect=DEV

[Connect - DEV]
adminhost=localhost
Server="planning sample"
user=admin
pwd=apple
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I am saving this file with the name TI.ini and calling this file from commant prompt as :

C:\Program Files\IBM\cognos\tm1_64\bin64>tm1runti -i "C:\Users\IBM_ADMIN\Desktop\TI.ini"

And this way its working fine. However when I am trying to use multiple process names such as - process=process1, process2 -- it does not work. I have tried other seprator as well - such as semicolon, putting double quote and comma combination - nothing works.

If somebody can share a sample configuration file content - with multiple TI process, it will be great. Some education on - how to leverage separate cores for different process also will be much appreciated.
Thanks.

N. Laha.

To my knowledge you don't call multiple processes using a single a config file nor can you call multiple processes on any single command line execution. To execute multiple TIs you have to call tm1runti once for each process you want to execute.

Thanks "PlanningDEV" for your response.
If I read the TM1RunTI guide - it says - multiple TI process can be called from a single configuration file. Let me paste the section of TM1runti guide here, for ready reference -

The configuration file contains:
1. A single TM1RunTI section.
2. One or more sections defining the TurboIntegrator processes that may be run.
3. Zero or more sections defining connection parameters.

Attaching the TM1runTI guide too.

So its apparent that, its doable. I am just wondering - how to write the configuration for such cases.
Calling multiple configuration file is another option - but that will definitely make the processes running in sequence, one after another - where my ultimate intent will be defeated. My ultimate intent is to run multiple TI processes simultaneously.

If anyone can suggest, it will be of great help.

My interpretation of the docs would lead me to believe it should be done like this:
Did you try this?

What I meant was that you can specify multiple TI processes in a config file by that's just the settings for that TI so you don't have to put them on the command line. You use the same config file for each command line execution but each command line can only call a single TI at a time.

PlanningDev wrote:What I meant was that you can specify multiple TI processes in a config file by that's just the settings for that TI so you don't have to put them on the command line. You use the same config file for each command line execution but each command line can only call a single TI at a time.

My last post may have been a little confusing. A single command line can execute multiple TI's, just NOT at the same time. They would execute in serial if you tried to perform this using the config file. According to your post, you are trying to achieve parallel execution. In order to achieve this you need to call each TI separately (using a new command line). You use the same config file for each command line, however you would need to specify the -process parameter on the command line for each call. Depending on whether or not you are calling the same TI or multiple TI's you may also have to specify the parameters for each TI on the command line. Different TI's could be configured in the file. A single TI using parameters would need them on the command line.

Does that make more sense? There isn't a switch to force the process listed in the [TM1RunTI] section to run in parallel. Its achieved through multiple command line executions.


Example to execute multiple TI's from a single command line call (Will run in SERIAL execution).
[TM1RunTI]
process=my_ti_process
process=my_ti_process2
connect=Production

[Process - my_ti_process]
num1="value1"
stringX="value2"
stringY="value3"

[Process - my_ti_process2]
num1="value1"
stringX="value2"
stringY="value3"

[Connect – Production]
adminhost=
server=MyTM1server
user="MyTM1AdminServer"
pwdfile="c:\tm1_admin_area\passwords\tm1_password.txt"
AdminSvrSSLCertAuthority=.\ssl\applixca.pem
AdminSvrSSLCertID=tm1adminserver
AdminSvrSSLCertRevList=
CAMNamespace=LOCAL_NTLM

You could use the Windows batch file keyword START to run TIs in parallel.

Thanks to all of you for your posts. Let me reply to all of one by one.
@Tomok : When in configuration file, [TM1RUNTI] section contains multiple process names, it executes only the first one, and ignores the rest. For your example -

[TM1RunTI]
process=TEST_RunTI_1
process=TEST_RunTI_2
process=TEST_RunTI_3
...

-- in such configuration file, only TEST_RunTI_1 is executed and rest other processes are ignored.

@PlanningDEV : Thanks for the detailed explanation. Understood. Yes, my ultimate intent is to run them in parallel, and you are saying there is no switch in the configuration file. So, if I need to call multiple processes from multi command prompt anyway to make them run in parallel, I dont mind to write multiple configuration files also for multiple processes. But it could have been better if multiple processes could have been called from a same configuration file.

@ David : Thanks for your suggestion.

So let me try to call them from 2 separate commnd lines to make 2 processes parallel. By mean time if somebody could try to call multiple processes from same configuration file - please post.

Hi planningDev,

I created two different configuration files, for two different independent TI processes. Opened two command prompt - executed them together, followed by monitoring them from TM1Top. Both the processes started executing - but at any point of time only one progresses on and another goes to WAIT mode. So ultimately intent of making them parallel execution, really didn't work.
Any idea how to make them run parallely using separate server resource?

I dont think you need 2 config files here. You can specify the process sections [process - process1], [process - process2] in the config file and instead of putting the process= under the [TM1RunTi] section you would specify it on the command line. This will eliminate the need for multiple files.

Now for the tricker part. If you are running 2 command line executions and seeing to TI process calls in TM1TOP but they are getting serialized then you have a contention issue. There are multiple causes for this however the big ones are...

1. Any dimensional update in the TI in any section of the TI would cause a lock and would block the other process from parallel write.
2. Any crossing of cells that you are attempting to write to. IE if you have one TI writing to Jan, Feb and the other attempting to write to Feb, Mar, then it would serialize.

Can you confirm if you have any of the 2 above issues?

Hey PlanningDev : You are the man! yes, I was trying with two Dim build processes, and they were being serialized. However, when I tried two Data load processes, which loads data to two separate cubes - they ran parallely. one didnt need to wait for another. Each of this process takes 25 mins to execute. Now if they are serialzed, they take good 50 mins to complete. Now being parallel, both of them are getting finished in approx 30 mins together. So this trick worked. Thanks a lot. I owe you a beer for sure.

nilmadhab.laha wrote:...when I tried two Data load processes, which loads data to two separate cubes - they ran parallely. one didnt need to wait for another. Each of this process takes 25 mins to execute. Now if they are serialzed, they take good 50 mins to complete. Now being parallel, both of them are getting finished in approx 30 mins together.

Using parallel interaction you can probably do better. Depending on the number of cores you have available you should be able to additionally split each cube load and run those in parallel as well and get further gains only limited by the number of cores.

I would question why you have several TI processes that each take 25+ minutes to execute. Perhaps there are tweaks you could do to increase the performance. In a well designed model, with average network performance (assuming you are going against an ODBC data source), you should be able to load a MASSIVE amount of data in 25 minutes. I'm talking well in excess of 100GB. What is causing it to be so slow? Is your bandwidth between the TM1 server and the source adequate? Do you have a lot of feeders on the cube(s) being loaded that have to evaluated after each cell update? What kind of metadata updates are being done, if any? There are things you can do to optimize this stuff.

Hi Lotsa,

Parallel interaction is already enable in my tm1s.cfg file and I have 8 cores in the server. Can you please guide, how to make sure - when two processes are running in parallel - they take 3 cores each or one takes 4 cores and another takes 3 cores?

Tom,

yes datasource is ODBC. There is no feeder in the cube, in fact there is no rule at all. There are certain calculations are done in the process before putting data into the cube - which is also not too many. bandwidth between the TM1 server and DB server is good - 1GBPS backbone. tnsping from TM1 server gets resolved within average 60-90 mili seconds. Cube having 5 dimensions. longest dimension has approx 14000 element. Not sure why its taking so long. I can sense from your post that it should take lesser time than this - but dont know where else it can be improved. its a very very straight forwards cube - with no rules, with not too many dimensions, entire cube size is within 30 MB. Any suggestion is much appreciated.

nilmadhab.laha wrote: Tom,

yes datasource is ODBC. There is no feeder in the cube, in fact there is no rule at all. There are certain calculations are done in the process before putting data into the cube - which is also not too many. bandwidth between the TM1 server and DB server is good - 1GBPS backbone. tnsping from TM1 server gets resolved within average 60-90 mili seconds. Cube having 5 dimensions. longest dimension has approx 14000 element. Not sure why its taking so long. I can sense from your post that it should take lesser time than this - but dont know where else it can be improved. its a very very straight forwards cube - with no rules, with not too many dimensions, entire cube size is within 30 MB. Any suggestion is much appreciated.

If you provide the code within the TIs I am sure someone will point you in the right direction.

I am afraid, I wont be able to share the code. However, thanks for all your posts. I could achieve the stuff I wanted with TM1runTI utility.

PlanningDev wrote:I dont think you need 2 config files here. You can specify the process sections [process - process1], [process - process2] in the config file and instead of putting the process= under the [TM1RunTi] section you would specify it on the command line. This will eliminate the need for multiple files.
Now for the tricker part. If you are running 2 command line executions and seeing to TI process calls in TM1TOP but they are getting serialized then you have a contention issue. There are multiple causes for this however the big ones are...

1. Any dimensional update in the TI in any section of the TI would cause a lock and would block the other process from parallel write.
2. Any crossing of cells that you are attempting to write to. IE if you have one TI writing to Jan, Feb and the other attempting to write to Feb, Mar, then it would serialize.

Hi PlanningDev, thanks for the explanation. This is very helpful. just to clarify on the 2nd point above. If both Ti are writing or laoding data to two different cubes with same month dimension, then also it would cause serialization of TI? Could you please confirm?

rksharmatm1 wrote:

PlanningDev wrote:I dont think you need 2 config files here. You can specify the process sections [process - process1], [process - process2] in the config file and instead of putting the process= under the [TM1RunTi] section you would specify it on the command line. This will eliminate the need for multiple files.
Now for the tricker part. If you are running 2 command line executions and seeing to TI process calls in TM1TOP but they are getting serialized then you have a contention issue. There are multiple causes for this however the big ones are...

1. Any dimensional update in the TI in any section of the TI would cause a lock and would block the other process from parallel write.
2. Any crossing of cells that you are attempting to write to. IE if you have one TI writing to Jan, Feb and the other attempting to write to Feb, Mar, then it would serialize.

Hi PlanningDev, thanks for the explanation. This is very helpful. just to clarify on the 2nd point above. If both Ti are writing or laoding data to two different cubes with same month dimension, then also it would cause serialization of TI? Could you please confirm?

If you are purely writing data then you can parallel load multiple cubes even if they share dimensions. With Parallel Interaction turned on the blocking is mainly limited to metadata changes (dimensional updates) and security changes. (There are other things that can cause locking but those are the big ones to look our for.

@ rksharmatm1 : PlanningDev has already answered your question - still just wanted to put my 2 cents here. I tried data loading to two separate cubes - by two separate TI processes - both cubes are having shared dimensions - still both processes went parallel. So, it works.

Just if I go another step ahead - I have tried calling same TI multiple times to load data to separate slice of the cube e.g. keeping month as parameter in the TI, 1st one to load data just to the month of Jan, same TI to load data for Feb and so on ... it works. All of them can run in parallel.

PlanningDev wrote:

rksharmatm1 wrote:

PlanningDev wrote:
If you are purely writing data then you can parallel load multiple cubes even if they share dimensions. With Parallel Interaction turned on the blocking is mainly limited to metadata changes (dimensional updates) and security changes. (There are other things that can cause locking but those are the big ones to look our for.

Thanks for your explanation. I tried to execute multiple processes ( via executecommand from a TI) loading data to different cubes with shared dimensions and both executed in parallel with no lock/reservation issues etc. This is really a great help help. Now I am planning to run some export and import daily processes in parallel, which were serialize before. This could save some good time. I will have a try. Thanks again..