NoWait command for TI processes

[lotsaram]

Even with all the enhancements made to the locking model over the last few versions no one would say it is easy to run jobs in parallel TM1 to better use available cores and generate faster application response time. Currently we are limited to 3 options
1/ schedule chores to run concurrently
2/ use an external 3rd party scheduling tool to call tm1runti to run processes in parallel via additional logins
3/ Use ExecuteCommand combined with batch files and tm1runti to run processes in parallel via additional logins

I suspect option 3 is the most common.

BUT adding a NoWait command option to TI script would be much, much simpler. Imagine if a no wait command gave the ability to call TI processes in parallel using ExecuteProcess and extra worker threads were added without needing a full additional external login to generate the extra thread. How simple would life be?

To run a load process in parallel all you would need to do is this

NoWait;
ExecuteProcess( sProc, 'pCube', 'SalesOrders', 'pFile', cLoadDir | 'File1.txt' );
ExecuteProcess( sProc, 'pCube', 'SalesOrders', 'pFile', cLoadDir | 'File2.txt' );
ExecuteProcess( sProc, 'pCube', 'SalesOrders', 'pFile', cLoadDir | 'File3.txt' );
ExecuteProcess( sProc, 'pCube', 'SalesOrders', 'pFile', cLoadDir | 'File4.txt' );

Does anyone else share my enthusiasm for this idea?

[Paul Segal]

Yes, that would be very useful.

[Duncan P]

I really like this.

However a couple questions need to be asked about expected behaviour in this scenario.

• Which of the uncommitted changes in the current process can the called "no wait" process see? Are we happy with "none"?
• What happens about global variables? Again the obvious answer is "no visibility".
• Do we anticipate any requirement for synchronisation for clean-up? If so how might that work? We would need to be very careful of deadlocks in this case.

[TableManagerOne]

Rather than diving into general asynchronous programming, would a split/join in the scheduler (the chore) suffice? A multi-commit chore could be extended to branch (\) and merge (/) arbitrarily. At any given branch point, the called processes have the potential to run in parallel, as new threads running in the context of the parent process. However, if a lock conflict/deadlock is encountered, the sub-process rolls back (but only to the branch point) and waits for its siblings to complete. The chore-branch waits for all children to complete one way or another before continuing on to call the next process/set of branching processes, or merge itself back into whichever stream it branched from.

P1, P2, -------------------, P7
____\ P3a /
____\ P3b------------ /
______\ P4a, P5a /
______\ P4b, P5a /

Each branch and merge includes a commit, making changes of the branching stream visible to spawned streams, and changes of the spawned stream visible to the branching steam (after all spawned streams commit.)

Not as powerful of course, but perhaps easier to wrap one's head around (mine at least.)

[lotsaram]

• Which of the uncommitted changes in the current process can the called "no wait" process see? Are we happy with "none"?

Given the current way of achieving parallelization is with independent logins and therefore each processes uncommitted changes aren't visible to any other thread I think this would be just fine.

• What happens about global variables? Again the obvious answer is "no visibility".

No visibility also fine. From a practical perspective any global variables that were needed could be passed to processes as parameters anyway.

• Do we anticipate any requirement for synchronisation for clean-up? If so how might that work? We would need to be very careful of deadlocks in this case.

To me the mechanism used be parallel interaction seems just fine, if there do happen to be any overlapping data writes then it is last commit wins. i.e. programmer beware, but in the scenarios where people would want to use this by rights some care should be taken to make sure that data sources and targets don't overlap

[rmackenzie]

• Which of the uncommitted changes in the current process can the called "no wait" process see? Are we happy with "none"?

Given the current way of achieving parallelization is with independent logins and therefore each processes uncommitted changes aren't visible to any other thread I think this would be just fine.

• What happens about global variables? Again the obvious answer is "no visibility".

No visibility also fine. From a practical perspective any global variables that were needed could be passed to processes as parameters anyway.

• Do we anticipate any requirement for synchronisation for clean-up? If so how might that work? We would need to be very careful of deadlocks in this case.

To me the mechanism used be parallel interaction seems just fine, if there do happen to be any overlapping data writes then it is last commit wins. i.e. programmer beware, but in the scenarios where people would want to use this by rights some care should be taken to make sure that data sources and targets don't overlap

Sounds good - just a simple command to automate the effects of option 2 or 3 by the sounds of it. This would be very useful.

[jstrygner]

Agree this would be very handy and I would love to have such an option.

I think the major difficulty in implementing it would be with running more than one action under the same thread ID.
The TM1RunTI and Chores will have own threads.
Probably running more than one TI within same login session would require serious change in the TM1 engine.
Question would be if such NoWait; could kind of clone login session and run parallel TIs under those clone threads (same user login, different thread ID). After that it should be good to handle closing those clones as well.

[lotsaram]

Agree this would be very handy and I would love to have such an option.

I think the major difficulty in implementing it would be with running more than one action under the same thread ID.

I figure if they can spawn new threads for running view calculation in parallel then it must be possible to do the same from a TI. I just think this would be a great feature to have, so much easier than external calls to TM1RunTI.