Run Process - How to Determine When All Completed.

[chewza]

Hi there

I am having the same issue you mentioned that someone put in a previous posts as per below. I am also finding this a bit of a show-stopper. Does anyone have any ideas of how to get around this?

Many thx!

Regards
Chris


The biggest problem with this new command is there is no way to know when all of them are done so that you can do the next sequential action. For example, after 12 months are loaded, how do you then kick of an allocation script, or a email saying everything is fine. You have to end up jumping through some synchronization hoops (i.e. updating a cube) to get this all to work properly. What happens when one fails? How do you stop subsequent processes based on a condition? How do you control the size of the queue to keep it within CPU limits? etc. etc.

In my opinion, it really isn't any better than TM1RunTI, other than it taking up a single connection.

We ended up having to write our own external command lines to really do what you need to for parallel processing. Hopefully IBM will add other control functions to make this truly usable outside of simplistic use cases.

[JohnO]

The answer is that there are no simple answers. The only orchestration tool within the standard product is cognos command center and it won't explicty help you with this.

For queue management there is a technique using synchronized function. Described in a cubewise article from memory. Lets say you have 4 cores you can use but 20 runprocess to issue. You can issue all 20 run process commands and limit the synchronized paramter to 4 different names across the 20. That way you will only have 4 running at a time. Should work fine if the duration of each is close to the average.

How do we know when it finishes? In the above scenario it is a matter of determing when there is nothing remaining of the 4.. My experience is that the TI calling runprocess may finish yet the called processes may still be running. I resorted to flag files to determine when the overall procees could be considered finished. Basically you need to stop the calling process from finishing until all runprocess are finished. A few o/s call and loops etc and asssumptions about how long you should wait.

My biggest problem with run process was locking, both between these processes and with other processes. I found that my biggest problem was everything becoming serial. So rather than say 20 processes running in 4 minutes it would all go back to 80 minutes.

Lots of challenges. There is a fundamental lack of orchestration across TI's and runprocess is an extra set of challenges with that.

[Steve Rowe]

As JohnO said the only way I know to do this is counting Flag files. Something like this.

Main Job
Clear your Flag folder of all files.

Figure out how many RunProcess calls you are are going to make = nFlagCount

Call all your Run Process jobs

Loop Until FlagCount>=nFlagCount
FlagCount is the sum of count of the files in the Flag Folder plus any new TM1ProcessError files for the Run Process Job. Use WildCardFileSearch plus Sleep.

Call any further jobs as required.
For bonues point process Flag files to update record keeping
End Main Job

RunProcess Job
In the epilog write out your Flag file. I actually put a load of record keeping in here, start / finish time, row counts, error count etc
End RunProcess Job

For locking your RunProcess job should not include any object creation (cubes, dimensions, elements, views and sets), watch out if you are using Asciiouput that you make sure the file names are unique (add the thread ID).

There maybe some use cases where creating objects in the parallel piece is OK but I just find it easier to avoid it entirely.

HTH

[JohnO]

I just found this relating to the new PA engine:

https://www.ibm.com/docs/en/planning-an ... -analytics

Have a read of the improvements to TI's

[Steve Rowe]

Nice spot JohnO, looks like they will provided a step change in our ability manage parallel processing.

[John Hammond]

So the TM1 Servers move to the cloud as a service for IBM managed clouds (Openshift I assume).

This will be the same black boxing as the PAW HTTP and Webservices and mean traditional access to the servers disk objects is gone without admin access to the cloud platform. All good until something breaks.

Developers kind of return to the old MVS/CICS world. It's a big rapid shift.

[MarenC]

Hi,

I stopped using RunProcess in production because the results were too erratic in development!

As for the new PA server, I think there are some pretty big show stoppers that would prevent many organisations moving to it, though off the top of my head I can't remember what they are?

So I am expecting hell to freeze over before we can utilise those benefits.

Maren

[Steve Rowe]

What kind of erraticness were you hitting with RunProcess MarenC?

I've a MT job running hourly that I am not seeing issues with, at least the customer has not raised them to me.

[MarenC]

Hi Steve,

Some issues were described here within the comments thread:

https://www.tm1forum.com/viewtopic.php?t=14841

Given the erratic nature I experienced, I basically gave up on RunProcess as I could have no definite confidence in it.

This is maybe something I will return to, as I know it can save hours of time.

Maren

[chewza]

Seem to be close to getting this working. (Just need to code/conquer one final part as per below). Used a SQL table to determine when all processes have completed, and thereby control next sequential step

Master Process (using 2 cores to run in parallel as an example)
Prolog:
RUNPROCESS(sProcess, sPara1, 'pLockObj', '1','pBatchNo', 1, 'pBatchNoTotalExpected', 4);
RUNPROCESS(sProcess, sPara2, 'pLockObj', '2','pBatchNo', 2, 'pBatchNoTotalExpected', 4);
RUNPROCESS(sProcess, sPara3, 'pLockObj', '1','pBatchNo', 3, 'pBatchNoTotalExpected', 4);
RUNPROCESS(sProcess, sPara4, 'pLockObj', '2','pBatchNo', 4, 'pBatchNoTotalExpected', 4);

Check out my Slave process further down at this point

Still to complete:
Set something up to query the SQL table to determine when all complete and call the Process to be run after all is complete(what this is all about). Still need to code and test this part, but here is my plan (Anyone got any better ideas for this part?):
H

• • ave While loop running say every 30 seconds for 10 minutes (a maximum amount of time this would take in the worst scenario)
    Call a second slave process that queries SQL table where query only returns a single record for situation where max batchno = BatchNoTotalExpected
    In the data section of this slave process I can then be in a position to execute the process that I want to run once all the RunProcess threads are completed (the goal behind this whole challenge)
    Clear records in flag control process
    Finally now be in situation to call the Ti I need to call after everything has run.

Have while loop to check "Flag_Control_ParallelRun" tble (see below) every (say) 30 seconds
When max batch number in table hits total expected number of batches, you know that everything in complete


Slave Process
Prolog:
IF (pLockObj @<>'');
SYNCHRONIZED (pLockObj);
ENDIF;

Data:
Do Some stuff

Epilog:
Use a sQl server table - e.g. (Flag_Control_ParallelRun)
Have field for batch no and total batch expected (and other useful fields - eg end date)
Insert record in epilog and include batch no and total batch expected

[chewza]

Seem to be close to getting this working. (Just need to code/conquer one final part as per below). Used a SQL table to determine when all processes have completed, and thereby control next sequential step

Master Process (using 2 cores to run in parallel as an example)
Prolog:
RUNPROCESS(sProcess, sPara1, 'pLockObj', '1','pBatchNo', 1, 'pBatchNoTotalExpected', 4);
RUNPROCESS(sProcess, sPara2, 'pLockObj', '2','pBatchNo', 2, 'pBatchNoTotalExpected', 4);
RUNPROCESS(sProcess, sPara3, 'pLockObj', '1','pBatchNo', 3, 'pBatchNoTotalExpected', 4);
RUNPROCESS(sProcess, sPara4, 'pLockObj', '2','pBatchNo', 4, 'pBatchNoTotalExpected', 4);

Check out my Slave process further down at this point

Still to complete:
Set something up to query the SQL table to determine when all complete and call the Process to be run after all is complete(what this is all about). Still need to code and test this part, but here is my plan (Anyone got any better ideas for this part?):
H

• • ave aWhile loop running say every 30 seconds for 10 minutes (a maximum amount of time this would take in the worst scenario)
    Call a second slave process that queries SQL table where query only returns a single record for situation where max batchno = BatchNoTotalExpected
    In the data section of this slave process I can then be in a position to execute the process that I want to run once all the RunProcess threads are completed (the goal behind this whole challenge)
    Clear records in flag control process
    Finally now be in situation to call the Ti I need to call after everything has run.

Have while loop to check "Flag_Control_ParallelRun" tble (see below) every (say) 30 seconds
When max batch number in table hits total expected number of batches, you know that everything in complete


Slave Process
Prolog:
IF (pLockObj @<>'');
SYNCHRONIZED (pLockObj);
ENDIF;

Data:
Do Some stuff

Epilog:
Use a sQl server table - e.g. (Flag_Control_ParallelRun)
Have field for batch no and total batch expected (and other useful fields - eg end date)
Insert record in epilog and include batch no and total batch expected

[PavoGa]

This can be accomplished with SYNCHRONIZE.

Say you have two groups of processes to execute. The second group should not execute until after the first group has all finished. For each RunProcess of the first group, SYCHRONIZE a distinct object (remember, it is just a token and does not have to be an object in the database) in each call. For example Larry, Curly and Moe are each passed, respectively, to three instances of RunProcess.

Pass Larry:Curly:Moe to a wrapper process of the second group of processes. Parse that parameter value and SYNCHRONIZE Larry, Curly and Moe in succession in that wrapper process. It will not execute its batch of second group processes until locks are released on all three of Larry, Curly and Moe.

I've used a form of this to run up to three batches of processes, each batch needing to complete before the next batch executes. The users are turned loose in a few seconds avoiding looking at a stuck screen while the processes run in the background.

[Wim Gielis]

With RunProcess I observe huge increases in memory.
Also, Synchronized when used in PAW does not seem to make the parent process wait for all child processes to finish.
Not sure if others see this behavior as well ?

[chewza]

Hi Guys

Thanks everyone for you feedback.

Just a question for you. Say for example, the server has 8 cores, and your process has 3 RunProcess commands. So this will utilise 3 cores.
Given your server has 8, this seems like it should not be an issue. However, what if you have a whole lot of chores activate at the same time - e.g. 6 chores kick of within a time span of 3 minutes. Will these try and use a core each, bringing the core count to 9 which exceeds the server maximum?

Not sure if this scenario can play out and if there is a potential risk?

[PavoGa]

With RunProcess I observe huge increases in memory.
Also, Synchronized when used in PAW does not seem to make the parent process wait for all child processes to finish.
Not sure if others see this behavior as well ?

I would not expect SYNCHRONIZE to make the parent process wait for the child processes and that behavior is pre-PAW as well. What is the usage context?

[Wim Gielis]

With RunProcess I observe huge increases in memory.
Also, Synchronized when used in PAW does not seem to make the parent process wait for all child processes to finish.
Not sure if others see this behavior as well ?

I would not expect SYNCHRONIZE to make the parent process wait for the child processes and that behavior is pre-PAW as well. What is the usage context?

The context is that a user clicks on a button in PAW. It kicks off the mother process.
The mother process should wait for the child processes to finish.
If not, the user sees the toaster message that the fired process is ready (very quickly) yet in fact,
the real is still about to begin in the background...

[PavoGa]

With RunProcess I observe huge increases in memory.
Also, Synchronized when used in PAW does not seem to make the parent process wait for all child processes to finish.
Not sure if others see this behavior as well ?

I would not expect SYNCHRONIZE to make the parent process wait for the child processes and that behavior is pre-PAW as well. What is the usage context?

The context is that a user clicks on a button in PAW. It kicks off the mother process.
The mother process should wait for the child processes to finish.
If not, the user sees the toaster message that the fired process is ready (very quickly) yet in fact,
the real is still about to begin in the background...

Yes, but you wouldn't use SYNCHRONIZE for controlling the parent process to wait for the child to finish.

[Wim Gielis]

I would not expect SYNCHRONIZE to make the parent process wait for the child processes and that behavior is pre-PAW as well. What is the usage context?

The context is that a user clicks on a button in PAW. It kicks off the mother process.
The mother process should wait for the child processes to finish.
If not, the user sees the toaster message that the fired process is ready (very quickly) yet in fact,
the real is still about to begin in the background...

Yes, but you wouldn't use SYNCHRONIZE for controlling the parent process to wait for the child to finish.

Any other 'easy' method as an alternative ? File semaphores, or other ?

[PavoGa]

File semaphores is what I use currently to control a parent process. I believe I read about using adding/removing attributes or elements and tested it a few years back, but never used in production. The file semaphores seem to work well enough.

Wasn't there a post the other day about some additional variables available for use with RunProcess? I searched but have not found it.

[Wim Gielis]

Wasn't there a post the other day about some additional variables available for use with RunProcess? I searched but have not found it.

That doesn't ring a bell unfortunately.