RunProcess and Process Ends msg
- roei61
- Posts: 41
- Joined: Wed Aug 22, 2012 5:44 am
- OLAP Product: PAX
- Version: 2.0.7
- Excel Version: 2016
- Location: Israel
RunProcess and Process Ends msg
Hi,
I have process that being executed by users (few users can execute the process on the same time).
I'm using few runprocess commands in this process to run other processes.
The main process ends after few seconds and the user gets msg of "Process Ends". However, there are still threats that running from the runprocess command.
I want to delay the "Process Ends" msg until all the runprocesses are done also.
Any ideas?
Keep in mind that there can be also runprocesses that being executed by other users
Thanks
I have process that being executed by users (few users can execute the process on the same time).
I'm using few runprocess commands in this process to run other processes.
The main process ends after few seconds and the user gets msg of "Process Ends". However, there are still threats that running from the runprocess command.
I want to delay the "Process Ends" msg until all the runprocesses are done also.
Any ideas?
Keep in mind that there can be also runprocesses that being executed by other users
Thanks
-
- MVP
- Posts: 1815
- Joined: Mon Dec 05, 2011 11:51 am
- OLAP Product: Cognos TM1
- Version: PA2.0 and most of the old ones
- Excel Version: All of em
- Location: Manchester, United Kingdom
- Contact:
Re: RunProcess and Process Ends msg
I handle this by having the master process create a text file in a folder and pass the name of that file to the sub process (doing this for ever sub process it runs.) The last line of the sub process deletes the file.
At the end of the master process it loops the folder counting the files that exist until it finds all the files created have been deleted (indicating all sub processes have completed.) I also put a backup option in where after a set amount of time it will finish with an error if there are still files that exist (e.g if I expect it to take 20 seconds total I would maybe set a failure time of 60 seconds.)
At the end of the master process it loops the folder counting the files that exist until it finds all the files created have been deleted (indicating all sub processes have completed.) I also put a backup option in where after a set amount of time it will finish with an error if there are still files that exist (e.g if I expect it to take 20 seconds total I would maybe set a failure time of 60 seconds.)
Declan Rodger
- roei61
- Posts: 41
- Joined: Wed Aug 22, 2012 5:44 am
- OLAP Product: PAX
- Version: 2.0.7
- Excel Version: 2016
- Location: Israel
Re: RunProcess and Process Ends msg
I thought on something similar: to create an element \ view and delete in the end.
In this way I can add an attribute that counts how many processes completed since not always the last process will ends the latest.
What would you do if one of the process has error? I don't want to show "Process Completed" msg.
How can I count how many process not completed correctly?
However, I hope there is more elegant way.
In this way I can add an attribute that counts how many processes completed since not always the last process will ends the latest.
What would you do if one of the process has error? I don't want to show "Process Completed" msg.
How can I count how many process not completed correctly?
However, I hope there is more elegant way.
-
- MVP
- Posts: 3651
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: RunProcess and Process Ends msg
I second Declan's suggestion. File based semaphores are really the only way to go when running processes on independent threads. Reading a cell value (or attribute value) is vey tricky to get right because of TM1's commit model and which version of the data your monitoring process is rteading versus the separate commits of each worker thread, file based semaphores are just much easier top get right and better for one's sanity.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
- tiagoblauth
- Posts: 20
- Joined: Wed Jul 08, 2015 11:50 am
- OLAP Product: TM1, Cognos BI
- Version: 10.2.1
- Excel Version: 2010
Re: RunProcess and Process Ends msg
I used to create control cubes, so I store start date and end dates and completion. You can provide read access to users, so they can monitor it.
You can also check for errors by during the process when you run trigger processes this way:
return_value = ExecuteProcess('create_sales_cube');
return_value can be compared with these functions:
ProcessExitByChoreQuit() - indicates that the process exited due to execution of the ChoreQuit function
ProcessExitNormal() - indicates that the process executed normally
ProcessExitMinorError() - indicates that the process executed successfully but encountered minor errors
ProcessExitByQuit() - indicates that the process exited because of an explicit "quit" command
ProcessExitWithMessage() - indicates that the process exited normally, with a message written to tm1server.log
ProcessExitSeriousError() - indicates that the process exited because of a serious error
ProcessExitOnInit() - indicates that the process aborted during initialization
ProcessExitByBreak() - indicates that the process exited because it encountered a ProcessBreak function
Based on comparison you can place message on the control cube or do externa actions like send an email to support.
You can also check for errors by during the process when you run trigger processes this way:
return_value = ExecuteProcess('create_sales_cube');
return_value can be compared with these functions:
ProcessExitByChoreQuit() - indicates that the process exited due to execution of the ChoreQuit function
ProcessExitNormal() - indicates that the process executed normally
ProcessExitMinorError() - indicates that the process executed successfully but encountered minor errors
ProcessExitByQuit() - indicates that the process exited because of an explicit "quit" command
ProcessExitWithMessage() - indicates that the process exited normally, with a message written to tm1server.log
ProcessExitSeriousError() - indicates that the process exited because of a serious error
ProcessExitOnInit() - indicates that the process aborted during initialization
ProcessExitByBreak() - indicates that the process exited because it encountered a ProcessBreak function
Based on comparison you can place message on the control cube or do externa actions like send an email to support.
Re: RunProcess and Process Ends msg
lotsaram wrote: ↑Mon Oct 14, 2019 7:49 am I second Declan's suggestion. File based semaphores are really the only way to go when running processes on independent threads. Reading a cell value (or attribute value) is vey tricky to get right because of TM1's commit model and which version of the data your monitoring process is rteading versus the separate commits of each worker thread, file based semaphores are just much easier top get right and better for one's sanity.
Just make sure you don't run too many things too fast with file semaphores
Any modern OS has it's own file system cache (and any storage system has another one as well), so if you have a lot of processes doing things in parallel a finished file write can be not 'flushed' to disk and another process can not see the completion and, for example, restart it. This is where you start seeing things like 'Write a lock file & Sleep 500ms' in TI code. Very nasty to trace
In 90% of cases files should suffice, but I'm starting to think that a sqllite instance to store the flags with guaranteed read / write might be better at large scale.
What would really be great is for IBM to come up with 'have X finished' call and then we'll all forget all these file / cube locking solutions as a bad dream they are.
Cheers,
Yuri
-
- MVP
- Posts: 3651
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: RunProcess and Process Ends msg
I find this idea quite interesting. But in exactly the kind of scenario where immediacy of file writes might be problemantic with lots of multi-threadig and dozens of TIs running at once wouldn't this have it's own locking problems as sqlite has an exclusive write access model?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
-
- MVP
- Posts: 3103
- Joined: Mon Dec 29, 2008 6:26 pm
- OLAP Product: TM1, Jedox
- Version: PAL 2.0.9.18
- Excel Version: Microsoft 365
- Location: Brussels, Belgium
- Contact:
Re: RunProcess and Process Ends msg
To check whether there are still certain files lurking around in the TM1 data directory (or a subfolder thereof), this condensed code could be used.declanr wrote: ↑Sun Oct 13, 2019 11:23 am I handle this by having the master process create a text file in a folder and pass the name of that file to the sub process (doing this for ever sub process it runs.) The last line of the sub process deletes the file.
At the end of the master process it loops the folder counting the files that exist until it finds all the files created have been deleted (indicating all sub processes have completed.) I also put a backup option in where after a set amount of time it will finish with an error if there are still files that exist (e.g if I expect it to take 20 seconds total I would maybe set a failure time of 60 seconds.)
Different processes that are running at the same time should be taking away the text files.
Code: Select all
# do we still have files lurking around ?
While( 1 > 0 ) ;
If( WildcardFileSearch( 'D:\TM1Data\*.txt', '' ) @= '' );
Break;
Else;
Sleep( 100 );
EndIf;
End;
Note the 2 undocumented functions Break and Sleep but they prove to be very useful in here ! Usually I like to condense the code to the essence
Best regards,
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Re: RunProcess and Process Ends msg
Yep, but it's not as much of problem as a solution from my point of view.lotsaram wrote: ↑Tue Oct 15, 2019 6:42 amI find this idea quite interesting. But in exactly the kind of scenario where immediacy of file writes might be problemantic with lots of multi-threadig and dozens of TIs running at once wouldn't this have it's own locking problems as sqlite has an exclusive write access model?
Write locks in SQLLite would guarantee that you'd never have a 'dirty' read as with file system (when you think there's no file present because it's still cached on OS level). And sqllite transactions are really fast, so you're sacrificing way less speed than the usual Sleep a second after a file write.
And you'd also be able to 'store' the execution timeline properly in a db table, making error tracing way simpler than trying to guess what thread created / deleted what file when.
Y
- roei61
- Posts: 41
- Joined: Wed Aug 22, 2012 5:44 am
- OLAP Product: PAX
- Version: 2.0.7
- Excel Version: 2016
- Location: Israel
Re: RunProcess and Process Ends msg
Hi,
Thank you all for the answers.
However, I'm still missing solution to know if there was a error \ minor error while running those processes.
I don't want the user to get the process ends msg if there were errors.
Since RunProcess brings no result I need Idea how can I register the process result.
Thank you all for the answers.
However, I'm still missing solution to know if there was a error \ minor error while running those processes.
I don't want the user to get the process ends msg if there were errors.
Since RunProcess brings no result I need Idea how can I register the process result.
-
- MVP
- Posts: 3103
- Joined: Mon Dec 29, 2008 6:26 pm
- OLAP Product: TM1, Jedox
- Version: PAL 2.0.9.18
- Excel Version: Microsoft 365
- Location: Brussels, Belgium
- Contact:
Re: RunProcess and Process Ends msg
Did you bother to check the documentation on the RunProcess function ?
https://www.ibm.com/support/knowledgece ... ocess.html
There is a section on “Return values”.
https://www.ibm.com/support/knowledgece ... ocess.html
There is a section on “Return values”.
Best regards,
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
- roei61
- Posts: 41
- Joined: Wed Aug 22, 2012 5:44 am
- OLAP Product: PAX
- Version: 2.0.7
- Excel Version: 2016
- Location: Israel
Re: RunProcess and Process Ends msg
Yes, I saw it.
1. This part is not clear to me, how can I get an error from the running process if it did not ended yet since main process already ends?
2. How can I recognize "minor errors", "Error" etc. ?
1. This part is not clear to me, how can I get an error from the running process if it did not ended yet since main process already ends?
2. How can I recognize "minor errors", "Error" etc. ?
-
- MVP
- Posts: 3103
- Joined: Mon Dec 29, 2008 6:26 pm
- OLAP Product: TM1, Jedox
- Version: PAL 2.0.9.18
- Excel Version: Microsoft 365
- Location: Brussels, Belgium
- Contact:
Re: RunProcess and Process Ends msg
Do you stretch the main process with the file semaphore, as suggested above ?
You can only see whether there was an error or not, AFAIK, looking at the documentation. Or the subprocess should be augmented with your ow custom logging and based on that, retrieve the process outcome.
As always, it would help if you post the exact code in TI.
You can only see whether there was an error or not, AFAIK, looking at the documentation. Or the subprocess should be augmented with your ow custom logging and based on that, retrieve the process outcome.
As always, it would help if you post the exact code in TI.
Best regards,
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
-
- MVP
- Posts: 3103
- Joined: Mon Dec 29, 2008 6:26 pm
- OLAP Product: TM1, Jedox
- Version: PAL 2.0.9.18
- Excel Version: Microsoft 365
- Location: Brussels, Belgium
- Contact:
Re: RunProcess and Process Ends msg
This works for ExecuteProcess but not RunProcess.tiagoblauth wrote: ↑Mon Oct 14, 2019 3:30 pm I used to create control cubes, so I store start date and end dates and completion. You can provide read access to users, so they can monitor it.
You can also check for errors by during the process when you run trigger processes this way:
return_value = ExecuteProcess('create_sales_cube');
return_value can be compared with these functions:
ProcessExitByChoreQuit() - indicates that the process exited due to execution of the ChoreQuit function
ProcessExitNormal() - indicates that the process executed normally
ProcessExitMinorError() - indicates that the process executed successfully but encountered minor errors
ProcessExitByQuit() - indicates that the process exited because of an explicit "quit" command
ProcessExitWithMessage() - indicates that the process exited normally, with a message written to tm1server.log
ProcessExitSeriousError() - indicates that the process exited because of a serious error
ProcessExitOnInit() - indicates that the process aborted during initialization
ProcessExitByBreak() - indicates that the process exited because it encountered a ProcessBreak function
Based on comparison you can place message on the control cube or do externa actions like send an email to support.
Best regards,
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
-
- MVP
- Posts: 3103
- Joined: Mon Dec 29, 2008 6:26 pm
- OLAP Product: TM1, Jedox
- Version: PAL 2.0.9.18
- Excel Version: Microsoft 365
- Location: Brussels, Belgium
- Contact:
Re: RunProcess and Process Ends msg
Hi Declan
Could you give some more information about the backup setup ?
Let's say there is a mother process that kicks off 12 times the same process (once for every month), and the whole process is taking 5 minutes where you except 2 minutes. So after 2 minutes, the mother process is stopped, but the subprocess keeps running for months 4, 5, 6, ..., 12 for example ?
The output is a message in the TM1 message log and the mother process that stops but you do not take action to stop the subprocess (runs) ?
If it's that, I could do it, but I would be interested in knowing if you also stop the subprocess in one what or another. For example with a different text file ?
Best regards,
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Wim Gielis
IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
- tiagoblauth
- Posts: 20
- Joined: Wed Jul 08, 2015 11:50 am
- OLAP Product: TM1, Cognos BI
- Version: 10.2.1
- Excel Version: 2010
Re: RunProcess and Process Ends msg
To monitor multiple processes running at the same time and having multiple users running it, I used to create control cubes. It lists the processes needed, parameters, start / end date, process status (queued / running / completed / error), dependency. So, you can monitor what is going on and also the process did not start running in the middle of another execution.
-
- Regular Participant
- Posts: 159
- Joined: Fri Aug 12, 2016 10:02 am
- OLAP Product: tm1
- Version: 10.2.0 - 10.3.0
- Excel Version: 2010
Re: RunProcess and Process Ends msg
Our manager in ibm temm us you need use rest api (pass them you QUEID) to retrive status execution process.
We still wait norml function retriving status in TI.
We still wait norml function retriving status in TI.
-
- Posts: 13
- Joined: Thu Jun 16, 2016 7:57 pm
- OLAP Product: Planning Analytics
- Version: 2.0
- Excel Version: 2016
Re: RunProcess and Process Ends msg
Hi,
I have built a bunch of new processes to replace our current solution with Rest API, using a lot of the solutions discussed above.
Our master process does almost everything in the prologue tab, there are just some cubelogging turned back on in the epilogue tab and deleting temp subsets. All of the code above is in Prologue. I have ran into the following issue:
I don't want to have a bunch of parameters to give information from one process to its sub, so to avoid this, I have implemented a set of global variables ("is this a chore?", "which chore?", "path to folder containing txt files that have to be wildcardsearched", etc). During testing, however, I have found that while everything works well when I use ExecuteProcess, when I switch it out for RunProcess, the global variables all end up being empty strings or 0s.
In the above code: when I replace RunProcess with ExecuteProcess, the global variable gets transferred and recognized. When I turn it back into RunProcess, the subprocess sees vGlobalTxtFileName as empty.
Has anyone else run into this issue? How do you send the name of the text file that has to be deleted to the subprocess? Do you use a parameter, do global variables work for you, is there a third option you use? (I have now created a small admin cube to write this info into, and read it from, but I still feel global variables would be more elegant)
Thanks in advance!
I have built a bunch of new processes to replace our current solution with Rest API, using a lot of the solutions discussed above.
Code: Select all
StringGlobalVariable( 'vGlobalTxtFileName' );
n_calcA = 1;
WHILE( n_calcA <= nASubsetSize );
sA = SubsetGetElementName( sDimA, cTemp, n_calcA );
vGlobalTxtFileName = cFolder | sProcess | '_' | sA | '_' | sVersion | '.txt';
ASCIIOUTPUT( vGlobalTxtFileName, 'Started' );
RunProcess( sProcess, 'pA', sA, 'pB', sB, 'pVersion', sVersion );
#This sleep is here to give time to the subset to register the current global variable into a local variable
Sleep( 100 );
ENDIF;
n_calcA = n_calcA + 1;
END;
While(
WildcardFileSearch( cFolder | sProcess | '*.txt', '' ) @<> ''
% WildcardFileSearch( cFolder | sProcess | '*.txt$', '' ) @<> ''
% n_FailSafe >= 6 * 30
% n_FailSafe = 1
) ;
Sleep( 10000 );
n_FailSafe = n_FailSafe + 1;
End;
I don't want to have a bunch of parameters to give information from one process to its sub, so to avoid this, I have implemented a set of global variables ("is this a chore?", "which chore?", "path to folder containing txt files that have to be wildcardsearched", etc). During testing, however, I have found that while everything works well when I use ExecuteProcess, when I switch it out for RunProcess, the global variables all end up being empty strings or 0s.
In the above code: when I replace RunProcess with ExecuteProcess, the global variable gets transferred and recognized. When I turn it back into RunProcess, the subprocess sees vGlobalTxtFileName as empty.
Has anyone else run into this issue? How do you send the name of the text file that has to be deleted to the subprocess? Do you use a parameter, do global variables work for you, is there a third option you use? (I have now created a small admin cube to write this info into, and read it from, but I still feel global variables would be more elegant)
Thanks in advance!
- PavoGa
- MVP
- Posts: 616
- Joined: Thu Apr 18, 2013 6:59 pm
- OLAP Product: TM1
- Version: 10.2.2 FP7, PA2.0.9.1
- Excel Version: 2013 PAW
- Location: Charleston, Tennessee
Re: RunProcess and Process Ends msg
Since RunProcess operates by opening a separate thread, it has no connection to the Session or Global variables declared in the master process.HyunaHyuna wrote: ↑Fri Feb 07, 2020 2:09 pm
I don't want to have a bunch of parameters to give information from one process to its sub, so to avoid this, I have implemented a set of global variables ("is this a chore?", "which chore?", "path to folder containing txt files that have to be wildcardsearched", etc). During testing, however, I have found that while everything works well when I use ExecuteProcess, when I switch it out for RunProcess, the global variables all end up being empty strings or 0s.
In the above code: when I replace RunProcess with ExecuteProcess, the global variable gets transferred and recognized. When I turn it back into RunProcess, the subprocess sees vGlobalTxtFileName as empty.
Ty
Cleveland, TN
Cleveland, TN
-
- Community Contributor
- Posts: 324
- Joined: Mon Jul 02, 2012 9:39 pm
- OLAP Product: TM1
- Version: PAL 2.0.8
- Excel Version: 2016
- Location: Sydney, Australia
Re: RunProcess and Process Ends msg
The return value is JobID which I understand is only possible to get via REST API? So you still need something like POSTMAN etc. to execute and capture the valueWim Gielis wrote: ↑Sun Oct 20, 2019 1:33 pm Did you bother to check the documentation on the RunProcess function ?
https://www.ibm.com/support/knowledgece ... ocess.html
There is a section on “Return values”.