BatchUpdateStart and BatchUpdateFinishWait in master process
-
- Posts: 16
- Joined: Tue Feb 12, 2013 7:31 am
- OLAP Product: TM1+Cognos Express
- Version: 9.5.2+10.1+10.2
- Excel Version: 2007+2010
- Location: Prague
BatchUpdateStart and BatchUpdateFinishWait in master process
Hello,
Is it ok to use BatchUpdateStart and BatchUpdateFinishWait in a master process that calls several child processes? Or is it necessary to put them in each of child process? When using batch updates in master process I get inconistent results in target cubes for each child process.
Thanks for help
Is it ok to use BatchUpdateStart and BatchUpdateFinishWait in a master process that calls several child processes? Or is it necessary to put them in each of child process? When using batch updates in master process I get inconistent results in target cubes for each child process.
Thanks for help
-
- Community Contributor
- Posts: 341
- Joined: Wed Nov 03, 2010 9:16 pm
- OLAP Product: tm1
- Version: 10 2 2 - 2.0.5
- Excel Version: From 2007 to 2013
- Location: Earth
Re: BatchUpdateStart and BatchUpdateFinishWait in master pro
Hi
It is ok to use as long as in the second child TI you do not reference results from the first child TI
So if the executed TIs are independent from each other, this should work. If TI 1 writes a value to a cell that TI2 reads then, I assume it won't work.
It is ok to use as long as in the second child TI you do not reference results from the first child TI
So if the executed TIs are independent from each other, this should work. If TI 1 writes a value to a cell that TI2 reads then, I assume it won't work.
-
- Posts: 16
- Joined: Tue Feb 12, 2013 7:31 am
- OLAP Product: TM1+Cognos Express
- Version: 9.5.2+10.1+10.2
- Excel Version: 2007+2010
- Location: Prague
Re: BatchUpdateStart and BatchUpdateFinishWait in master pro
Thank you mvaspal!mvaspal wrote:Hi
It is ok to use as long as in the second child TI you do not reference results from the first child TI
So if the executed TIs are independent from each other, this should work. If TI 1 writes a value to a cell that TI2 reads then, I assume it won't work.
The above seems logical, however is the isolation of data provided per each child process? Just hypothetically:
1) We have master process that calls BatchupdateStart, then spawns child process A and then process B. Finally it will fire BatchUpdateFinishWait.
2) Child process A modifies cube C and writes data to slice S1.
3) Child process B modifies cube C and writes data to slice S2.
4) Slices S1 and S2 are independent.
Do I understand correctly that BatchUpdate will create sort of working space that will be later commited by BatchUpdateFinish? I'm still lost how these spaces are distributed per child processes. If only one working space is created in master process I would assume both child processes are contributing to the same space. I understand that reading from such space would provide inconsistent results because missing commit action.
Thank you
EDIT: It seems that the child processes can't see any changes provided by other ones as they will see just the state before BatchUpdateStart. Will try it to confirm.
-
- Community Contributor
- Posts: 341
- Joined: Wed Nov 03, 2010 9:16 pm
- OLAP Product: tm1
- Version: 10 2 2 - 2.0.5
- Excel Version: From 2007 to 2013
- Location: Earth
Re: BatchUpdateStart and BatchUpdateFinishWait in master pro
Hi
I would think that if the data source is odbc or txt or another cube, and Process B does not use any data from the target cube's S1 slice then it should work, but this is just an assumption, I have only ever used batch update in a master process when on its Data tab I went through the }Cubes dimension and called the child processes one by one for each cube, and these child processes only copied/cleared/froze versions within the same cube (btw I called the same child process for each cube, just with diffferent parameters), so my case is a bit different from yours
I would think that if the data source is odbc or txt or another cube, and Process B does not use any data from the target cube's S1 slice then it should work, but this is just an assumption, I have only ever used batch update in a master process when on its Data tab I went through the }Cubes dimension and called the child processes one by one for each cube, and these child processes only copied/cleared/froze versions within the same cube (btw I called the same child process for each cube, just with diffferent parameters), so my case is a bit different from yours
I would think so, yesDo I understand correctly that BatchUpdate will create sort of working space that will be later commited by BatchUpdateFinish?
-
- Posts: 16
- Joined: Tue Feb 12, 2013 7:31 am
- OLAP Product: TM1+Cognos Express
- Version: 9.5.2+10.1+10.2
- Excel Version: 2007+2010
- Location: Prague
Re: BatchUpdateStart and BatchUpdateFinishWait in master pro
Hello mvaspal,mvaspal wrote:Hi
I would think that if the data source is odbc or txt or another cube, and Process B does not use any data from the target cube's S1 slice then it should work, but this is just an assumption, I have only ever used batch update in a master process when on its Data tab I went through the }Cubes dimension and called the child processes one by one for each cube, and these child processes only copied/cleared/froze versions within the same cube (btw I called the same child process for each cube, just with diffferent parameters), so my case is a bit different from yours
I would think so, yesDo I understand correctly that BatchUpdate will create sort of working space that will be later commited by BatchUpdateFinish?
I gave it a short try and results are interesting:
I have created simple cube X with dimension element A in one dimension and element Value in the second. I filled manually value as 100.
I have created 3 simple processes:
Code: Select all
### Master Process
asciioutput('test3.txt', 'Master Check 1', numbertostring(cellgetn('X', 'A','value')));
batchupdatestart;
executeprocess('Child 1');
asciioutput('test3.txt', 'Master Check 2', numbertostring(cellgetn('X', 'A','value')));
executeprocess('Child 2');
asciioutput('test3.txt', 'Master Check 3', numbertostring(cellgetn('X', 'A','value')));
batchupdatefinishwait(0);
asciioutput('test3.txt', 'Master Check 4', numbertostring(cellgetn('X', 'A','value')));
Code: Select all
### Child process 1
asciioutput('test1.txt', 'Child 1, Check 1', numbertostring(cellgetn('X', 'A','value')));
cellputn(200, 'X', 'A', 'value');
asciioutput('test1.txt', 'Child 1, Check 2', numbertostring(cellgetn('X', 'A','value')));
Code: Select all
### Child process 2
asciioutput('test2.txt', 'Child 2, Check 1', numbertostring(cellgetn('X', 'A','value')));
cellputn(300, 'X', 'A', 'value');
asciioutput('test2.txt', 'Child 2, Check 2', numbertostring(cellgetn('X', 'A','value')));
Code: Select all
test3.txt
"Master Check 1","100"
"Master Check 2","200"
"Master Check 3","300"
"Master Check 4","300"
Code: Select all
test1.txt
"Child 1, Check 1","100"
"Child 1, Check 2","200"
Code: Select all
test2.txt
"Child 2, Check 1","200"
"Child 2, Check 2","300"
Best
Petr
EDIT2:
I have updated the processes to be able to run in parallel, I have added new element B for parallel test and let two master processes run in parallel, each one writing independently to element A, resp. B. Result is again values are consistent across each parallel run.
-
- Posts: 13
- Joined: Thu Jun 16, 2016 7:57 pm
- OLAP Product: Planning Analytics
- Version: 2.0
- Excel Version: 2016
Re: BatchUpdateStart and BatchUpdateFinishWait in master process
Hi, I hope you don't mind me bringing this back.
I am trying to have a master process execute other processes in parallel, and some of those need BatchUpdate. Here is what I am doing now:
Master Process:
Content of FileThatParallelRunsProcessA.txt (ProcessB is a separate .txt file with the same lines, just with the process name changed) (this is the first line, repeated n times over pCountry):
Both ProcessA and ProcessB create the necessary source and target views, set the sourcedata as the source view, viewzeroout the target view, then do some CellPutNs in the data tab (ProcessB a bunch more than ProcessA), and then delete temporary objects in epilogue. ProcessB in addition has BatchUpdateStart in Prologue and BatchUpdateFinishWait(0) in Epilogue. Both of them load into the same cube, but to different measures, and from different source cubes.
ProcessA has no BatchUpdate in the process, and it runs fine.
ProcessB has an insane amount of CellPutNs, so it needs BatchUpdate, or it runs for days.
Here is what I found: When I have BatchUpdateFinish in Epilogue of ProcessB, this causes the parallel processes to get stuck in an eternal "Wait" cycle. I guess the first process that is ready tries to finish batch updating, but all the others are still updating the cube. Removing the BatchUpdateFinish for parallel loads solved the infinite loop, but here came the problem I am struggling with now.
When I remove BatchUpdateStart from the Prologue of ProcessB, and have it outside in the master process, it doesn't seem to work, I don't think BatchUpdate is turned on for the case. It runs for ages, when it should be done in minutes. When I have BatchUpdateStart in the Prologue tab of the process AND the master process, but only have BatchUpdateFinishWait(0) in the master process, the data isn't saved. All the calculations run (textoutput in the data tab confirms that data is fine) and they run fast, but I get an empty view as a result, because the BatchUpdate didn't seem to finish. (Strange that I get the textoutputs, though)
It seems to me from all this that BatchUpdateStart does not work outside in a master process with how I am doing it, and I wondered whether you could offer some advice on what could I be doing wrong. My other option (I think, I haven't tested it) is to somehow determine which country runs for the longest, and have the BatchUpdateFInish turned on for that one in the Epilogue of ProcessB, but that kind of changes with the data input in the cells, so it's a wonky solution.
I am trying to have a master process execute other processes in parallel, and some of those need BatchUpdate. Here is what I am doing now:
Master Process:
Code: Select all
ExecuteCommand('"(...)Hustle.exe" "\\(...)\FileThatParallelRunsProcessA.txt" 20', 1);
BatchUpdateStart;
ExecuteCommand('"(...)Hustle.exe" "\\(...)\FileThatParallelRunsProcessB.txt" 20', 1);
BatchUpdateFinishWait(0);
Code: Select all
\\(location on server)\tm1runti.exe -process ProcessA -adminhost host -server servername -user username
-Camnamespace CNS -pwd password pCountry="xxx1" pMonth="yyy" pVersion="zzz" pParallel=1
ProcessA has no BatchUpdate in the process, and it runs fine.
ProcessB has an insane amount of CellPutNs, so it needs BatchUpdate, or it runs for days.
Here is what I found: When I have BatchUpdateFinish in Epilogue of ProcessB, this causes the parallel processes to get stuck in an eternal "Wait" cycle. I guess the first process that is ready tries to finish batch updating, but all the others are still updating the cube. Removing the BatchUpdateFinish for parallel loads solved the infinite loop, but here came the problem I am struggling with now.
When I remove BatchUpdateStart from the Prologue of ProcessB, and have it outside in the master process, it doesn't seem to work, I don't think BatchUpdate is turned on for the case. It runs for ages, when it should be done in minutes. When I have BatchUpdateStart in the Prologue tab of the process AND the master process, but only have BatchUpdateFinishWait(0) in the master process, the data isn't saved. All the calculations run (textoutput in the data tab confirms that data is fine) and they run fast, but I get an empty view as a result, because the BatchUpdate didn't seem to finish. (Strange that I get the textoutputs, though)
It seems to me from all this that BatchUpdateStart does not work outside in a master process with how I am doing it, and I wondered whether you could offer some advice on what could I be doing wrong. My other option (I think, I haven't tested it) is to somehow determine which country runs for the longest, and have the BatchUpdateFInish turned on for that one in the Epilogue of ProcessB, but that kind of changes with the data input in the cells, so it's a wonky solution.
-
- MVP
- Posts: 3657
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: BatchUpdateStart and BatchUpdateFinishWait in master process
You say you need BatchUpdate but do you really?
BatchUpdate more or less went the way of the dinosaurs when Parallel Interaction was introduced. So why do you think you need it?
BatchUpdate more or less went the way of the dinosaurs when Parallel Interaction was introduced. So why do you think you need it?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
-
- Posts: 13
- Joined: Thu Jun 16, 2016 7:57 pm
- OLAP Product: Planning Analytics
- Version: 2.0
- Excel Version: 2016
Re: BatchUpdateStart and BatchUpdateFinishWait in master process
Thank you for your reply
I have edited out the detailed description of my problem, as I believe I have found a solution. Instead of using CellPutNs, I TextOutPut everything into a .csv file, and then import everything from there using a separate process. I just had to be sure to clarify the data source name for parallel runs in the prologue of the second, import process:
Thank you for suggesting that BatchUpdate was a thing of the dinosaurs, it was what motivated me to look for a completely different solution!
I have edited out the detailed description of my problem, as I believe I have found a solution. Instead of using CellPutNs, I TextOutPut everything into a .csv file, and then import everything from there using a separate process. I just had to be sure to clarify the data source name for parallel runs in the prologue of the second, import process:
Code: Select all
sExportFile = sExportLocation | sFileName;
DataSourceType = 'CHARACTERDELIMITED';
DatasourceNameForServer = sExportFile;
DatasourceASCIIDelimiter=',';
DatasourceASCIIDecimalSeparator='.';
DatasourceASCIIThousandSeparator=',';
-
- Community Contributor
- Posts: 341
- Joined: Wed Nov 03, 2010 9:16 pm
- OLAP Product: tm1
- Version: 10 2 2 - 2.0.5
- Excel Version: From 2007 to 2013
- Location: Earth
Re: BatchUpdateStart and BatchUpdateFinishWait in master process
Correct me if I'm wrong but it could be still useful in a TI that loads data within a cube where the source view has lot of rule calc'd cells...at least as long as the new feature of invalidating only a slice of a cube's cache is under developmentBatchUpdate more or less went the way of the dinosaurs when Parallel Interaction was introduced.
-
- MVP
- Posts: 3657
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: BatchUpdateStart and BatchUpdateFinishWait in master process
Yes batchupdate could be useful in that scenario as long as you don't want to query any of the new data within the process. The batch updates are kind of like a pre-sandbox sandbox where the new input data is held totally outside the model and not merged at all until the commit. So any CellGetN performed during the process will read data from the status of the cube BEFORE the process. I guess that not having to build a merged model pre-commit is one reason why batchupdate is able to be so much faster. But the lack of ability to query any consolidated cells post update until AFTER the batch update has finished really limits the usefulness and potential use cases IMO.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
-
- MVP
- Posts: 2832
- Joined: Tue Feb 16, 2010 2:39 pm
- OLAP Product: TM1, Palo
- Version: Beginning of time thru 10.2
- Excel Version: 2003-2007-2010-2013
- Location: Atlanta, GA
- Contact:
Re: BatchUpdateStart and BatchUpdateFinishWait in master process
Interesting. Does that mean the the function CellIncrementN would not work in a BatchUpdate process?lotsaram wrote: ↑Mon Apr 09, 2018 3:03 pmYes batchupdate could be useful in that scenario as long as you don't want to query any of the new data within the process. The batch updates are kind of like a pre-sandbox sandbox where the new input data is held totally outside the model and not merged at all until the commit. So any CellGetN performed during the process will read data from the status of the cube BEFORE the process. I guess that not having to build a merged model pre-commit is one reason why batchupdate is able to be so much faster. But the lack of ability to query any consolidated cells post update until AFTER the batch update has finished really limits the usefulness and potential use cases IMO.
-
- MVP
- Posts: 3657
- Joined: Fri Mar 13, 2009 11:14 am
- OLAP Product: TableManager1
- Version: PA 2.0.x
- Excel Version: Office 365
- Location: Switzerland
Re: BatchUpdateStart and BatchUpdateFinishWait in master process
Been a long while since I used it but pretty sure the answer is yes, exactly this problem.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
-
- Community Contributor
- Posts: 341
- Joined: Wed Nov 03, 2010 9:16 pm
- OLAP Product: tm1
- Version: 10 2 2 - 2.0.5
- Excel Version: From 2007 to 2013
- Location: Earth
Re: BatchUpdateStart and BatchUpdateFinishWait in master process
http://tm1forum.com/viewtopic.php?f=18& ... ntN#p49881
Last time I tried CellIncrement did not work, but the old method did, which is surprising
I think in some general TIs like archiving a rule-intensive dynamic scenario to a static scenario - batch update is still a great feature
Last time I tried CellIncrement did not work, but the old method did, which is surprising
I think in some general TIs like archiving a rule-intensive dynamic scenario to a static scenario - batch update is still a great feature