Parallel Processing
Posted: Wed Jun 27, 2012 2:09 pm
Hello,
I have created a suite of processes to perform a data transformation exercise that runs ordinarily overnight following a data load. Occasionally it is necessary to re-run the load and transformation processes for a specific subset of data, let’s say for a specific department. To achieve this I have built the following:-
1) TI process to cycle through all departments to execute the CMD file (2)
ExcecuteCommand passing parameters
psDeptID – 00 for all depts., Dept ID for specific Dept
psDataClearOnly
Wait is set to zero so that all departments are transformed using parallel processing
2) CMD file to launch (3)
tm1runti.exe .... psDeptID, psDataClearOnly
3) TI process performs data clear operation then if calls psDataClearOnly is FALSE calls (4)
ExcecuteProcess ... psDeptID
4) TI process performs data transformation.
This all works perfectly well, with the CMD file logging the execution of each process (3) by dept and process (4) logging the end. Several CMD.EXE’s are shelled on the server which then close down as each finishes processing for their designated department.
I tried to add another parameter to process (1) that would allow me to control the Wait option. If I set Wait to 1 then the CMD.EXE & tm1runti.exe processes that are shelled never close on the server although the data processing is completed and is visible in the target cube. Does anyone have ideas why this might be? I wondered if it was because I used tm1runti.exe to call proc 3 which in turn calls proc 4.
I hope my explanation makes some kind of sense.
I have created a suite of processes to perform a data transformation exercise that runs ordinarily overnight following a data load. Occasionally it is necessary to re-run the load and transformation processes for a specific subset of data, let’s say for a specific department. To achieve this I have built the following:-
1) TI process to cycle through all departments to execute the CMD file (2)
ExcecuteCommand passing parameters
psDeptID – 00 for all depts., Dept ID for specific Dept
psDataClearOnly
Wait is set to zero so that all departments are transformed using parallel processing
2) CMD file to launch (3)
tm1runti.exe .... psDeptID, psDataClearOnly
3) TI process performs data clear operation then if calls psDataClearOnly is FALSE calls (4)
ExcecuteProcess ... psDeptID
4) TI process performs data transformation.
This all works perfectly well, with the CMD file logging the execution of each process (3) by dept and process (4) logging the end. Several CMD.EXE’s are shelled on the server which then close down as each finishes processing for their designated department.
I tried to add another parameter to process (1) that would allow me to control the Wait option. If I set Wait to 1 then the CMD.EXE & tm1runti.exe processes that are shelled never close on the server although the data processing is completed and is visible in the target cube. Does anyone have ideas why this might be? I wondered if it was because I used tm1runti.exe to call proc 3 which in turn calls proc 4.
I hope my explanation makes some kind of sense.