How to avoid queuing at the end of processes parallelization ?

[hugzer77]

Hi all,

I am currently working on parallelizing processes using the RunProcess function.

I do have a lot of processes to run (around 2000 every time) and to avoid CPU exceeding, I use the synchronized function to force serial execution in the same way as explained in this Cubewise article : https://www.cubewise.com/blog/how-to-us ... f-threads/.

So I have limited the number of processes running simultaneously to 50 and everything works pretty find until the end when certain processes start to queue up i.e there is only one process running and a lot are in wait mode. I think that it may be because of the synchronized function when the subprocesses all share the same lock object name, they need to wait for the one currently running to finish. In the end, I have to wait for each process with this lock object to finish individually before the next one can be launched (even if there are less than 50 subprocesses remaining!) which could be parallelized instead.

Is there any solution to avoid this problem of queuing and to start again parallization in this case ? By modifying the lock object name or any other means ?

Thanks in avance

[ardi]

Do yourself a favor, use RushTI. Before I started using RushTI, process parallelization was always causing challenges.

[JohnO]

There is a setting where locking is reported in TM1Server.log. This is a good place to start to gain insight.

I used RunProcess in the same manner in my prior role and had a lot of problems with locking. When these specific locks were encountered the parallel processes would go into serial mode. And the locking was not just internal to the RunProcess collection, it was due to users updating a cube which had a dimension which was an attribute of one of the dimensions of the cube being updated by the RunProcess collection. There's not much joy in this!

[ykud]

Data commit (the last step of the process) is single threaded / locking as it updates the cube index structure, so it might be the case when a lot of Tis are committing at the same time and are queuing due to this. It might be better to split data less evenly, so that they're committing at different times.

[lotsaram]

Data commit (the last step of the process) is single threaded / locking as it updates the cube index structure, so it might be the case when a lot of Tis are committing at the same time and are queuing due to this. It might be better to split data less evenly, so that they're committing at different times.

This! Don't make all your processing chunks equal sized. This will inevitably lead to a commit queue bottleneck.

[Wim Gielis]

Also, 2000 processes doing less work is most of the time less optimal than less processes doing more work (on average).