How to avoid queuing at the end of processes parallelization ?
Posted: Mon Mar 25, 2024 3:17 pm
Hi all,
I am currently working on parallelizing processes using the RunProcess function.
I do have a lot of processes to run (around 2000 every time) and to avoid CPU exceeding, I use the synchronized function to force serial execution in the same way as explained in this Cubewise article : https://www.cubewise.com/blog/how-to-us ... f-threads/.
So I have limited the number of processes running simultaneously to 50 and everything works pretty find until the end when certain processes start to queue up i.e there is only one process running and a lot are in wait mode. I think that it may be because of the synchronized function when the subprocesses all share the same lock object name, they need to wait for the one currently running to finish. In the end, I have to wait for each process with this lock object to finish individually before the next one can be launched (even if there are less than 50 subprocesses remaining!) which could be parallelized instead.
Is there any solution to avoid this problem of queuing and to start again parallization in this case ? By modifying the lock object name or any other means ?
Thanks in avance
I am currently working on parallelizing processes using the RunProcess function.
I do have a lot of processes to run (around 2000 every time) and to avoid CPU exceeding, I use the synchronized function to force serial execution in the same way as explained in this Cubewise article : https://www.cubewise.com/blog/how-to-us ... f-threads/.
So I have limited the number of processes running simultaneously to 50 and everything works pretty find until the end when certain processes start to queue up i.e there is only one process running and a lot are in wait mode. I think that it may be because of the synchronized function when the subprocesses all share the same lock object name, they need to wait for the one currently running to finish. In the end, I have to wait for each process with this lock object to finish individually before the next one can be launched (even if there are less than 50 subprocesses remaining!) which could be parallelized instead.
Is there any solution to avoid this problem of queuing and to start again parallization in this case ? By modifying the lock object name or any other means ?
Thanks in avance