Background processing / queue (Not chores)

[JohnO]

I have set of challenges (Locking and number of parallel threads being executed) that I want to solve by moving processing of certain user driven TI's into a queue rather than being executed immediately. So essentially when the user clicks the button it would take the parameters and pass those details into a queue whereby the user requests would then be run sequentially with appropriate email notifications back tot he user. I won't go into the detail for wanting to do this.

I have a design in mind and some existing code I can re-use but before going down the path I was wondering if anyone had already done this / whether there is anything readily available and sufficiently robust. Perhaps one of the consulting firms has already built one?

[Alan Kirk]

I have set of challenges (Locking and number of parallel threads being executed) that I want to solve by moving processing of certain user driven TI's into a queue rather than being executed immediately. So essentially when the user clicks the button it would take the parameters and pass those details into a queue whereby the user requests would then be run sequentially with appropriate email notifications back tot he user. I won't go into the detail for wanting to do this.

I have a design in mind and some existing code I can re-use but before going down the path I was wondering if anyone had already done this / whether there is anything readily available and sufficiently robust. Perhaps one of the consulting firms has already built one?

It may be worthwhile to take a look at Cubewise's Hustle, which can be found here. It's a freebie which, and I quote:

... is a small utility that can be used to manage threads when executing command line tools. The tool was built to take advantage parallel loading in IBM Cognos TM1, specifically the tm1runti.exe. Hustle enables you to specify the number of concurrent threads you want to be executed at any one time and pass a batch of commands to be executed on these threads.

Although Hustle was designed to be used with TM1 it can be used to execute any command line executable. It is great for run batch processes concurrently allowing you take advantage of all of your CPU cores.

The only issue is that it does require the .Net 3.5 framework to run. If you're on Windows you should be fine, if you're on Linux, eeeehhhh...

I did use it once or twice but never really had a huge need for it in the past... though I may do in the near future.

[Terramup]

Hi,
If you just want them to run sequentially, there is the synchronized function that can be added to your TI scripts. It has a lot less features than a queueing process, but it is a simple way to force multiple user run processes to run sequentially.

https://www.ibm.com/docs/en/planning-an ... nchronized

Good luck!

[ykud]

Have a look at IBM's one for inspiration. it's usually something in the same vein + some pegs and bandaids for a particular operating system or any other idiosyncrasies.

https://ibm.ent.box.com/s/v2r2np0ijarak ... 7bae75v9ne

[Wim Gielis]

if I understand it well, it's about running in sequence, not in parallel.

[JohnO]

if I understand it well, it's about running in sequence, not in parallel.

Correct. The scenario is that we have 10 users who are triggering TI's (May be the same or different) and these processes are overlapping and impacting each other. So what I want to do is set up a queue and change things so that when users click on an action button their request goes into a queue and then these queue of actions/ processes occur in sequence. In addition, one of these TI's involves the issuing of parallel threads via RunProcess and this is an added complexity. But the fundamentally need is for a queue of processes to run in the background. It would help in terms of conflicts and also with the parallel processes, in terms of the latter the issue is that we are running 20 parallel threads and currently the server ends up getting overloaded, in particular when we have 2 requests (Different users or maybe same user - another story) and there are conflicts which end up in rollbacks ...... It's a world of pain.

I am building my solution now, should have it done next week.

[Steve Rowe]

One thing that would be worth doing in your build is before (and maybe after) a job runs check for another occurence of the same job with the same parameters and clear it from the queue. Just to protect yourself from impatient users who sit there clicking the button becuase "nothings happening"...

Isn't there a method of forcing all TIs to be serial using synchronised

Basically have a cube called "Make TIs Serial"

Then the launch job of each TI set has Synchronized("Make TIs Serial"); in the prolog

And then TM1 will makes its own queue. Obviously not something you can monitor or intefere with though.

[EDIT : Apologies for mentioning Sychronised again, I somehow missed the post higher up]

Cheers,

[PavoGa]

I actually do have a solution that does this with a mix of Synchronized and a file semaphore to control users kicking off multiple executions of the same master process.

The overall process runs processes that multi-thread in preparing and loading cubes and performing allocations. One group of threads must finish before the next group begins.

Using SYCHRONIZED one can lock multiple objects. A simple example:

ProcessDoThisFirst is executed three times by its master process. Each thread locks respectively: Larry, Curly and Moe.

The second process, ProcessDoThisSecond, is actually going to be managed by a sub-master process which will run ProcessDoThisSecond multiple times, each spawned child thread with its own unique SYNCHRONIZED object. The sub-master is executed by the orginal master process with RUNPROCESS.
The key here is that the sub-master process is passed a string Larry:Curly:Moe (the individual objects locked by the first group's threads) that it parses and locks each object in turn until all have been unlocked. Once it obtains locks on all three of the passed parameter lock objects, it spawns off it's child threads.

In my case, we work through about four groups of multi-threaded processes, each group which has to finish before the next set executes. Because all are executed with RUNPROCESS, they all stack up waiting on each precedent group to complete before executing. I manage the workload for each individual thread within each group through a control cube; I can change the number of spawns and/or adjust the size of the slice of the cube each is processing.

By the way, using multiple SYNCHRONIZED lock objects within one process works, but has a couple of quirks that can have one scratching one's head. Have been contemplating writing up a paper on SYNCHRONIZED lock management, but have not gotten around to it.

[PavoGa]

One thing that would be worth doing in your build is before (and maybe after) a job runs check for another occurence of the same job with the same parameters and clear it from the queue. Just to protect yourself from impatient users who sit there clicking the button becuase "nothings happening"...

Isn't there a method of forcing all TIs to be serial using synchronised

Basically have a cube called "Make TIs Serial"

Then the launch job of each TI set has Synchronized("Make TIs Serial"); in the prolog

And then TM1 will makes its own queue. Obviously not something you can monitor or intefere with though.

[EDIT : Apologies for mentioning Sychronised again, I somehow missed the post higher up]

Cheers,

By the way, the argument for SYNCHRONIZED does not have to be an actual object in the model. I use virtually anything, for example, the names of University of Georgia football players, entertainers (see: The Three Stooges), battles from War Between the States (being from Georgia, these tend to favor Southern victories), favorite songs, etc. This is handy for managing groups of spawned threads as mentioned above.

[Alan Kirk]

[By the way, the argument for SYNCHRONIZED does not have to be an actual object in the model. I use virtually anything, for example, the names of University of Georgia football players, entertainers (see: The Three Stooges), battles from War Between the States (being from Georgia, these tend to favor Southern victories), favorite songs, etc. This is handy for managing groups of spawned threads as mentioned above.

Georgia has a football team? Why, I never knew!

[PavoGa]

[By the way, the argument for SYNCHRONIZED does not have to be an actual object in the model. I use virtually anything, for example, the names of University of Georgia football players, entertainers (see: The Three Stooges), battles from War Between the States (being from Georgia, these tend to favor Southern victories), favorite songs, etc. This is handy for managing groups of spawned threads as mentioned above.

Georgia has a football team? Why, I never knew!

That is funny, but Alabama fans take it, what's the proper phrase here...yes! Deadly serious.

In the last ten years, two Alabama fans have been shot and killed after Alabama losses including one woman gunned down by her sister-in-law for not taking a loss to Auburn seriously enough.

And by the way, Goooooooooooo DAWGS! Sic'em! Woof! Woof! Woof!

[JohnO]

An update. I have gone down the route of using SYNCHRONIZED. No doubt a very simple solution however upon initial testing I did find something strange.

At a minimum I have 2 different processes I want to control, by that I mean I don't want either to run at the same time and I also don't want multiple instances of either to run at the same time.

. One of the two processes calls many sub processes in series - call this 'TI A'
. The other calls multiple sub-processes using runprocess - call this 'TI B'. We are NOT using SYNCHRONISED within the TI to control subprocesss

In my initial test of 'TI B' , the subprocesses ran in series, not in parallel. But in subsequent tests, runprocess was respected in that the sub processes were performed in parallel. Seems more than a bit strange, we'll be doing more testing to see if this reoccurs or whether it is a 'first instance' thing for some reason.

[JohnO]

We have gone down the path of testing with the users. What I see is things running in sequence with a number of processes in a wait state and then change to run in sequence. Everything appears to go ok.

However what I see in the server log is the following message for each new case. It appears before the prior process has completed.

Process "xxxx": Execution rolling back due to lock exception

I would not have expected to see this.

For those who have used SYCHRONIZED is this what you see in your logs? Does it just mean that it attempted the process but could not because of the lock placed by SYNCHRONIZED?

[PavoGa]

We have gone down the path of testing with the users. What I see is things running in sequence with a number of processes in a wait state and then change to run in sequence. Everything appears to go ok.

However what I see in the server log is the following message for each new case. It appears before the prior process has completed.

Process "xxxx": Execution rolling back due to lock exception

I would not have expected to see this.

For those who have used SYCHRONIZED is this what you see in your logs? Does it just mean that it attempted the process but could not because of the lock placed by SYNCHRONIZED?

Really need more information to help with this. What objects are reported as locked? Where are your SYNCHRONIZED statements within the processes (i.e. are they at the very beginning, epilog before finishing)? TI B is executing subprocesses with no control, so is TI B finishing before it's child threads are?

The rollback indicates some object the TI needs (a dimension/subset) is causing a lock after some processing has taken place and therefore the rollback occurs. Here is a gotcha: if you are creating subsets on a common dimension(s), these also have to be created in a separate thread or they create a lock for the duration or lifetime of the TI being executed. Does not matter if they are created with the "temporary" argument or not, they will halt any other TI with a lock exception. DIMIX will cause a lock exception, so avoid it like the plague when multi-threading.

[JohnO]

Really need more information to help with this. What objects are reported as locked? Where are your SYNCHRONIZED statements within the processes (i.e. are they at the very beginning, epilog before finishing)? TI B is executing subprocesses with no control, so is TI B finishing before it's child threads are?

The rollback indicates some object the TI needs (a dimension/subset) is causing a lock after some processing has taken place and therefore the rollback occurs. Here is a gotcha: if you are creating subsets on a common dimension(s), these also have to be created in a separate thread or they create a lock for the duration or lifetime of the TI being executed. Does not matter if they are created with the "temporary" argument or not, they will halt any other TI with a lock exception. DIMIX will cause a lock exception, so avoid it like the plague when multi-threading.

SYNCHRONIZED statements are very early on in the Prolog.

I would have expected the processes to run in sequence and hence there be no opportunity for overlap of locking.

Our testing started with the order of running the single thread process first and the parallel master process second. And still we had these lock exceptions.

[PavoGa]

Really need more information to help with this. What objects are reported as locked? Where are your SYNCHRONIZED statements within the processes (i.e. are they at the very beginning, epilog before finishing)? TI B is executing subprocesses with no control, so is TI B finishing before it's child threads are?

The rollback indicates some object the TI needs (a dimension/subset) is causing a lock after some processing has taken place and therefore the rollback occurs. Here is a gotcha: if you are creating subsets on a common dimension(s), these also have to be created in a separate thread or they create a lock for the duration or lifetime of the TI being executed. Does not matter if they are created with the "temporary" argument or not, they will halt any other TI with a lock exception. DIMIX will cause a lock exception, so avoid it like the plague when multi-threading.

SYNCHRONIZED statements are very early on in the Prolog.

I would have expected the processes to run in sequence and hence there be no opportunity for overlap of locking.

Our testing started with the order of running the single thread process first and the parallel master process second. And still we had these lock exceptions.

Again, what is reported in the server log as the subject of the lock?