When does a chore commit data changes?

[Steve Rowe]

Hi,
I'm working on a complex data processing instance with many mutli-threaded chores running in parallel.

If Chore 1 is running TI A which writes a value to Cube Z when is the change available to Chore 2 running TI B which wants to read the value in Cube Z.

A. As soon as the CellPutN executes in TI A
B. As soon as tab in TI A completes
C. As soon as TI A completes
D. As soon as chore 1 completes.

I really want A, B or C but only D explains some of the erratic behaviour I am seeing...
Any views (or even better facts)?
Cheers,

[qml]

It's option D if you're running the chore in Single Commit Mode and option C if you're running it in Multi Commit Mode. This is provided that we're talking about data commits. Metadata committing works slightly differently.

[Steve Rowe]

Ta, hopefully single commit is the default, can't check from here. Tks a lot Kamil

[qml]

Single Commit is indeed the default. Multi Commit was only introduced in 10.1 and you have to tick a checkbox to turn it on for any given chore.

[Steve Rowe]

I had a play around with this and it doesn't really work the way I needed it to.

My Chore looks like this

Chore calls T1_1 which calls TI_1.1 or TI_1.2 or TI_1.3 depending on various factors, I need the multiple commit mode to work at the TI 1.X level but it only works at the top level of the chore.
So for a chore with only a single TI the two modes of running the chore makes no difference.
I couldn't get the process flow I needed if I put TI_1.1 to TI_1.3 into the chore instead so ended up putting out a text file with the value in it that wasn't getting committed in time for another chore to read.

Thanks for the pointer though!

Cheers,

[declanr]

You could also look at instead using TM1RunTI.exe to execute the sub processes. That way they essentially work outside of the chore but you can still use the wait parameter to make sure they are finished before you move on to the next step.
Just need to be careful about locking etc.

[TableManagerOne]

May be worth mentioning that the SaveDataAll/CubeSaveData commands will inject a commit (now two) into the mix.

http://www-01.ibm.com/support/docview.w ... wg27039732

[BariAbdul]

so ended up putting out a text file with the value in it that wasn't getting committed in time for another chore to read.

Hi Steve,Could you please elaborate it? Thanks

[Steve Rowe]

Hi,
@BariAbdul,
I have two chores. Both run a loop over available work to do which is driven by batch files being available in a folder. Chore 2 needs information that Chore 1 works out. But Chore 1 doesn't complete (because it is working on "Batch2") by the time Chore 2 is working on Batch1 and so the information Chore1 established is not available in Chore2. The solution was to add the information that Chore2 needs to the batch file that is generated by Chore1.

@TableManageOne - Although a savedata would commit the data (I think) it increases the chance of a lock occurring which is something we have put a lot of effort into avoiding, so we had ruled that out.

Cheers,

[BariAbdul]

Thanks a lot Steve.

[Steve Rowe]

Ok one more question...
At time =0.0 Chore 1 is running and writing (but not committing) to a cube.
At time =5.5 Chore 1 completes and commits its data.

but

At time =5.0 Chore 2 launches, i.e. before Chore 1 commits.
At time =6.0 Chore 2 reads the data that Chore 1 has committed.

Is the data from Chore 1 available to Chore 2?

The behaviour I am seeing is that it is not but I'd like to sense check this as although I get a chore having its own write buffer and then committing, it seems very strange that these changes are not available everywhere, including jobs already in-flight, as soon as it is committed.

(I might need to be committed by the time I finish this project.. )

Cheers,

[TableManagerOne]

It depends. The version of a cube a chore (or any transaction) will see depends on when that cube is first referenced (and any subsequent changes to it within the transaction.) With Parallel Interaction enabled, Cube Data is separate from the main cube, and the version of that seen will depend on the first time cube data is referenced. However, it's isn't as transparent as you'd think. This referencing happens internally, maybe unexpectedly, as a result of accessing other things. For example, referencing a dimension may reference a cube. Referencing a cube will likely reference all upstream cubes (to keep caches based on inter-cube rules consistent.) Referencing something like the transaction logging property may reference cube data (as it is "versioned" along side to avoid locking when it is updated.)

So, on the surface, yes, you would expect that if chore 2 only reads the data at time 6.0, after the first chore committed at 5.5, it should see the updated data. Probably though, there is some other (indirect) access to the cube (data) that causes an earlier version to be referenced - for the duration of chore 2.

You could enable debug logging around cube data versions, etc. to determine exactly when this stuff happens - probably not why though.

[Steve Rowe]

Hi Thanks,
I should have been clearer in my description I think.

At time =0.0 Chore 1 is running and writing (but not committing) to a cube.
At time =5.5 Chore 1 completes and commits its data.

but

At time =5.0 Chore 2 launches, i.e. before Chore 1 commits. This is calling a sub TI in a loop
At time =5.2 a sub TI in chore 2 exports the data from "the cube".
At time =6.0 the same sub TI in chore 2 exports the data from "the cube" that was committed at time 5.5.

So by your explanation the event at time=6.0 will fail since the chore "cloned" a copy of the cube when it was first referenced at time=5.2.
This is exactly what I see so thanks for the confirmation and swift response.
Cheers,

Steve

[TableManagerOne]

Sounds right. It's for these complications that I've personally favored using an external scripting language, with built in synchronization primitives, in conjunction with RunTI.

[David Usherwood]

Good idea.
Which language is your preference?

[TableManagerOne]

Good question. When I said I favored using an external scripting language, I should have said, I favored the idea of using an external scripting language.

I have done this with C++ as it was most familiar to me. Easy to hit the ground running. It has it's downside being compiled rather than interpreted. The newer incarnations (C++11, 14) can be elegant enough.

I made an attempt with Python at one point, but found the process API cumbersome (there were a few competing APIs if I recall.) Though, I didn't expend much effort.

These days I'm writing in Ruby and Javascript primarily. I'd be curious to see how well Ruby could handle the task, but I've moved on from TM1, and doubt I'd ever find the time to investigate now (not to mention the difficulty in testing w/o access to TM1.) Many would say Javascript will be the One language... Could be done with Node.js..

Again, good question. I'm interested to hear what others have used to do this.

[failurehappening]

One way that involves mostly TM1 is the use of preset dependencies and flags on the file system...

• Every process in your cube load section of the chore is kicked off at the same time using RunTI.
• The first thing each process does is look into a dependency cube to see if it is dependent on any other process.
• If it is not dependent, the cube load process runs, when it completes it creates a file on the file system with its process name in the filename.
• If there is a dependency, the process will wait until a file is created on the file system with that process name in. When that file arrives, the main process may continue.

You can get quite complex dependencies running in this way. With each process committing the data before the next one access it.