TM1 Chore Re-Executing

[dludwig]

We have a chore that is scheduled to execute 1 time each night starting at 10:20pm. This chore has been running 1 time each night for over 5 years on version 8.2.12 through 9.0. Over this past weekend we upgraded to 9.4 FP2 and the chore ran 1 time each night starting at 10:20pm until last night. Last night the chore executed as expected at 10:20pm, but then before it finished running, it executed a second time starting at 1:06am.

Here are the first few lines from the tm1server.log file that shows the chore started to execute at 10:20pm:
7984 INFO 2010-04-01 02:20:00.105 TM1.Process Process "DailyStart": finished executing normally, elapsed time 0.02 seconds
7984 INFO 2010-04-01 02:20:00.121 TM1.Process Process "SyncOff": finished executing normally, elapsed time 0.00 seconds
7984 INFO 2010-04-01 02:30:58.567 TM1.Process Process "lCTC_Save": finished executing normally, elapsed time 658.45 seconds


Here are a few other lines from teh tm1server.log file that shows the chore started to execute a second time at 1:06am:
7984 INFO 2010-04-01 05:06:08.561 TM1.Process Process "DailyStart": finished executing normally, elapsed time 0.02 seconds
7984 INFO 2010-04-01 05:06:08.577 TM1.Process Process "SyncOff": finished executing normally, elapsed time 0.02 seconds
7984 INFO 2010-04-01 05:17:09.304 TM1.Process Process "lCTC_Save": finished executing normally, elapsed time 660.73 seconds

Has anybody experienced something like this before?

[Martin Ryan]

I recall something like that. Can't remember how I fixed it. I think I deleted the chore and rebuilt it.

Martin

[BigDSter]

I've noticed that on a couple of chores since we upgraded to 9.4.1 FP1

[dludwig]

Has anybody who has seen this found a way to fix it?

[George Regateiro]

Have you tried the rebuilding like Martin suggested? Sadly this is seems to cure most chore related problems.

[dludwig]

Yes, I have rebuilt the chore. The same thing still happens. It does not happen every night.

[image2x]

Yes, I'm definitely seeing this and have had disable specific chores as a result... rebuilding them does not seem to fix the problem.

9.4.1 HF3

-- John

[dludwig]

All indications are pointing to locking issues while the chore is running which causes it to "fail" and rollback and start the chore over again from the beginning. For example, if the chore is running and then a separate process or chore executes something which locks an object and then the chore executes something that would need to lock the same object, since the chore attempted to lock the object second, it fails and rolls everything back and starts the chore over. I haven't 100% confirmed this but from talking with IBM support, this appears to be the case.

So my question is has anybody tried something like this??? As a way to prevent what appears to be locking that causes the rollback can this technique be used? Instead of having a chore with all of the TI jobs as part of the chore, can the chore just have one TI job in it that uses the ProcessExecute command to execute all of the other TI jobs? If I did that, wouldn’t TM1 acquire a lock with that single TI job that is calling all of the other TI jobs and then no other connections would be able to “get in” to cause a locking problem and a rollback?

Alternatively, is there a way to disconnect and disable all logins programmatically so that even if another process tired to execute their login would prevent them from even being able to access anything in TM1?

[Martin Ryan]

Ah, I do remember something that may be related. We had locking problems where two processes were getting confused, so we had a zLock cube that just had one cell in it. Every single process we had would write a random value to the zLock cube as the first thing it did in the Prolog. That would lock that cube and prevent any subsequent process from writing to it. That meant the subsequent processes had to wait for the first process to finish and release the lock on the zLock cube, before they could write to it, and then continue from there. The end result is that all the processes executed sequentially rather than simultaneously.

Might be worth a shot.

Martin

[dludwig]

I haven't implemented this yet since we are only on FP2, but there is a new feture in 9.4.1 FP3 which I think will solve the chore roll back issue...here is the info from IBM support...

There is a new parameter called Bulk Load Mode in 9.4.1 FP3.

If you upgrade to 9.4.1fp3 this parameter would work for them and would lock out users until TI process has completed.

BLM recreates the (write) lock model of TM1 Version 9.0:
The user executing a write operation sets a global write lock which forces all other users into the wait state.

The BLM is enabled by wrapping a TI process, using as first TI function in the Prolog EnableBulkLoadMode() and as last TI function in the Epilog DisableBulkLoadMode().

For detailed information see for instance the TM1 9.4.1 FP3 Release Notes, Chapter: "Documentation Updates for TM1 9.4.1 FP3", Paragraph: "Using TM1 in Bulk Load Mode", pages 11 to 12.

[macsir]

Hi, all

We have this issue recently as well and I believe it is because of contention locking. The only thing I can do is to rearrange them to give the main chore enough buffer to finish.