Chore "Done" but fails to actually finish

[ryanr]

I know the subject sounds like a complaint from my wife… but I know exactly why that happens, but this TM1 issue has me stumped!

We are running 9.5.2. FP2 .One of the chores that runs every night, locked up my dev sever yesterday and my production system today. My tm1sever.log looks something similar to this normally(I have condensed it quite a bit for readability)…

TM1.Chore “Load Assets” executed by scheduler
TM1.Process 1 executed by Chore Load Assets
TM1.Process1 finished executing normally, time 100 seconds
TM1.Process 2 executed by Chore Load Assets
TM1.Process2 finished executing normally, time 200 seconds
TM1 Chore “Load Assets” time = 300 seconds
TM1.Cube.Dependency Adding cube dependency : Cube ‘Calendar’ depends on ‘}ElementAttributes_Day’
TM1.Cube.Dependency Adding cube dependency : Cube ‘‘}ElementAttributes_Per’ depends on ‘}ElementAttributes_Day’
TM1 Chore “Load Assets” finished executing

But now it shows all but the last line….


TM1.Chore “Load Assets” executed by scheduler
TM1.Process 1 executed by Chore Load Assets
TM1.Process1 finished executing normally, time 100 seconds
TM1.Process 2 executed by Chore Load Assets
TM1.Process2 finished executing normally, time 200 seconds

TM1 Chore “Load Assets” time = 300 seconds
TM1.Cube.Dependency Adding cube dependency : Cube ‘Calendar’ depends on ‘}ElementAttributes_Day’
TM1.Cube.Dependency Adding cube dependency : Cube ‘‘}ElementAttributes_Per’ depends on ‘}ElementAttributes_Day’

*It is usually only about a 10 second delay between the last 4 lines…

My TM1 top is black and shows nothing when I open it, no “sever not available message”, just nothing… if I try to log into the server from perspectives, it just freezes while trying bring up the login dialog box…

CPU on the sever is basically 0%, and I have plenty of RAM & disc space.

I ended up restarting the DEV server yesterday, but hadn’t been able to test this chore on it yet after the restart. – I honestly thought it was some of the changes I made in DEV yesterday.

Anyone have any thoughts, I’m not exactly sure where to start looking? Thanks in advance!

[tomok]

I honestly thought it was some of the changes I made in DEV yesterday.

Do you have the option "LetsRandomlyMessWIthTheDeveloper" set to F in the tm1s.cfg file? I'm no genius but the first thing I would do is run the chore again, over and over, to see if it was just an odd occurrence or if I royally hosed something up. Only then would I start going back through my changes to try and figure something out.

[SueZhaZa]

nice parameter...

[foogy]

But now it shows all but the last line….
[...]
My TM1 top is black and shows nothing when I open it, no “sever not available message”, just nothing… if I try to log into the server from perspectives, it just freezes while trying bring up the login dialog box…

CPU on the sever is basically 0%, and I have plenty of RAM & disc space.

I ended up restarting the DEV server yesterday, but hadn’t been able to test this chore on it yet after the restart. – I honestly thought it was some of the changes I made in DEV yesterday.

We've had and still have from time to time the same problem. If it happens, then it happens during the nightly master data update. Our release is also 9.5.2 FP2. Just upgrade DEV to FP3, let's see if that helps.

This is so frustrating because this kind of system freeze is not reproducible. Once frozen and restarted, the same process finishes without any problem again and again. The process includes lots of security assignments which are executed in a loop and also lots of attribute assignments, also in a loop. Do not ask me why, but I have them in suspect. Unfortunately, it could take weeks until the system freezes during the night.

So, if anyone ever got an idea on such freezes, please share them.

KR,
foogy.

[tomok]

The process includes lots of security assignments which are executed in a loop

Are you calling the function ElementSecurityPut or just writing to the cube with a regular CellPutS. I would not recommend using ElementSecurityPut in a nightly update. That's because it has a lot more overhead in it, namely updating the Security after each record. Better to do all your security changes with CellPutS and then call SecurityRefresh once, at the end of the routine, when all security updates have been done.

[declanr]

When you mention no longer being able to log on but resource being available... do you enable bulk load mode at some point in there? If so it is possible the process isn't releasing it at the end.

[foogy]

Thanks for your suggestions and recommendations and sorry for my late feedback.

I've checked your suggestions, but unfortunately none of them apply (no bulk load mode, securityrefresh only at end of process and CellPutS otherwise).
As workaround, we've now implemented a nightly TM1 server restart, followed by the master data update. We did not have any system freeze after the master data update so far.

Not a solution I prefer or recommend, but to honestly, I almost gave up here. May be a future release will get rid of this problem.

Again, thanks for your support.
foogy