TM1 server load failed - Persistent Feeder error

[Rams]

Hi Experts,

I have an issue in loading the TM1 server after the scheduled chore to restart TM1 server at 3AM, because of the below error server load failed.

Code: Select all

3832   []   INFO   2013-03-17 16:10:24.770   TM1.Cube    Loading body for cube }ElementAttributes_Group_Scorecard
3832   []   ERROR   2013-03-17 16:10:24.770   TM1.Server   ***** Persistent Feeder error *****
3832   []   ERROR   2013-03-17 16:10:24.770   TM1.Server   Feeder file error/inconsistancy detected for cube "}ElementAttributes_Group_Scorecard" reason:
Have processed feeder files but this cube does not have a feeder file
3832   []   ERROR   2013-03-17 16:10:24.770   TM1.Server   **********
3832   []   ERROR   2013-03-17 16:10:24.770   TM1.Server   All persistent feeder files will be deleted and server will shut down
3832   []   ERROR   2013-03-17 16:10:24.770   TM1.Server   Restart server to re-load database.
3832   []   ERROR   2013-03-17 16:10:25.005   TM1.Server   Phase 1 of server load failed

The error points out "Persistent Feeder" but I've chore that runs at 2AM to delete all Persistent Feeders before the server restart. Interestingly server gets loaded without any error if I manually kick start. This is happening only for last 2 weeks, the Group_Scorecard has in the system for longtime and no major changes has been done in the server side recently.

The only thing I suspect is a chore which updates the dim hierarchies at 1AM and in last 2 weeks it throws "circular reference" error will that affect the server load? If anybody come across this issue please throw some light.

TM1 Version: 9.5.1

Any inputs are welcome!! Thanks!

[rmackenzie]

There's a known issue (definitely in 9.5.1) where if that error occurs at start-up (file inconsistency) then the server fails to load objects in the correct order. You can see this yourself by following the log - an attribute cube may get loaded before its dimension causing all reporting based on those attributes to fail, etc. Stopping and re-starting the instance after this faulty start-up solves the problem, as you note.

but I've chore that runs at 2AM to delete all Persistent Feeders before the server restart

It sounds like you really need to test if that is happening - is it in the chore where a process is throwing a circular reference error? It may be that you need to run a one-liner process with the DeleteAllPersistentFeeders command on its own rather than in another TI, or part of a faulty chore.

[Rams]

First of all thanks for your reply!!!

It sounds like you really need to test if that is happening - is it in the chore where a process is throwing a circular reference error? It may be that you need to run a one-liner process with the DeleteAllPersistentFeeders command on its own rather than in another TI, or part of a faulty chore.

Yeah its working fine and deleting the feeder files too, 2AM chore has only one liner with "DeleteAllPersistentFeeders" and thats not part of the circular reference chore error, which is coming from 1AM chore where it updates account and cost centre hirerachy. Anyhow I had a similar issue in the other TM1 instance, once after I resolved the circular refernece error, that works well. So i did the same for this and waiting for the Monday restart to go smooth.

I've raised a PMR for this with IBM, and their first approach is to solve the circular reference error, once I did it works well so far . But that shouldn't be the case to fail the server load for sure .

[lotsaram]

Surely if you are doing a nightly service restart at 2am WITHOUT persistent feeders and doing a workaround to delete persistent feeders why don't you just run the server without the persistent feeders switched on in the first place?!

[Rams]

why don't you just run the server without the persistent feeders switched on in the first place?!

Thanks lotsaram!! I changed the config file to switch off the persistent feeders. I hope this would resolve for next Monday.

Instead of changing the config file, thought of deleting the feeders first and let the server starts up with fresh feeders to resolve the error like below order, but that's not working.

1AM >> Hierarchy update chore ----> it has some circular reference error, now it been resolved
2AM >> DeleteAllPersistentFeeders ----> to resolve the feeder error which throws in a log before the server restart
3AM >> Server Restart ------> when getting restart it got stuck in feeder calculation for "}ElementAttributes_Group_Scorecard"" cube and failed to load.

No hope even after I resolve the circular reference & delete persistent feeder; the server failed y'day to restart after Monday 3AM run.