Chore getting stuck to same process everyday.

[SBK88]

Hi All,
Need ur help.
(TM1 model size approx. 90 GB)
We have a chore (scheduled daily for the night run), which usually takes 8-10 hours to get completed.
Now after completing around 70% of the processes it is daily getting stuck to one of the process.
I have tried running that particular process manually, its working fine.

2 months back also we had similar issue, that time the problem was with the free space in server (disc in which data server resides) it was less than 100 GB, after deleting all unnecessary files n folders everything was fine.
Now although we have around 240 GB of free space, and still facing the similar sort of issue.
Anybody having any idea.

Thanks in advance.

[TrevorGoss]

Hello,

Do you turn logging off when you run this process?

It may well be the case that huge logs are being created whilst the process runs over night, which then fills up the disk and brings down your service.

Thanks,

Trevor.

[Alan Kirk]

Hi All,
Need ur help.
(TM1 model size approx. 90 GB)
We have a chore (scheduled daily for the night run), which usually takes 8-10 hours to get completed.
Now after completing around 70% of the processes it is daily getting stuck to one of the process.
I have tried running that particular process manually, its working fine.

2 months back also we had similar issue, that time the problem was with the free space in server (disc in which data server resides) it was less than 100 GB, after deleting all unnecessary files n folders everything was fine.
Now although we have around 240 GB of free space, and still facing the similar sort of issue.
Anybody having any idea.

You really haven't given anyone much to go on. There's no indication of:
- What this chore is doing (whether it's uploading data from another source or merely processing data in the cubes, or something else for that matter),
- if it's uploading data, then what the data source is (if it's an ODBC connection, for instance, have you looked into the possibility that something is stopping things on the source side?),
- How many processes are involved,
- What the process it's getting stuck on is actually doing,
- Whether the processes are being called directly from the chore or by ExecuteProcess commands,
- What's in the server log files...

Pretty much everything that's requested in the Request for Assistance Guidelines isn't in this post.

You're assuming that a "similar issue" (how similar, was it the same process?) from a few months back has any bearing on this, which is the kind of thing that leads to plenty of wild geese being chased.

Trevor's suggestion is worth checking out (exactly how big is the log file at the point of hanging?), though, but obviously the server log should have been your first port of call.

Also it would not be a bad idea to have TM1Top tracking and logging the overnight activity to see if there is something in particular that is happening at the point of hanging.

One last suggestion; 8-10 hours is a very, very long time. If it is at all possible to break up the processes (that is, if they aren't dependent on each other) it may well be worth trying to run them in parallel. There is a tool called Hustle which is linked to in this thread which made a remarkable difference to upload times in a new implementation that we're doing. The less time a process runs, the less chance it'll go pear shaped.

[TrevorGoss]

Hello Alan,

One last suggestion; 8-10 hours is a very, very long time. If it is at all possible to break up the processes (that is, if they aren't dependent on each other) it may well be worth trying to run them in parallel. There is a tool called Hustle which is linked to in this thread which made a remarkable difference to upload times in a new implementation that we're doing. The less time a process runs, the less chance it'll go pear shaped.

Indeed, I can just imagine everything waiting behind it in frustration. This Hustle tool...I could not find anything to do with this tool in the thread link you provided, sorry if I missed it.

Does this Hustle tool stop the Wait thread situation you see in Ops Console?

Thanks.

Trevor.

[Alan Kirk]

One last suggestion; 8-10 hours is a very, very long time. If it is at all possible to break up the processes (that is, if they aren't dependent on each other) it may well be worth trying to run them in parallel. There is a tool called Hustle which is linked to in this thread which made a remarkable difference to upload times in a new implementation that we're doing. The less time a process runs, the less chance it'll go pear shaped.

Indeed, I can just imagine everything waiting behind it in frustration. This Hustle tool...I could not find anything to do with this tool in the thread link you provided, sorry if I missed it.

My apologies, that was the Request For Assistance Guidelines link. Unfortunately my pitchin' arm is getting sore from throwing it tonight. I've corrected the link in the previous post and will repeat it here for convenience.

Does this Hustle tool stop the Wait thread situation you see in Ops Console?

It's not so much that it stops waits; nothing can do that if there are dependencies during the load. But it means that we can load multiple years of data concurrently rather than sequentially, so it gets the job done way faster. As for Ops Console; 10.2.2 is something that we're only transitioning to, I'm still primarily on 9.5.2. But even when we do I don't plan to be using Ops Console. My disdain for the Java apps that come with the 10.x series is well known (maybe less for Ops Console than some others, and I reserve a special brand of loathing for Cognos Configuration), and more to the point we have a licence for Cubewise Pulse. It's not the cheapest tool on the market but is far and away better than anything that has come out of the IBM stable. Between that and MS Powershell, I find that even on 10.2.2 it makes using Ops Console a pointless exercise.

[TrevorGoss]

My apologies, that was the Request For Assistance Guidelines link. Unfortunately my pitchin' arm is getting sore from throwing it tonight. I've corrected the link in the previous post and will repeat it here for convenience.

That's quite alright, the link now works, thanks.

But even when we do I don't plan to be using Ops Console. My disdain for the Java apps that come with the 10.x series is well known (maybe less for Ops Console than some others, and I reserve a special brand of loathing for Cognos Configuration), and more to the point we have a licence for Cubewise Pulse.

I am quite the fan of Ops Console, but thats not to say it has weaknesses as well as my only tool of the same nature before this was TM1 Top. If by "Cognos configuration" you mean the IBM Cognos Configuration tool, then indeed, it is very buggy, clunky...you could even say typical Java lol.

Thanks for your help, I will look into Hustle and see what it can do for us.

Trevor.