SaveDataAll PA 2.06 Deadlocks

[npatel27]

Hi all - we have been experiencing some strange behavior with our install of PA 2.06 where a typical SaveDataAll function in a TI process appears to be conflicting with normal user operations (i.e. viewing data in a cube / refreshing DBRW's in reports, etc..) on occasion. The SaveDataAll is called in its own TI process by itself with no other operations in the process at all.

If you look at the data directory, the cub$ files start to get created as the SaveData is committing data but it rolls back and starts all over. After digging into the server log with some exception logging turned on, we noticed that thread 205612 (savedata) was in conflict with a user thread 207636 (he was just idle at that time). Have you come across this type of situation before? Do you have any recommendations on how to ensure that the SaveData runs without being interrupted by other activity?

205612 [2f25] DEBUG 2019-07-18 20:08:12.955 TM1.Lock.Exception Deadlock detected. Thread '207636' is blocked by thread '205612'.
205612 [2f25] DEBUG 2019-07-18 20:08:12.955 TM1.Lock.Exception Deadlock detected. Thread '205612' is blocked by thread '207636'. Thread '205612' will rollback.'
205612 [2f25] DEBUG 2019-07-18 20:08:12.955 TM1.Lock.Exception Deadlock detected locking object 0x000001055E00F4E0(0x000001055E00F410) by thread 205612 waiting for current readers to drain.

[paulsimon]

Hi Neil

You said that the SaveDataAll is in a process by itself, but is that process being called by other processes? If there is a top level process then any locks obtained by it or other processes that it calls will only be released when the top level process completes.

I would recommend putting the SaveDataAll into a standalone process and running this from a separate Chore overnight. Do not include it in any other overnight Chore that you may have.

In Planning Analytics there seems to be a lot more conflict between SaveDataAll than there was in older versions of TM1.

Not doing a SaveDataAll immediately after a load where you typically turn logging off does expose you to more risk if the server suffers a power outage or fails for some other reason. However, later versions of TM1 and Planning Analytics are a lot more stable than earlier versions. The only failure we have had was when an IT guy got confused about which box he was working on and accidentally turned off the Prod box by mistake. All the entered data was automatically recovered from log files when the server restarted, but I needed to work out which processes to run to complete the recovery which took about 3 hours. However, that was a one off.

By comparison if you do start trying to run a SaveDataAll alongside user activity during the day, you might actually end up triggering the crash that the SaveDataAll is designed to protect you from.

You can cut things down by saving individual cubes, but that can still cause the sort of locking contention issue that you describe.

It is not a perfect situation. Certainly when I was writing processes years ago I always put a SaveDataAll in the Epilog after a bulk load when I had turned logging off. That approach was OK when things were single threaded, but it is not the recommended approach now.

Regards

Paul Simon

[npatel27]

Thanks Paul - the SaveDataAll process is being called from Cognos Command Center where we perform a series of tasks after the TI process is complete outside of TM1 such as zipping/backup of our data directories. From my understanding, Command Center performs a REST call to trigger that TI process. I will certainly try the approach of putting this in a chore by itself and see if that helps. I just find it odd as older versions such as 10.2.2 FP7 did not encounter such conflicts with SaveDatas.

In terms of prevent a collision with other user activity, have you ever tried using a function such as bulkloadmode to suspend activity while the Save is running (although not the best idea during peak periods)?

[Wim Gielis]

What we tend to do in the bigger TM1 models, is 2-3 save data operations. 1 in the morning after nightly chores updating metadata and loading data. Then after business hours (8 PM seems fine in general). Then some other chores run and we have a savedata chore around midnight, 7zip for daily model backup and finally we are ready for the nightly chores again. All 3 save data are chores with only that - and a simple TI check process to stop chores immediately if they would run outside a certain time window. Sure there is the possibility of a crash during business hours, at which manual data entry will be recovered yet the TI’s that users run need to be rerun in general.

[paulsimon]

Hi

If your system does not need to be up 24 hours a day, then you could do something similar to our approach.

We use a Windows Scheduled Task which needs to run under an Admin Profile with Highest Privileges. This task runs a DOS Batch File. The Batch File checks to see if the TM1 Service is running and if so it shuts it down and loops until it has actually shut down. It then takes a backup using 7ZIP. It then copies in any files to be promoted. It then restarts the service. The service has a Startup Chore which amongst other daily tasks, also runs any processes necessary to complete the promotion.

We therefore have no real need for a SaveData, since the act of shutting the Service down effectively does a SaveData. We still do actually have a SaveDataAll run from a Chore at around 10pm but that is just a belt and braces approach.

Not sure about bulk load mode. I have never really needed to use it.

Regards

Paul Simon