Server freezes in SaveData

[upali]

Hi all,

We've got this very strange behaviour in our Windows Prod box, which is not apparent in the Dev box.

Prod: Windows Server 2012, TM1 10.2.2.4, 1.5Tb RAM, 2Tb D:\, 32Cores
Dev: Windows Server 2012, TM1 10.2.2.4, 512Gb RAM, 1Tb D:\, 16Cores (x2 threads)

TM1 Models uses about 450-490Gb of RAM. C:\ Only Windows and System software. D:\ TM1 application and Models.

We are noticing that during SaveDataAll (or CubeSaveData) the server freezes after the following steps of operation.
Saving X.CUB
- Create X.CUB$ and start writing Transaction log entries
- Complete X.CUB$ - All fine
- Delete X.CUB
- Rename X.CUB$ to X.CUB – I’m not sure of the order of the last two steps, but the server freezes during the delete operation.

The bigger the CUB or FEEDER files, the loner the time the server takes to "get back". For example, a 50Gb CUB take about 10mins to write to disk, but another 6-10mins of freeze, during which all access to the server is lost (i.e. no RDP, can't remotely browse the folders, no FTP etc...).

We have excluded the TM1 Application and Data directories from Antivirus - So that is ruled out. If we load the same models into Dev and perform the Save operations, everything is fine. The big boy Prod has the issue, while 'lil Dev works fine on identical software.

I would like to know if you have seen similar issues (and how they were solved), especially if there is some contention of resources between RAM, CPU and HDD during a TM1 Save operation. Any recommendations on the application setup?

Thanks...

[BrianL]

While I haven't heard of this exact problem, is this network storage that's being written to in prod? Theoretically at least the large file write could saturate that network connection and prevent any of those operations you're describing. Maybe IT could check if a direct console login is also frozen during this time?

[upali]

Yes Brian... Storages are in SAN, and we have done diagnostics on Prod and Dev. During the freeze no connections are available to the server, unfortunately.

[yyi]

Hmm, I wonder if the x2 threads have something to do with it.

[PlanningDev]

Have you looked to see if you have any Anti Virus Software or any specialized logging software? I know its not TM1 but we had an issue with IBM Watson Content Analytics where some kind of logging program would get triggered when we crawled data for a collection and the server would attempt to build log files that were hundreds of gigs in size and essentially bring down the server.

[upali]

Have you looked to see if you have any Anti Virus Software or any specialized logging software? I know its not TM1 but we had an issue with IBM Watson Content Analytics where some kind of logging program would get triggered when we crawled data for a collection and the server would attempt to build log files that were hundreds of gigs in size and essentially bring down the server.

AV has an exclusion on all TM1 related directories, an no other logging programs are installed. We have also raised an incident with IBM, but still no solution.