Failed to open temp file for processing tm1rawstore.<DateTime>

[faisal]

Hello,

I have a monitoring message log on our tm1 production service for a few weeks now and I get a recurring error from TM1.Audit.PostProcessor with message "Failed to open temp file for processing tm1rawstore.<DateTime>". It follows with error from TM1.Server with message "sf_DeleteFile: Error on file "s:\tm1data\horizon\data\tm1rawstore.<DateTime$>". Error = error code:32 reason:" The process cannot access the file because it is being used by another process."

This error first occurred on Sunday, August 20, 2023 at 03:53 am and is still happening to this day. I get these error messages after the TM1.Chore process of savedata is running. But at the end of the TM1.Chore process, it will always show message from TM1.Server, "sf_SafeDeleteFile() SUCCEEDS after 1 retries, "s:\tm1data\horizon\data\tm1rawstore.<DateTime$>""

One more thing, this tm1rawstore error entails tm1 application memory usage. It is gradually increasing and there is no sign that it will stabilize at any point like before. So in this situation I have to restart tm1 server every day.

Another fact, whenever I try to restart tm1 server, tm1 server's service status is still stuck in stopping state. I check windows services and command prompt using sc queryex syntax and there are no other tm1 servers running on my production server (because there is only one tm1 server running on my production server) . So to fix the problem I rebooted the machine so I could restart the tm1 service.

I compared the tm1s.cfg and tm1s-log.properties files of my save data before and after this error occurred. The only difference is in the last tm1s.cfg, I add RawStoreDirectory to specify tm1rawstore folder on tm1s.cfg and reduce memory size from 2000 MB to 10 MB on tm1s-log.properties as IBM recommends, but the problem persists.

I've been working on this issue with IBM for 19 days and need more insight if anyone is experiencing the same problem as me.

What causes this error to appear in the message log? and how did I resolve this error?

For your information we are using TM1 version 2.0.910.5 on our production server.

[gtonkin]

The error typically indicates that another process has the file locked.
Have you ruled out antivirus or other program locking the file to scan it?
ProcessExplorer can assist in telling you what process is locking.

Your service stuck in stopping state could be that it is just taking too long to shut down.
Does it shut down eventually?

How much memory is it using (commit size) vs available RAM?
Once you start using the paging file, shutdown takes exponentially longer.

[faisal]

Hi gtonkin,

I have excluded antivirus software from the installation, data, and log folders. I monitored the resource monitor of the tm1rawstore file and no other application was accessing the file except the tm1 application itself.

I checked on Windows Services to see if there were any other TM1 servers running on the same machine, but as I mentioned before, there was only one TM1 server running on my production server.

IBM has suggested enabling the log4j.logger.TM1.Audit=DEBUG and log4j.logger.TM1.Audit.PostProcessor=DEBUG settings but there is no guarantee that this will get us any further.

No, it didn't stop. It was stuck on stopping state for almost an hour, which is strange because normally shutting down the tm1 server takes less than 3 minutes.

The current memory used at the time of reply to this message is gradually increasing and is currently 53 GB out of 64 GB available on the server. I have to restart the server every day for the memory to drop again.

Regards,
Faisal

[faisal]

Hello,

Got good news from me.

The tm1rawstore error no longer occurs on our production servers. I discovered on our backup server, with the exception of DFS Management which I had previously disabled, there was a task scheduler that copied files from the production server to the backup server still running.

Then I turned off task scheduler and restarted the server last night. It's been 12 hours since the TM1 server restarted and tm1rawstore error is no longer happening. I will check periodically for a week, just in case the error comes back.

Regards,
Faisal

[Wim Gielis]

Thanks for providing the resolution.
I have never put that directory inside the TM1 data directory.
Just like the TM1 logging directory, never located inside the TM1 data directory.

[tomok]

Hello,

Got good news from me.

The tm1rawstore error no longer occurs on our production servers. I discovered on our backup server, with the exception of DFS Management which I had previously disabled, there was a task scheduler that copied files from the production server to the backup server still running.

Then I turned off task scheduler and restarted the server last night. It's been 12 hours since the TM1 server restarted and tm1rawstore error is no longer happening. I will check periodically for a week, just in case the error comes back.

Regards,
Faisal

FYI, the logging settings that produce the tm1rawstore file is really intended for debugging purposes only. It captures almost every single user interaction with TM1, beyond just updating data and dimensions. It can produce massively sized log files, filling up your server storage in pretty short order if you aren't careful.