Memory usage gradually increased to maximum (64 GB) after restarting the tm1 service

[faisal]

Hello,

I have an issue about memory usage continues to increase gradually throughout the day.
I have set PersistentFeeder=T, MTQ=1 and I do not set MaximumCubeLoadThread on tm1s.cfg (so I assume the cube loading and loader processing is not multi-threaded).

Has anyone encountered this problem and knows what causes memory usage to gradually increase after restarting the TM1 server?

For your information, I am using TM1 with product version 11.8.00900.3

Thanks in advace!

Regards,
Faisal

[gtonkin]

I would say that this is expected behaviour.

Even if you have set tm1s.cfg parameters to load sets and views, I would expect that opening views where TM1 needs to recalculate would increase memory usage (assuming no memory to recycle from garbage).
If you are running processes, these would consume memory too as the metadata and data would be held in memory before being committed to the dimensions/cubes and then released (to garbage not OS).

Is there something specific you are concerned about that is increasing memory or just general increasing throughout the day?

[faisal]

Hi Gtonkin,

Thanks for the reply.

What can be said if it is strange because memory usage increases gradually even when there is no user activity (midnight). This has never happened before because I checked memory usage from July to mid-August, memory was between 60-68% and now it can consume all the memory I have on the server in a day.

I wonder if this is affected by the computing feeder during TM1 server restart? So there may be an unintentional over feeding rule that applies. If so, how do I determine which rules/feeders I need to adjust? Is there an efficient way to define enhancement rules/feeders on cube?

Previously, I started Performance Monitor and checked }StatsByCube, but I don't know how to make sure whether the cube using the most memory needs to be tuning due to over feed.

Regards,
Faisal

[gtonkin]

Once started up, your feeders would have fired and memory usage should be based on the data and the feeders related to them.
Only when data changes would you expect more memory to be used due to additional areas being fed.
"data changes" is ambiguous here as there is a concept of going from undefined to a value and then from one value to another e.g. CellIncrementN.
Depending on your tm1s.cfg and parameters like ForceReevaluationOfFeedersForFedCellsOnDataChange, memory may or may not change depending on the changes in data and cells impacted.

Tuning feeders is an art and there are lots of articles out there dealing with the topic.
Try a search here to being with.

[Steve Rowe]

You should be clear about what RAM is increasing.

If the server is maxing out are you sure the TM1 DB is the cause? Can you see what is using this RAM?

I ask this because I recently had an issue at a two different clients where the RAM on the box was maxing out but not visible vs anything in task manager.
One issue was due to an old windows driver doing something funky in the background.
The other due to a script being called by PAAA not stopping properly leading to many 1,000 powershell scripts open in the background.

You really need to understand the problem in detail before you look for solutions....

[faisal]

Once started up, your feeders would have fired and memory usage should be based on the data and the feeders related to them.
Only when data changes would you expect more memory to be used due to additional areas being fed.
"data changes" is ambiguous here as there is a concept of going from undefined to a value and then from one value to another e.g. CellIncrementN.
Depending on your tm1s.cfg and parameters like ForceReevaluationOfFeedersForFedCellsOnDataChange, memory may or may not change depending on the changes in data and cells impacted.

Tuning feeders is an art and there are lots of articles out there dealing with the topic.
Try a search here to being with.

Thanks for the article gtonkin.

I checked the article from the link you gave me and also tracked the .feeders file in the Data folder. The .feeders file did not increase as much as I assumed (only increased by a few KB per day), which proves to me that the increase in memory usage must be caused by something else because as you said before, only When the data changes I should expect more memory to be used due to additional powered areas, which is strange since the only time data changes can be made is during office hours (when users upload data).

In the meantime, I'm trying to replicate the issue on my development server and the production server's backup server. The tm1 application memory usage issue only occurs on the backup server. For your information, I used the same backup data from production server on the development server and the backup server.

I tried comparing tm1s.cfg and tm1s-log.properties, the only difference between the development server and the backup server is port number and AdminHost parameter (in which I set AdminHost on development server, but not on backup server production server because it uses localhost).

After almost a month of troubleshooting this issue, I'm starting to wonder what else could be causing the tm1 application's memory usage to gradually increase until it consumes all available memory on the server. One more thing I noticed, I have to restart the tm1 service daily to restart memory usage on the production server. Usually the tm1 service status is stuck for hours and if I check the tm1server.log file the last logged process is "terminating unregistered cleanup thread".

I'm wondering if memory usage could be tied to thread of tm1 application, thus affecting the termination of service process tm1?

As I have checked regularly on PAW admin there is no pending process in the thread section.

[faisal]

You should be clear about what RAM is increasing.

If the server is maxing out are you sure the TM1 DB is the cause? Can you see what is using this RAM?

I ask this because I recently had an issue at a two different clients where the RAM on the box was maxing out but not visible vs anything in task manager.
One issue was due to an old windows driver doing something funky in the background.
The other due to a script being called by PAAA not stopping properly leading to many 1,000 powershell scripts open in the background.

You really need to understand the problem in detail before you look for solutions....

I checked Task Manager as well as Resource Monitor and the highest memory consumption was the tm1sd.exe application. It continues to grow for a day until it consumes all available memory on the server, which can sometimes cause the service down.

This happened once, so to prevent this from happening again until I figure out what's causing the tm1 app to use memory, I need to restart the tm1 service (or even the server) everyday so memory of tm1 service restarted.

I cannot detail the memory usage that tm1sd.exe uses because on Resource Monitor the only tm1 process running is tm1sd.exe.
Do you have a way to get detail information of the memory consumption of the tm1 service?

[WilliamSmith]

Do you notice that the memory usage spikes when a view or MDX query is run against a cube that includes rules/feeders?

[MarenC]

the other due to a script being called by PAAA not stopping properly leading to many 1,000 powershell scripts open in the background.

I am sure I have also seen this, out of interest Steve, how did you resolve this?

I once remember that out tm1server would go down because one of the subsets was referencing an element that no longer existed. I have not seen this issue in any of the latest versions but might be worth checking.

Maren

[faisal]

Hello everyone,

Sorry for being away for a few days.

In my absence, I tried to track down the issue and execute a test scenario to validate my hypothesis. The result is good and the problem is now resolved. I just moved the tm1auditstore that was causing this problem outside the data folder.

The reason the server memory gradually increased was because the TM1 application was unable to read and retrieve the tm1auditstore when the TM1 service started.s. It's so silly that this error occurs every time I start the TM1 service without me even noticing. The TM1 application cannot read and retrieve tm1auditstore because it has a strange datemodified year (January 1, 1980) while the name of tm1auditstore includes the actual year.

I don't know why tm1auditstore has 1980 as the datemodified year, it seems like the file is corrupted. I asked IBM support but there is not enough evidence as to the cause of this issue.

If you have any ideas about this anomaly, please share with me.

Thank you.