TM1 server memory usage jumped from 110GB to 150GB

[gshock888]

What would be the reason that my tm1 instance, currently at 100GB running on an AWS EC2 R3 Windows instance (244GB memory), when copied to another R3 instance for testing, (exact same environment, same OS, same configuration and same TM1 version) that it jumps to 150+GB?

It just seems like my TM1 is really awful at handling memory, it grew from 60GB to 110GB with little to no development changes promoted to our production box in the past month and it's now growing at an alarming rate by us just copying the tm1 instance to another windows instance.

[Alan Kirk]

What would be the reason that my tm1 instance, currently at 100GB running on an AWS EC2 R3 Windows instance (244GB memory), when copied to another R3 instance for testing, (exact same environment, same OS, same configuration and same TM1 version) that it jumps to 150+GB?

It just seems like my TM1 is really awful at handling memory, it grew from 60GB to 110GB with little to no development changes promoted to our production box in the past month and it's now growing at an alarming rate by us just copying the tm1 instance to another windows instance.

Have you turned on Performance Monitor? Looked at the memory used by cube, the memory used for feeders, etc to determine where the differences are?

Also remember that TM1 calculates values dynamically; memory won't be used until it needs to be used. Consequently the startup memory is immaterial compared to the "been running for a while" memory. Once people start calculating values in the model the calculated amounts will be cached and use more memory.

Similarly it's generally meaningless to compare one instance of TM1 to another except right at the moment of startup and even then that presupposes that there aren't any differences in configuration like the use / non-use of persistent feeders.

[TrevorGoss]

Hello,

could it also be the case that your Server that contains these TM1 windows services has been running for a while without a Shutdown/Restart? This happens to us somtimes, allthough not on the same scale, we run low on RAM on our development box when we keep shutting and starting a TM1 service and see our available RAM become lower and lower but, when we get windows patch updates, the server restarts and the RAM has calmed down again.

[gshock888]

I'm bumping my thread - the usage of ram is getting ridiculous now that it's been another month.

my server is a AWS based R3 instance that has 244GB RAM on Windows 2012 R2. The RAM usage is now 200+GB with 130GB of Garbage Memory as reported by StatsForServer cube. Statsforcube confirms the total memory usage by cubes is only 70GB, which is a lot but should be more than manageable for my 244GB instance. How do I tell TM1 to give some of the 120GB garbage memory back? if this goes up any further it will start to eat into pagefile.sys. This will definitely degrade performance, if not TM1, then for other services in the windows server that requires to use some RAM.

[TrevorGoss]

Hello again,

have you tried restarting the Server that contains all the TM1 windows services? This restart will cause all RAM to be released.

How do I tell TM1 to give some of the 120GB garbage memory back?

You can go to the }CubeProperties cube and set the VMM and VMT cells to the amount of memory it can hold. This link will explain more:
https://www.ironsidegroup.com/2014/06/0 ... explained/

This cube will allow you to select memory usage on a cube by cube basis, but yes....make sure you have performance monitor on, but also..be aware of logs, it would be awful if you used all your energy on trying to sort out memory...and then disk space ran out!

Trevor

[lotsaram]

I'm bumping my thread - the usage of ram is getting ridiculous now that it's been another month.

my server is a AWS based R3 instance that has 244GB RAM on Windows 2012 R2. The RAM usage is now 200+GB with 130GB of Garbage Memory as reported by StatsForServer cube. Statsforcube confirms the total memory usage by cubes is only 70GB, which is a lot but should be more than manageable for my 244GB instance. How do I tell TM1 to give some of the 120GB garbage memory back? if this goes up any further it will start to eat into pagefile.sys. This will definitely degrade performance, if not TM1, then for other services in the windows server that requires to use some RAM.

Ouch. I wouldn't want to be paying for an AWS instance with 244GB of memory. Cloud still isn't really set up for large in-memory applications, as soon as you start needing more than a "standard" amount of RAM the financials of cloud rapidly disintegrate vs on premise.

If you have 70 GB of memory in garbage this usually comes from one of 2 possibilities. The first and most likely being multi-threaded server load without persistent feeders, this can generate excessive amounts of garbage memory. Solution: make sure PersistentFeeders=T in tm1s.cfg save data and restart the service.
Another possibility is a long running TI process with transaction logging on or exporting data to text file (reason the entire export file is held in memory before the final write to disk in the commit phase of the TI). Solution: turn of transaction logging during loads, break large exports up into smaller jobs.

But also understand you CAN'T make TM1 release it's internal garbage memory. The only way to release this is to restart the service. If you want TM1 to come up without the garbage memory either use persistent feeders or set MaximumCubeLoadThreads=0.

[BrianL]

Depending on what version you're running, another potential culprit is ViewZeroOut of large views. Pre-10.2 zero'ing out data would never shrink the cube data AND would require a temporary copy of the entire zero'd out view that gets released back to "garbage" when the operation commits.

The only way to get TM1 to "release" garbage memory back to the OS, is to restart TM1.

[gshock888]

I opened a ticket with IBM, and all of you including the tech support is saying restarting will release some memory, which makes complete sense but in reality every restart it only takes up MORE RAM. I'm at 200GB now, I bet restarting now would surely bring it up to 210GB.

I'm going to try destroying old snapshot scenarios and see if that helps. If so I would just have to archive them.

Honest question, coming from Cognos Planning, TM1 is a blessing with large cubes. I was told to not worry about the size of my dimensions, cubes when I made the move to TM1. Was I given bad advice? How are you guys all coping with memory usage for large applications? Let's say if my application finally goes above the 244GB RAM limit and now tm1sd.exe start eating into pagefile, how bad would performance get? On this R3 AWS instance I do have a 1TB SSD EBS volume attached to it, and I can always spin up more EBS volumes for pagefile purpose, but no matter what IOPS I configure I reckon it would never be as fast as RAM.

[lotsaram]

I'm fairly sure that your problem is a simple one and recommend you get an experienced consultant to check you model for a day to resolve it.

I am also 99.99% sure that this is not a IBM support issue.

[kangkc]

I opened a ticket with IBM, and all of you including the tech support is saying restarting will release some memory, which makes complete sense but in reality every restart it only takes up MORE RAM. I'm at 200GB now, I bet restarting now would surely bring it up to 210GB.

I'm going to try destroying old snapshot scenarios and see if that helps. If so I would just have to archive them.

Honest question, coming from Cognos Planning, TM1 is a blessing with large cubes. I was told to not worry about the size of my dimensions, cubes when I made the move to TM1. Was I given bad advice? How are you guys all coping with memory usage for large applications? Let's say if my application finally goes above the 244GB RAM limit and now tm1sd.exe start eating into pagefile, how bad would performance get? On this R3 AWS instance I do have a 1TB SSD EBS volume attached to it, and I can always spin up more EBS volumes for pagefile purpose, but no matter what IOPS I configure I reckon it would never be as fast as RAM.

Have you started looking at TM1 Performance Monitor to see where the memory does to? What cube ? Feeders memory of respective cubes ? I think only then you can isolate the issue. A simple/bad overfeed rule statement can easily eat up huge amount of memory if not careful.
eg: 1GB input data memory with 50GB feeder memory

As soon as TM1 goes into paging, it is as good as dead. Don't ever go into paging.

[TrevorGoss]

which makes complete sense but in reality every restart it only takes up MORE RAM. I'm at 200GB now, I bet restarting now would surely bring it up to 210GB.

How are you restarting the TM1 services? What about the Server? Have you tried restarting that?

I agree with lotsaram, I do not think this is an IBM support issue, as most of the people in this forum are saying more or less the same thing about administration.

[Wim Gielis]

I'm going to try destroying old snapshot scenarios and see if that helps. If so I would just have to archive them.

Are these snapshot scenario's free of rules and feeders ? Than memory usage and performance will not be affected too much, though obviously there will be a positive effect.

Honest question, coming from Cognos Planning, TM1 is a blessing with large cubes. I was told to not worry about the size of my dimensions, cubes when I made the move to TM1. Was I given bad advice?

It all depends. Rules and feeders are superb but experience when creating them is needed once you go beyond the obvious and once your dimensions/cubes increase in size.