11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post bug reports and the status of reported bugs
Post Reply
Martin Ingram
Posts: 52
Joined: Thu May 15, 2008 9:11 am
OLAP Product: Planning Analytics
Version: IBM SaaS - Digital Pack
Excel Version: Office 365
Location: Reading / London
Contact:

11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by Martin Ingram »

I've just been doing some testing in preparation for a potential upgrade from 9.0 SP3 to 9.4 MR1 (32bit running on WS 2003)

We update TM1 from numerous datasources via TI - approach used is to execute one process including SQL which then writes data out to ascii file. Subsequent processes update dimension structures and load data.

What I'm finding under 9.4 MR1 is that the amount of memory being used by the application ramps up during the data loading stage of my update chores;

Under 9.0 SP3 the server is using a pretty steady 600mb of RAM (this doesn't change when refreshing the data)
In 9.4 MR1 the memory jumped up to 1.6 gb after running TI updates

I did experience something similar at another site running 8.x, but it wasn't to such a great extent and an overnight restart was used to free the memory up again.
jallengt
Posts: 2
Joined: Wed Dec 03, 2008 2:35 pm

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by jallengt »

Martin,

Have you had any response on your inquiry? I saw this post today and your inquiry looks suspiciously similar to mine, only the TM1 version is 9.0SP3 U9.


SR #: 11-112820151
Below is an update on the inquiry. Unfortunately we found no harbor in the U9 upgrade. If the next move is to install the de-bug version please let me know how I can get my hands on it, and if I need any special install instructions. Also, we need to know how to get the full U9 installer. The install available on the Cognos downloads site is an older version, 9.0.3.173.

Saturday, April 04, 2009
Updated server to 9.0 SP3 U9 using loose files
Ran the same tests as previously described with the same results.

Test Outline:
Stop & Start the TM1 service.
Switch on MS Performance Log to record total machine RAM
Switch on the TM1 Performance Monitor
Run one of the existing hourly chores every 30 seconds until RAM approaches 1.9GB
Stop the Chore
Run the same Chore manually until ODBC errors appear
Snapshot the StatsForServer cube


Collected the following files uploaded to the inquiry Hopefully the attachment here will make it to you, the support site is down for maintenance today.

1. StatsforServer.xls
TM1 performance monitor cube. Shows that memory use and memory in garbage remain relatively stable while the RAM reported in Task Manager approaches the 2GB limit on TM1SD.exe. Note - previous test on Thursday did show the increase in Memory in Garbage.

2. TM1SD_000005.csv
MS performance log output. Shows server total RAM increase by approximately 1.5GB, which is the increase seen against the TM1 server. Note the box has 16GB total RAM.

3. tm1smsg_RAM_TI_ODBC_Issue.log
TM1 message log for the test cycle. Near the end of the file, the ODBC error messages appear when the TM1SD RAM approaches 2GB.

4. tm1s_RAM_TI_ODBC_Issue.log
TM1 transaction log as recorded during the test.


Other notes & responses to earlier questions:

The problem appeared around the beginning of February. We saw a similar issue in July 2007. The ODBC connection and basic chores have not changed to any significant degree.

The ODBC connection is not being compromised. At the point the server reports ODBC connection problems we can still use the ODBC driver via Excel Microsoft Query.

We have not set the 3GB switch. The server normally runs in the 750MB to 1.0GB range and should not be required.

As a short term workaround, the service is restarted every night.
Martin Ingram
Posts: 52
Joined: Thu May 15, 2008 9:11 am
OLAP Product: Planning Analytics
Version: IBM SaaS - Digital Pack
Excel Version: Office 365
Location: Reading / London
Contact:

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by Martin Ingram »

I got a reply which was basically;

"Well, you'll need to upgrade to 64bit, won't you?"

Nice :roll:
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by lotsaram »

Hi Martin

I am pretty sure you will find that your issue is associated with cube logging not being turned off somewhere in the batch upload process. In 9.4 the log file is not continually written to disk with each transaction but held in memory and committed at the end of the process with the data changes. This is different from all previous versions. In 32 bit 9.4 leaving logging on during a cube load can quite easily blow the memory and cause a crash.

This memory spike issue still occurs in 64 bit so that's not necessarily the answer (you might just have a bigger memory buffer).

IMO this is quite a serious design flaw but the workaround is easy - turn logging off.

HTH
User avatar
mattgoff
MVP
Posts: 516
Joined: Fri May 16, 2008 1:37 pm
OLAP Product: TM1
Version: 10.2.2.6
Excel Version: O365
Location: Florida, USA

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by mattgoff »

lotsaram wrote:IMO this is quite a serious design flaw but the workaround is easy - turn logging off.HTH
Unless you replicate :twisted: (god forbid)
Please read and follow the Request for Assistance Guidelines. It helps us answer your question and saves everyone a lot of time.
User avatar
Steve Rowe
Site Admin
Posts: 2410
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by Steve Rowe »

@lotsaram,

I hadn't heard about this, though I've not been studying the release notes.

You've really got to question the priorities and well intelligence of who decides what get's built in the TM1 dev team.

Head Designer : Well we seem to have a problem in 9.4, the memory requirements are ramping up over the prior releases and "people" are not happy. Anyone got any ideas what we can do about it?

Developer : Well we could make the transaction log cache in the memory too, that would work wouldn't it.

HD : Brilliant, let's change a piece of functionality that works perfectly well to make the problem worse. Excellent here's a gold star Mr/s Developer!

I think that the increased availability of 64 bit is really being used as cover for some really strange changes in the product...

Cheers,
Technical Director
www.infocat.co.uk
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by Alan Kirk »

Steve Rowe wrote:@lotsaram,

I hadn't heard about this, though I've not been studying the release notes.

You've really got to question the priorities and well intelligence of who decides what get's built in the TM1 dev team.

Head Designer : Well we seem to have a problem in 9.4, the memory requirements are ramping up over the prior releases and "people" are not happy. Anyone got any ideas what we can do about it?

Developer : Well we could make the transaction log cache in the memory too, that would work wouldn't it.

HD : Brilliant, let's change a piece of functionality that works perfectly well to make the problem worse. Excellent here's a gold star Mr/s Developer!

I think that the increased availability of 64 bit is really being used as cover for some really strange changes in the product...

Cheers,
Is this really 9.4-specific, though, or 9.1 onwards? There was a question that was left dangling in the old forum with regard to chore rollbacks, which were introduced in 9.1; specifically, how this was implemented. If logging is turned off, the only way TI could do a rollback would be to cache the data that has changed in memory since there would be no log file to read it from. However even if logging was turned on, it would probably be faster (to finalise the data entry into the cube) to write the data into the cube in memory first, then flush the cached changes to the file.

The unfortunate side-effect, of course, is that as I speculated in that earlier thread 32 bit users get screwed and rotated a few times.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
User avatar
Steve Rowe
Site Admin
Posts: 2410
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by Steve Rowe »

That's still a strange way to implement rollback though, surely?

1. Rollback requires logging in some form.
2. If logging is turned on we can use the file to rollback.
3. If logging is turned off then we can either, a)Turn logging on in the background, b)Develop some entirely new functionality to hold the changes in RAM.

Doesn't seem like the sensible choice to me, though I can understand the justification would be the speend of the data load. IMO though the speed of data loads is not really a problem with logging turned on.

Cheers
Technical Director
www.infocat.co.uk
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by lotsaram »

Alan Kirk wrote:Is this really 9.4-specific, though, or 9.1 onwards?
As far as I'm aware this "memory leak" due to logging doesn't affect 9.1 the way it does 9.4. Rollback seems to be far improved in 9.4 vs 9.1 and I suspect this bug is an unintended result (although why this wasn't foreseen begs a few questions). There are 2 easy technical workarounds; install more memory (obviously a 64 bit option only) or turn logging off during TI processes. But these workarounds are not always viable/acceptable. As Matt already pointed out there can be valid reasons why logging can't be switched off such as replication, and throwing hardware (and money) at a problem is clearly not always the best solution. Simple point - it's acceptable (and expected) that leaving logging on during a load might slow performance, but what this should NOT do is have any impact on server stability which is not currently the case.
Steve Rowe wrote:I hadn't heard about this, though I've not been studying the release notes.
I wouldn't necessarily expect this to be in the release notes (hopefully it will be in 9.4.1 FP3 ...) it seems this might have slipped through the cracks during testing (preferable to believe "slipped through the cracks" than "swept under the carpet" hoping no one would notice.)

Note to admins: if it turns out that Martin's original issue isn't actually this one then perhaps the last bit of this thread should be split off into a separate post?
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: 11-108922781 - TM1 9.4 MR1 TI Memory Leakage

Post by lotsaram »

Steve Rowe wrote:That's still a strange way to implement rollback though, surely?

1. Rollback requires logging in some form.
2. If logging is turned on we can use the file to rollback.
3. If logging is turned off then we can either, a)Turn logging on in the background, b)Develop some entirely new functionality to hold the changes in RAM.

Doesn't seem like the sensible choice to me, though I can understand the justification would be the speend of the data load. IMO though the speed of data loads is not really a problem with logging turned on.

Cheers
I think the issue with rollback and logging was more to do with how to resolve the problem of if the server determines there is a need to rollback and the log has already been written how to determine what portion of the log to erase? ... Easiest engineering solution, don't write the log file in the first place until it has been determined that the data changes will be committed. From what I can tell rollback in 9.4 doesn't depend on logging at all but rather the new batch updating mechanism.

Caching additions to the log file in memory is not a bad idea per se, but not having an upper limit on the size of the log cache is ...
Post Reply