TI Memory query

[AWTM1]

HI, I hope someone can help.

I have 3 cubes - Source cube, Lookup cube that contains data that will be inserted to a Target cube.

The source cube is 1.5GB in size (memory used in Architect), the lookup cube is hardly anything.

I have a TI process that pulls the data from the source cube to the Target cube. After the process is run the data in the target cube is 90,000KB, however the memory that is being used by tm1sd.exe in task manager grows by nearly 2 GB.

I believe the source views that are created and then destroyed in the TI process are sitting in memory. However I am not sure on this as the memory grows during the data load of the TI process. If I restart the server I get back the 2GB and the data remains in the target cube.

Can any advise?

[EvgenyT]

Its normal behavior.Check your garbage memory.

The garbage memory is not given back to OS until you 'kick' the server, hence you get your 2 GB back when you reboot

[AWTM1]

Ok thanks, I thought this may have changed in 10.2

http://www-01.ibm.com/support/docview.w ... wg1PM62840

I have just tested in 10.2 and the same happens. I have noticed this in the past but its a large level of garbage memory. How can this be reduced?

[David Usherwood]

Start Performance Monitor and watch the stats in the Cube Properties and Server Properties cubes.
In the past I have managed garbage by deliberately emptying the cache every so often by writing an irrelevant value somewhere in one of the cubes. These days I just make sure there's enough RAM to cope.

[tomok]

writing an irrelevant value somewhere in one of the cubes.

What is the definition of an "irrelevant value" and are sure doing this releases garbage memory back to the OS? I'm pretty sure it does not but I'm always willing to learn something new.

[David Usherwood]

@tomok - it absolutely does not release garbage memory. But if you update a cube where you are calculating values you clear the cache - and if you do it 'often enough' within your process then you can discourage TM1 from creating too large a cache. As I said I have done it in the past but generally I don't bother these days - tell the client to get a large enough box.

[Steve Rowe]

Interesting idea David, currently have a problem instance that is 70GB of data but running at 230GB.....Huge G memory being driven by 20 x parallel processes pre-calculating data for BI.

Something to try next week.
Thanks,
Steve

[AWTM1]

Thanks for the responses, the garbage memory is not caused by cache in my instance. Our models memory requirements are getting close to the capacity of the server, and I am just looking for new ideas regards tuning due to the timescales on server upgrade. I have many GB tied up in Garbage memory, and just wondered if the latest upgrade would help or if any alternative suggestions were out there. FYI - Having tested 10.2.1 against 9.5.2 it does seem to be about 15% more memory efficient but there is still a lot of garbage memory created (similar amount to 9.5.2).

[Steve Rowe]

A couple of things you can try to manage this down.

Only use single threaded launch for the TM1 instance.
If you are doing large copies in TIs do them in smaller chunks, i.e. If you are doing a full year copy do it in 12 monthly smaller chunks. The garbage memory is used by the TI for its working space, the larger the source view the larger the working space required. So if you can cut your source into small chunks and loop through them then you will use the same working smaller space over and over. (I think!)
Cheers

[AWTM1]

Steve,

Thanks - I had not thought of that, I will look into it.

Andrew