Issue with Loading Feeders with MaximumCubeLoadThreads

[mce]

Hi,

I have MaximumCubeLoadThreads is set to a high value to utilize more than one processor while loading cube data to memory.
However this causes feeders also to be loaded in multithreaded manner, and is causing a large amount of TM1 garbage memory.
For example, if my feeders are 12GB for a cube, it is causing 43GB garbage memory while loading. Pleaese note that there is no conditional feeders in this cube, so no order of evaluation issue.

I do not want to disable MaximumCubeLoadThreads to continue benefiting for loading multiple cubes to memory in parallel, but I do not want feeders to be multithreaded for the same cube as it is causing too much of garbage memory.
Is this possible?

Is it possible to avoid the excessive garbage memory caused by multithreaded feeders loading?

Thanks in advance for replies.

Regards,

[mattgoff]

AFAIK, there is no way around it. If you're on 9.5.1 you can set MaximumCubeLoadThreads=0 and PersistentFeeders=T. You'll pay the price on the first load, but subsequent loads should be much faster.

Matt

[lotsaram]

I do not want to disable MaximumCubeLoadThreads to continue benefiting for loading multiple cubes to memory in parallel, but I do not want feeders to be multithreaded for the same cube as it is causing too much of garbage memory.
Is this possible?

Is it possible to avoid the excessive garbage memory caused by multithreaded feeders loading?

Yes kind of. Do a rule save or CubeProcessFeeders then save data on the server and make sure you load the server with PersistentFeeders=T. However if you are loading the server with full feeder evaluation of rule files then there's no way to avoid it other than MaximumCubeLoadThreads=1 or =0.

[paulsimon]

mce

Rather than try going straight down to a value of 0 or 1, you could try experimenting a little. We have 24 processors. I have found that setting it to 5 gives a reasonable compromise between load time and less garbage. The exact value will vary depending on your application and what is considered acceptable.

In case you are not already clear, the garbage isn't actually a problem so long as you have enough RAM to accomodate it. Any TM1 system will naturally grow in size as results are cached, so the memory on the garbage will just get used to accomodate the growth, and you may not see the overally memory used by TM1 increase at all during operation.

Regards


Paul Simon

[mce]

Thanks for all replies.
I will keep PersistentFeeders=T option in mind.
and I will look at experimenting on different values for MaximumCubeLoadThreads in different applications running in different servers when I hit memory problems.