MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post Reply
EvgenyT
Community Contributor
Posts: 324
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: 2016
Location: Sydney, Australia

MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by EvgenyT »

Hi,

Although, MTCubeload and MTFeeders.AtStartup have been around for some time now, I'm still trying to cement my understand on how exactly do they operate vs MaximumCubeLoadThreads. In particular its impact on conditional feeders.
It is well-known that MaximumCubeLoadThreads could not be turned on when conditional feeders were present, as TM1 engine could not control the sequence of how these feeders would be processed. Fair enough. Now, we have these two new parameters that leverage MTQ framework. IBM states that
MTCubeLoad also eliminates the risk of changing feeder generation sequences

Does this mean that data (raw or calculated) querying is now multithreaded, while feeder construction remains single threaded? Although, documentation For MTCubeLoad does not state anything in relation to conditional feeders, I'm making this assumption based on the fact that MTFeeders.AtStartup parameter needs to be set to F if there are conditional feeders in a model.
In the nutshell, I'm trying to understand what impact MTCubeLoad has on feeders, whether conditional or not. If we don't have conditional feeders, would MTCubeload leverage MTQ framework during feeder processing? Or do we still need to include MTFeeders.AtStartup? If we do have conditional feeders, would MTCubeLoad be smart enough to recognize that and turn disable MTQ for feeder processing?

Conditional Feeder = MTCubeLoad On + MTFeeders.AtStart Off ?
Regular Feeders = MTCubeLoad On ? Or MTCubeLoad On + MTFeeders.AtStartUp On?

Hope it makes sense!

Look forward to your ideas

Regards,

Evgeny
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by lotsaram »

To try and keep the explanation as brief as possible MaximumCubeLoadThreads=N would release N threads at once, each one loading a separate cube. As cubes are not uniform in size and each thread finished at a different time then goes onto the next cube this means the order of processing cannot be controlled. Not only do you have the situation where conditional feeders may be incorrectly processed because a dependent cube is processed first before its dependencies but you also have the situation where the same cube may be processed multiple times by different threads.

With MTCubeload the order of loading is serial cube-by cube but multiple threads are leveraged to load the cube data and process feeders. So the cores are utilized but the execution order is controlled as per single threaded loading. It's a much better model and you see much bigger performance improvements for systems that have really large cubes.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
EvgenyT
Community Contributor
Posts: 324
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: 2016
Location: Sydney, Australia

Re: MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by EvgenyT »

Thanks Lotsaram

This all true and well, however it still begs a question, If
Wth MTCubeload the order of loading is serial cube-by cube but multiple threads are leveraged to load the cube data and process feeders.
then the order of execution of cubes with conditional feeders must be structured accordingly, correct? Simple example: Cube B has conditional feeders going to Cube A. MTCubeLoad must structure the queue in a way that Cube A loads first, then Cube B. Correct?
Also, how does MTFeeders.AtStartup tie into this entire scenario? If MTCubeload meant to handle limitation of MaximumCubeLoadThreads then what purpose does MTFeeders.AtStartup serve? Mind you, it states that MTFeeders.AtStartup should be avoided with conditional feeders.
I have also been sceptical that MaximumCubeLoadThreads has only impact on cubes with conditional feeders, I would have thought any out of order cube processing with cross-cube feeders (DB and ATTRS/n references) could potentially impact data integrity if the Dependent Cube runs to a completion before the Base Cube.

Thank you

Evgeny
User avatar
ykud
MVP
Posts: 148
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by ykud »

EvgenyT wrote: Thu Feb 21, 2019 12:01 pm
I have also been sceptical that MaximumCubeLoadThreads has only impact on cubes with conditional feeders, I would have thought any out of order cube processing with cross-cube feeders (DB and ATTRS/n references) could potentially impact data integrity if the Dependent Cube runs to a completion before the Base Cube.
All these parameters only affect feeders processing, data is not updated during load time / rule calcs are not persisted, data is just loaded from disc. So even if you have a rule that uses ATTRS and the attribute cube is not loaded, this rule will not cause any adverse effect unless you're feeding using that attribute and potentially missing a feeder flag.
EvgenyT
Community Contributor
Posts: 324
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: 2016
Location: Sydney, Australia

Re: MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by EvgenyT »

Thanks Yuri,
So even if you have a rule that uses ATTRS and the attribute cube is not loaded, this rule will not cause any adverse effect unless you're feeding using that attribute and potentially missing a feeder flag.
- Well, that where I was kind of getting to...

On the hand, putting MaximumCubeLoadThreads aside as it seem it might be depreciated in the future, the overlap between MTCubeLoad and MTFeeders.atStartup is still somewhat unclear to me.

Two contradicting statements (at least to me):
  • MTCubeLoad also eliminates the risk of changing feeder generation sequences.
  • When MTFeeders.AtStartup is enabled, Cognos® TM1® cannot manage the order in which feeders are calculated. There might be cases where processing order has an adverse effect on your application due to some order-of-evaluation dependencies in the multi-threaded environment.
So documentation suggests that MTLoad is fine for conditional feeders processing, on the other hand MTFeeders.AtStartup may cause the same issues as MaximumCubeLoadTreads used?


Many Thanks

Evgeny
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by lotsaram »

There are 4 parameters here (actually some extra related ones for fine tuning as well but the documentation doesn't really explain what they do e.g. IndexStoreDirectory, PreallocatedMemory.Size, PreallocatedMemory.BeforeLoad)
MTQ
MTCubeLoad
MTFeeders
MTFeeders.AtStartup
  • Without MTQ enabled as a prerequisite none of the other MTQ related functions are going to do anything.
  • MTCubeLoad affects the loading of cube data to enable within cube multi-threaded loading. It doesn't affect feeder processing only loading of data. This parameter affects the loading of .cub and .feeders files
  • MTFeeders allows the mult-threaded processing of feeders while the server is in session (e.g. when saving a rule or doing CubeProcessFeeders via TI)
  • MTFeeders.AtStartup is a separate flag to control whether rule evaluation and feeder processing should be multi-threaded during the server load. The feeder processing happens after data is loaded. Unless this parameter is true then rule and feeder evaluation is single-threaded.
If using PersistentFeeders with normal startup then MTFeeders.AtStartup isn't used as there will be no feeder processing. The parameter only has an effect if not using PersistentFeeders (or if using them and feeder files have been cleared or there is a validation error in the timestamps).

Strictly speaking I guess it would be more consistent if MTCubeLoad had been named MTCubeLoad.AtStartup
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
Paul Segal
Community Contributor
Posts: 306
Joined: Mon May 12, 2008 8:11 am
OLAP Product: TM1
Version: TM1 11 and up
Excel Version: Too many to count

Re: MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by Paul Segal »

So for maximum performance (in theory) these MTx parameters should be set or T as standard? Even in the case of using Persistent Feeders it's going to show a gain when for some reason they fail/get blown away.
Paul
User avatar
paulsimon
MVP
Posts: 808
Joined: Sat Sep 03, 2011 11:10 pm
OLAP Product: TM1
Version: PA 2.0.5
Excel Version: 2016
Contact:

Re: MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by paulsimon »

Hi

From my reading of this, for most people, MTFeeders.AtStartup should be set to False, since most of us use conditional feeders

Regards

Paul Simon
EvgenyT
Community Contributor
Posts: 324
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: 2016
Location: Sydney, Australia

Re: MTCubeLoad vs MTFeeders.AtStartup vs MaximumCubeLoadThreads

Post by EvgenyT »

paulsimon wrote: Mon Feb 25, 2019 10:51 am Hi

From my reading of this, for most people, MTFeeders.AtStartup should be set to False, since most of us use conditional feeders

Regards

Paul Simon
Yes, it's very true. I guess you can try your earnest to avoid conditional feeders but would seem rather hard in any drive-based planning model.
Probably not a big of a deal if you have it off and use Persistent Feeders in your Production environment (MTCubeLoad will take care of multithreading here), while you can leverage MTFeeders = T in your development on a case by case scenarios (this parameter is dynamic).

Regards,

Evgeny
Post Reply