TI load data with delayed rule and feed recalculation

[ioscat]

TI load data with delayed rule and feed recalculation - is it possible?
When we load huge amounts of data to huge models with a lot of dependence connections we can get stack (about 100 operations per sec). So we have to comment rule files to delay recalculation. Is it possible not to resave rules but use special TI code?

[qml]

Take a look at the TI function RuleLoadFromFile() which you can use to load an empty file to clear your rules in the Prolog and to load back the original rules file in the Epilog.

[tomok]

Just be aware that the RuleLoadFromFile command is going to re-evaluate all your feeders so this can be very costly, time-wise.

[ioscat]

but if put at prolog dummy file - i get temporally clear cubes (without any rules and feeders). Not?

[qml]

Isn't this what you asked for in your first post? Yes, you will remove all the rule-calculated values in the cube when you load an empty rule file. Actually, you do not even need to load an empty one, loading one with no feeder statements should be enough as typically this is what slows down data loads.

Whether it will actually result in an overall load time improvement, it largely depends on your specific circumstances, but it might be a solution worth looking at.

[ioscat]

my reply was to last speaker - tomok. at that time i've tested your advice on dummy model. thanks for the solution, we will test it on our base model

[Duncan P]

If processing feeders is taking a long time on cube load check that you don't have a situation where a number of leaf source cells are all targeting the same consolidated target cell in order to feed all its leaf descendants.

What happens in this case is that each source value will trigger the feeding of all the target leaf cells. There is no "memory" in the consolidated value that it has already been fed and it just passes the message on to its descendants.

One way to get round this is to create a dummy cube in which that consolidated value is leaf, feed from the source into the dummy and from the dummy into the target. If the consolidated value is the top item of its dimension you can simply do this by omitting the dimension from the intermediate dummy cube.

[lotsaram]

If processing feeders is taking a long time on cube load check that you don't have a situation where a number of leaf source cells are all targeting the same consolidated target cell in order to feed all its leaf descendants.

What happens in this case is that each source value will trigger the feeding of all the target leaf cells. There is no "memory" in the consolidated value that it has already been fed and it just passes the message on to its descendants.

One way to get round this is to create a dummy cube in which that consolidated value is leaf, feed from the source into the dummy and from the dummy into the target. If the consolidated value is the top item of its dimension you can simply do this by omitting the dimension from the intermediate dummy cube.

That is a top notch tip.
(shame performance modler is a few release generations away from taking heed.)

[Harvey]

That is indeed a top-notch tip, Duncan!

When I was working on my complex calc engine model, I did something similar and it made a huge difference to the speed and kept the RAM usage similar, as you'd expect.

I was planning an article on the Flow blog at some point, do you guys think that would be useful?

To the OP, as far as I know, removing the rule statements is the only way to speed up a TI process' data phase. However, keep in mind that you will not be saving time overall, as all the same feeders still have to trigger at the end of the process.

In fact, it's likely it'll take longer overall, as ALL feeders will run when you re-load the feeder statements. I'd only think about it if there was some reason you wanted to release the database connection more quickly.

In the past, I've looked into having conditional feeders that can be turned off by flags in another cube and again, any gains that gives you in data load, it loses in other areas.

You're much better off looking into optimizing your feeders.

[Steve Rowe]

Hi,
Thats an interesting approach to a common problem.
So
a, b, c, d, e are children of A in our detail cube, say all are populated
We have A^ as an N in our feeder staging cube
And then A* as a big conso in our destination cube, say this has five children too

Instead of writing, which results in 25 feeder events, (A has 5 children so does A* so that is 25 feeders)
A=>A*

We write
A=>A^=>A*
Which I think results in 11 feeder events. 5=>1=>5

So even in this trivial example we can see far less feeders get fired.

The big Q for me though is does A^ need to be populated? This isn't mentioned in the approach, though I think it is implied that it does not need to be.

So since A^ is a feeder source it triggers any onwards feeders as soon as itself is fed? I guess this implies that once the feeder for A^ has fired into A* then it doesn't fire again or we would be back to 25 feeders again?
Cheers,