Source Cube Feeder Memory Issue

[DeeDee]

TM1 9.5. I have a Target SalesPlan cube that has 7 regular dimensions and 1 measure dimension. The cube size is 126k, the dimensions are
Company (12k – 3 elements), Scenario (14K – 8 elements), Year (14K – 7 elements), Period (22k – 13 elements), Salesman (38k – 50 elements),
Customer (1478k – 2000 elements), Product (262k – 2000 elements), Measure (12k) : where ‘Quantity’ is an element

Server has 20G, which is dedicated to TM1, with 19G available before I save the Source cube rule. If my Source Cube Feeder is targeted to only one Product and the rest of dimensions ‘All’ , the Rule is saved very quickly. If I change the Feeder to feed to ‘ALL PRODUCTS’, the memory will go down from 19G to less than 0.5G and the system will hang, and I will need to restart services.
Here is the detail, I need to feed a constant to the SalesPlan cube, I created a dummy feeder cube which has 2 dimensions that contains a fixed constant.

On the Target Cube, I have this rule, this works fine.
skipcheck;
[{‘Scenario 1’,'Scenario 2’,'Scenario 3','Scenario 4'},'Period99','Quantity']=N:
if (['Total 12 Periods','Quantity'] > 0, ['Total 12 Periods','Quantity'], DB('zFeeder','FeederElement','Value'));
Feeders;
['Total 12 Periods','Quantity']=>['P99','Quantity'];

On the Source Cube, I have this feeder feeding the Target cube, cause 'out of memory' from 19G to 0.5G...
skipcheck;

feeders;
['FeederElement','Value']=>
DB('SalesPlan Cube’,'Total Companies',Total Scenario','Prior 3 Years','P99','Total All Salesmen','Total Customers','All Products','Quantity');
*** The above cause Out of Memory ***

If I replace 'All Products' with one specific product Id, the Rule is saved very fast.

****

I tried to reorder the dimensions, maybe the cube/dimensions are too small to result in any benefits. Please suggest where else I should look into? Thanks!

[tryinghard]

I would also like to know about this.

[Gregor Koch]

Hi

Without going into the detail of your rule and model...
Just to explain what is happening in short...

Feeders you set obviously need memory.
I did a quick check on a server and the memory usage per feeder there ranges quite a bit (actually I get values between 0.3 and 40 bytes) but let's just say it uses one Byte.

Under the assumption that the dimensions you listed have one consolidation (the one you use in your feeder) and the rest are children of that consolidation, the feeder you wrote

DB('SalesPlan Cube’,'Total Companies',Total Scenario','Prior 3 Years','P99','Total All Salesmen','Total Customers','All Products','Quantity');

would feed

(the combination of 2 Companies,7 Scenarios, 3 years, 12 months ?....)

2*7*3*12*49*1999*1999*1 = 98,685,240,696 cells

And under the assumption of 1 Byte per Feeder would need 91.9 Gigabyte. If you only feed one product this goes down to

2*7*3*12*49*1999*1 = 49,367,304 cells

Quite a difference?

The way it (your feeder statement) is at the moment I don't think the reordering of dimensions will help you.
For more help on the feeder for your rule it would be best if you could explain some of the business logic around that rule.

Cheers

[mce]

As the feeder cube is very small, I do not think that you should use SKIPCHECK and FEEDERS in that cube. You may try removing feeders and SKIPCHECK in the source cube.

[kpk]

If I change the Feeder to feed to ‘ALL PRODUCTS’, the memory will go down from 19G to less than 0.5G and the system will hang, and I will need to restart services.
...
If I replace 'All Products' with one specific product Id, the Rule is saved very fast.

TM1 will be much slower when your system runs out from physical memory and start to use the disk.

[kpk]

[{‘Scenario 1’,'Scenario 2’,'Scenario 3','Scenario 4'},'Period99','Quantity']=N:
if (['Total 12 Periods','Quantity'] > 0, ['Total 12 Periods','Quantity'], DB('zFeeder','FeederElement','Value'));

Could you explain what is the business content of the 'zFeeder' cube which can store a generic valid value for all product, all salesman, etc. without the product and salesman etc. dimensions?

[DeeDee]

Hi,

I got the idea from http://forums.olapforums.com/viewtopic.php?f=3&t=2850

I need to use Active Form with dynamic rows on products and zero suppression as an Input Template. There are total of 10 scenarios, only 4 of the scenarios should show up in the Input Template. So the idea is to populate those 4 scenarios with a Period 99 with a constant number, so that they will remain on the Active Form without being zero suppressed.

All Title Dimension at base levels

SCENARIO PERIOD
01 02 03 ...

ACTUAL 99 99 99
BUDGET 99 99 99
SALESPLAN 1 xx xx xx
SALESPLAN 2 xx xx xx

My first question is then, is there other ways to do the above?

My second question is, there are situations that I will need to feed between cubes with massive cells affected, what is the limit of Feeder in terms of cells affected and the RAM you have?

Question 3, is there anywhere within TM1 system setup that I can tell TM1 to use the RAM certain ways so that the FEEDER will work?

I will try to remove SKIPCHECK from the source cube today and see if it makes any difference. But I will need the FEEDERS so that the TARGET cube will be FED properly so that they rollup properly, true? I will also try to FEED ALL PRODUCTS and SCENARIO by SCENARIO, so instead of one FEEDER statement, have FOUR FEEDer statement, wonder if that may work...

Quite puzzled by TM!...

Thank you all in advance!!!

[David Usherwood]

You didn't respond to kpk's very valid question.
Writing good feeders involves both TM1 technical knowledge and business knowledge. (Think that's why I like them ) Describe what content you are trying to feed and why and you will likely be able to get better guidance on the problems you are having.

[DeeDee]

I was trying to feed a constant value to a measure element such that the row will not be zero suppressed during Active Form, as explained above. Is this a wrong way to go about forcing a row to show up in a zero suppressed Active Form? Tks.

[Gregor Koch]

DeeDee,


1. Yes, and you should consider them. More later.

2. Is basically answered with my earlier post in combination with kpk's statement.
Once TM1 has to use virtual memory (hard disk) it becomes very slow to a degree that you cannot work with it anymore or will be unstable (crash). As an absolute theoretical minimum you need 1 Bit per Feeder (Yes/No) but I think more than that is realistic.
To be able to do the math on the memory usage you need to understand of how many cells you feed. More importantly how many you actually need to feed and why.

3. Your Feeder won’t work with what you have! Even if you had the necessary memory it would be light years from optimal (to put it mildly).
There are settings which put limitations to the memory usage for certain things in TM1 but a Feeder will need memory no matter what.
So you cannot write a Feeder statement which feeds a gazillion cells and then tell TM1: 'but only use 1GB of ram to do this'.

Removing Skipcheck; and Feeders; from you source cube won't change the fact of how many cells your Feeder statement feeds in your target cube. Let alone that it won't compile in the first place as ['A']=>['B']; without Feeders; above it will be an invalid statement. But I guess you found that out by now.


Without further information I could think of 4 general ways

1. Populate valid combinations of e.g. Salesman, Company, Product, Scenario etc with Turbo Integrator

2. Populate valid combinations by using Spreading (maybe based on Actuals, see below) with meaningful quantities.

3. Creating a ‘Yes I need to forecast’ Flag (data, not calculated) in the target cube which is entered or spread against Customer, Salesman, Product which could be used as first column in your Active Form. This is basically what you are trying to do but you would not need to write a feeder again the key here is to focus on valid combinations.

4. Create the report with a more specific selection of products which narrows down the rows to a workable extend (e.g. by product categories for one salesman and customer) and not zero suppress at all.


I can only repeat my (and kpk’s) question: What is the business logic (and the forecasting requirement)?

And maybe to give you some pointers as to what you need to think about…

Do you want to create a report which has 2000 Products times 4 Scenarios down the rows? Does the user need to enter the quantities?

Who is the audience of the report? Salesman, Analyst?

Do all salesman sell to all customers?

Do you want to forecast sales of all products to all customers?

Do both companies sell all products?

Do you actually set targets by product by customer by salesman? Or is there maybe a difference in how you forecast your Budget and Salesplan?

What happens with new products, customers, salesman?

Do you have the real ‘Actual’ quantity (defined by the dimensionality you use or less) for any period that could be used as a basis for your scenarios?



Cheers

[mce]

As the feeder cube is very small, I do not think that you should use SKIPCHECK and FEEDERS in that cube. You may try removing feeders and SKIPCHECK in the source cube.

Removing Skipcheck; and Feeders; from you source cube won't change the fact of how many cells your Feeder statement feeds in your target cube. Let alone that it won't compile in the first place as ['A']=>['B']; without Feeders; above it will be an invalid statement. But I guess you found that out by now.

In my comment, I ment removing feeder statements as well from the source cube in addition to removing SKIPCHECK and FEEDERS.

As pr my understanding of Feeders in TM1 Rules, for small cubes that does not have many cells, we do not need to use feeders even if these cubes are feeding some bigger cubes. Could you please correct me if this statement is wrong?

In our TM1 models, for example; we have a small assumption cube that holds currency exchange rates for various currencies on a monthly basis for each version. We have some big cubes where we do currency translation from local currency to many other currencies for the purpose of reporting. In this model we do not have any feeder statements in the source cube that hold exchange rates, even we do not have any rules defined, but the currency translation calculations in the big cubes are working fine although those big cubes have complex rule statements including feeder statements.

Regards,

[tryinghard]

I have a quick question, if my feeder has All products and the Products dimension has 5 levels , so in other words, we have 115 elements in the dimension and 75 are leaf and the rest 40 are consolidated elements at different levels, then does the feeder feeds all the elements i.e. 115 or just the leafs that is 75.

[Andy Key]

TryingHard:
Feeders only fire at the leaf level, as what you are doing with a feeder statement is marking a leaf level cell as needing consolidating. Once that cell is marked as fed, the in-built consolidation engine takes over and sorts out calculating the rest of the levels in your hierarchy. So in your example, feeding the All Products total element will result in the 75 leaf elements being fed.

MCE:
You are correct in saying that data from your small cube without SkipCheck and Feeders can be used in rules in your large cube without a specific Feeder statement in the small cube. The only proviso is that the cells in your large cube that does have SkipCheck and Feeders do still need to be fed from somewhere. In your case of exchange rate calculations, I would expect you to have a feeder from Local Currency to your other translated currencies, all within your large cube, as this will minimise the number of fed cells.