Clearing Out Rules during Run Time

BariAbdul · Post by **BariAbdul** » Sun Mar 08, 2015 3:55 am

Hi David,Could you please expand on it on the below part:
http://www.tm1forum.com/viewtopic.php?f=3&t=6764

Re: Clearing Out Rules during Run Time

Postby David Usherwood » Thu Jan 12, 2012 11:42 am
I use a somewhat different approach to get the same result. I load my data into a measure which does not fire feeders, then copy it across to the fed area in batches. The load runs quickly and I can control the amount I copy, the time it takes and the size the server grows to. (If I loaded all the data across and fully calculated the model the TM1 server would blow my 400gig physical server )

David Usherwood
Site Admin

Posts: 1102
Joined: Wed May 28, 2008 9:09 am

Thanks in advance.

David Usherwood · Post by **David Usherwood** » Mon Mar 09, 2015 11:33 am

I'll try.
The application forecasts the 'development' of insurance policies in force over time ie cancellations, claims, earnings, URR, blah blah. A single group of policies generates values till the policy expires which could be 120 months' time. We had many hundreds of thousands of policy groups.
Originally, when we loaded in the data, feeders would fire right through the model (of course), and thus load time and server size would rise to unworkable levels.
To address this we added an element into the (spare) 'Measure' dimension and called it 'Loaded' and adjusted the feeders to specify the (existing) 'Value' element. We would then load the data into 'Loaded' and the feeders would not fire.
To process the content, we built a set of routines which copied controlled subsets of the data from 'Loaded' to 'Value'. This would fire the feeders and thus activate the calculations but only for that data subset. We then froze the results of the calculations into a static cube. When this finished, we cleared the 'Value' content and restarted the server.
This enabled us to process the content in manageable subsets. Processing everything at once blew the 1/2 terabyte of RAM we had on the server.
HTH

BrianL · Post by **BrianL** » Mon Mar 09, 2015 2:53 pm

What's the benefit of doing this instead of just doing the calculations in TI? Seems that would be both simpler and more efficient.

jim wood · Post by **jim wood** » Mon Mar 09, 2015 3:23 pm

BrianL wrote:What's the benefit of doing this instead of just doing the calculations in TI? Seems that would be both simpler and more efficient.

If you want to follow the Essbase calc once model then may be. What I can't tell form the post is that some of the calculation might be needed for on the fly planning, rather just for a one off.

David Usherwood · Post by **David Usherwood** » Mon Mar 09, 2015 3:42 pm

Apologies, I didn't mention in this thread (but have in earlier threads) that the rules file for this model is forty-three pages long. Not, I would suggest, suitable for TI.

jim wood · Post by **jim wood** » Mon Mar 09, 2015 4:14 pm

43 pages long!!! Blimey. Did you hot any rule length limits? If memory serves it was extended a few years back.

pandinus · Post by **pandinus** » Mon Mar 09, 2015 4:16 pm

It lookt like your model has grown into this current situation.
If you were to rebuild this model however, would you consider using TI to perform and store the calculations instead of your current setup? After all, your rules were also non-existant at one point...

David Usherwood · Post by **David Usherwood** » Mon Mar 09, 2015 5:30 pm

Not the case.
The rules were created for a single project to meet the (admittedly complex) business requirements. Using TI for such complex calculations would be a very (very) bad idea.

jim wood · Post by **jim wood** » Mon Mar 09, 2015 7:28 pm

David,

I've noticed a big move away from rules over the last few years. People seem to think they are bad idea and do everything in TI. I don't know why. After all it's the real time rules calculation engine that is it's chief selling point. I can only it's one of 2 things: 1) People don't how to feed properly or 2) people don't know how to feed properly,

Jim.

BariAbdul · Post by **BariAbdul** » Tue Mar 10, 2015 7:12 am

Thanks a lot David for detail explanation and Jim and all making it quite informative.

lotsaram · Post by **lotsaram** » Tue Mar 10, 2015 9:41 am

jim wood wrote:David,

I've noticed a big move away from rules over the last few years. People seem to think they are bad idea and do everything in TI. I don't know why. After all it's the real time rules calculation engine that is it's chief selling point. I can only it's one of 2 things: 1) People don't how to feed properly or 2) people don't know how to feed properly,

Jim.

I don't think this is a "new" thing at all. There have always been people doing TM1 implementations (TM1 consultants included) who can't quite get their heads around multi-dimensionality, especially multi-dimensional rules and therefore prefer direct TI input over rules. As TM1 has expanded rapidly over the last few years and drawn in people working in TM1 projects from more traditional IT fields who are more used to procedural programming and unfamiliar with multi-dimensional expressions the balance has shifted and the "do everything in TI" approach has certainly become more common.

Still you do also see "over-rulled" applications which suffer poor performance due to long dependant chains of calculations and things being calculated in rules that don't need to be (or in the case of historical values that are never going to change, outright shouldn't be) rule calculated.

I think that using TI in place of rules can sometimes be quite legitimate, especially with 64 bit servers with 100s of GB of RAM. Pure input data gives much better performance vs. rule values. We're not back in the naughties anymore trying to do everything possible to keep the server under a 2GB or 3GB limit.

jim wood · Post by **jim wood** » Tue Mar 10, 2015 12:29 pm

lotsaram wrote:I don't think this is a "new" thing at all. There have always been people doing TM1 implementations (TM1 consultants included) who can't quite get their heads around multi-dimensionality, especially multi-dimensional rules and therefore prefer direct TI input over rules. As TM1 has expanded rapidly over the last few years and drawn in people working in TM1 projects from more traditional IT fields who are more used to procedural programming and unfamiliar with multi-dimensional expressions the balance has shifted and the "do everything in TI" approach has certainly become more common.

Agreed. I've always worked pretty much on my own over years I guess for me it's more about exposure, hence the "last few years bit".

lotsaram wrote: Still you do also see "over-rulled" applications which suffer poor performance due to long dependant chains of calculations and things being calculated in rules that don't need to be (or in the case of historical values that are never going to change, outright shouldn't be) rule calculated.

The balance can differ greatly depending on your point of view. I've always leaned more towards rules due to experience.

lotsaram wrote: I think that using TI in place of rules can sometimes be quite legitimate, especially with 64 bit servers with 100s of GB of RAM. Pure input data gives much better performance vs. rule values. We're not back in the naughties anymore trying to do everything possible to keep the server under a 2GB or 3GB limit.

Again agreed. The key to any TM1 implementation is getting the balance right. As I said above I've always leaned towards rules but they do have their limits. I would say however that just writing rules of is dangerous. TM1 is after all sold as a real time planning application, which of course stems from rules.

David Usherwood · Post by **David Usherwood** » Tue Mar 10, 2015 6:41 pm

Pretty much what we did - rules do the complex calculations, then TI freezes the results (at a somewhat consolidated level) to static cubes where the only rules are (or should be) 'KPI' rules which apply at all levels - which you can't do with TI.
The special sauce here was doing it in stages to manage memory usage.
Personally I don't mind if our competitors overuse TI for calculations as it gives us an opportunity to come in later and sort out the mess.
But if IBM was able to provide decent debugging functionality things might be different. They do tout that with the server-side Java capability in 10.2.2 but really that's not a solution unless you want to migrate all your logic to Java.

tomok · Post by **tomok** » Tue Mar 10, 2015 8:28 pm

i think one of the reasons you see the push towards TI is because that's the official stance of IBM, at least their consulting group. I have been involved in several projects where IBM came in to "review" my work and the first thing they advised was to move all the calculations to TI to increase performance. I think it's on their checklist. They don't want any rules so that the customers won't call to complain about poorly performing cubes, regardless of whether it makes business sense.

David Usherwood · Post by **David Usherwood** » Tue Mar 10, 2015 11:35 pm

i think one of the reasons you see the push towards TI is because that's the official stance of IBM, at least their consulting group. I have been involved in several projects where IBM came in to "review" my work and the first thing they advised was to move all the calculations to TI to increase performance. I think it's on their checklist. They don't want any rules so that the customers won't call to complain about poorly performing cubes, regardless of whether it makes business sense.

Would that be Global Services? I've certainly heard tell of some very poor systems they have built. Sounds like lots more opportunities to sort out cr@p implementations.
I'd love to hear what your response was to their suggestion....
What's your view about the 'hybrid' approach - rules + freeze?

Steve Rowe · Post by **Steve Rowe** » Wed Mar 11, 2015 8:49 am

Doesn't the rules versus TI debate also involve scale as well?

If we you take something simple like a currency translation, pretty hard to get wrong and on a small scale works at an acceptable speed. What about when each period has in excess of a million data points per month?

Done in the traditional N level rule approach this is pretty soggy.

Also pre-MTQ rules are single threaded where as TI is not.

Cheers,

jim wood · Post by **jim wood** » Wed Mar 11, 2015 12:39 pm

I've heard stories about the IBM implementation team and they're not good. They're probably pushing a TI approach as it's easier for them. After all internally they don't encourage specialist roles. I went for interview there. When I got to the technical interview the guy starts asking me about EDWs. I tried to answer some questions then I pointed out that EDWs are not mentioned on my CV anywhere and wondered why I was being asked about them. I was told that all of their consultants have to have a broad product base. I read this as 2 things:

1) They don't know their own product

2) He didn't know TM1 either and asked me EDW questions as that's all he knew

Now if he was right it means that IBM consultants are "Jack of all trades masters of non" which kind of explains the TI approach,

Jim.

David Usherwood · Post by **David Usherwood** » Wed Mar 11, 2015 7:38 pm

If we you take something simple like a currency translation, pretty hard to get wrong and on a small scale works at an acceptable speed. What about when each period has in excess of a million data points per month?
Done in the traditional N level rule approach this is pretty soggy.
Also pre-MTQ rules are single threaded where as TI is not.

If the calculation is that far up the low end of the simple...complex spectrum, yes, TI is probably the right approach for your million data points.
I'm interested, though, in your approach to multithreading TI. I'm thinking you would have to build a fair amount of code to multithread - a single TI will (surely?) singlethread.

Duncan P · Post by **Duncan P** » Thu Mar 12, 2015 8:50 am

A single TI will definitely single-thread, and furthermore if you kick off a number concurrently then any calculations that they do will be held in caches local to the TI threads, and only shared globally when each TI commits.

This was also a characteristic of early MTQ which is why in some cases it did not achieve all of the expected performance gains. I don't know if they have got round this or not.