9.5.2 ParallelInteraction

[David Usherwood]

I'm making some progress with this.
I have a client app where I have to process the calculations in sections as the cube cannot be fully calculated on our hardware (we only have a 400gig server ) I have been able to parallelise the first section (copying the data into a fed area) and plan to extend to cover the main calculation (copying from the summary area to a static cube) next.
It was quite frustrating because, having had it working, it then stopped working. The solution was to read the log messages about adding cube dependencies and put Add CubeDependency(from,to) into a preliminary TI. There were some oddities in that the dependencies the engine found were not the same as the ones I built and know .
I think that, for certain apps, this is a gamechanger for TM1, especially in the context of current server hardware, with lots of cores. But it needs careful thought and practice.

[blackhawk]

David,

I am intrigued by what you have been doing. Can you explain a little more on these two topics?

It was quite frustrating because, having had it working, it then stopped working. The solution was to read the log messages about adding cube dependencies and put Add CubeDependency(from,to) into a preliminary TI.

and

There were some oddities in that the dependencies the engine found were not the same as the ones I built and know

[David Usherwood]

Not quite sure how I can expand on what I have said.
I found that the second chore locked when it hadn't before. When I looked at the message log I found Adding Dependency messages - so I added them in a TI and re-scheduled. Eventually, I found them all and the chores started running in parallel.
I have a feeling that if you ran one very small job right through first then the dependencies would be added and the locking wouldn't be a problem - but I haven't yet tried that.
Update - this worked. Much better than ploughing through 80-odd dependencies and putting them in a TI. Now to upscale to 10 chores!

[blackhawk]

Not quite sure how I can expand on what I have said.
I found that the second chore locked when it hadn't before. When I looked at the message log I found Adding Dependency messages - so I added them in a TI and re-scheduled. Eventually, I found them all and the chores started running in parallel.

I thought the point was to not have dependencies so that you wouldn't have contention, you must have inter cubes-rules in this case, right? So are you saying that we have to manually identify in the TI processes which dependencies that it should be concerned with?

As I was writing this, I just saw your update....

Update - this worked. Much better than ploughing through 80-odd dependencies and putting them in a TI. Now to upscale to 10 chores!

I guess I am going to have to play with this a little more because logically speaking why would it make a difference to have the dependencies pre-set before the processes run? You would think it would just do that at start up internally as if it were a "small job".

Hmmmmm....very interesting....

This means we will have to set up a number of dummy jobs that set the dependencies depending upon the cubes the TI deals with, just to avoid this issue....

Thanks for your input....keep us posted on your progress.

[David Usherwood]

I certainly do have inter-cube rules . The app is very complex and large.
The fix was to run the batch routine first on the smallest possible data set (2 model points - I have 240,000 to process). This 'fired' all the dependencies. Think IBM were expecting customers to work it all out and add the dependencies via TI. You don't need to if you follow my suggestion.

[Mike Cowie]

Hi Blackhawk:

I have to do some testing of the new Parallel Interaction functionality myself, but I think what David is getting at is roughly the following (most of which you probably know, but for others' potential benefit):

• You can have cube dependencies between cubes as a result of rules/feeders.
• Establishing these cube dependencies requires TM1 to do something internally that presumably requires some form of locking of objects on the server. TM1 apparently doesn't automatically establish these references when it starts up or when rules are saved.
• TM1 will establish these cube dependencies for you automatically when you calculate something that involves establishing a dependency which TM1 has not yet established.
• You can also manually establish a cube dependency proactively via the AddCubeDependency TI function.

Let's say there's a cube A => cube B dependency that TM1 has not yet established since the last server restart, and that you're hoping to get benefits from parallel interaction when running processes that use data from cube B, which has calculated cells that depend on data in cube A. TM1 will recognize that the cube dependency needs to be established when you run one of these processes, but the act of TM1 finding and establishing that dependency dynamically will prevent parallel interaction from being possible with these processes on that run. However, if you had previously done work to establish this cube dependency (via a separately run TI process) then parallel interaction would have been possible.

I've since seen David's later suggestion about running on a small data set, which seems like a great workaround. First of all, knowing all the cube dependencies could be tricky in complex models and probably involves things like underlying attribute and other cubes that might not be obvious. Second of all, you'd really want these dependencies to be processed when the TM1 Server restarts, but there's no built-in way to trigger TI processes around that kind of event.

I got the sense Manny was not thrilled about this whole AddCubeDependency thing popping up, so hopefully it will become unecessary in future releases!

Regards,
Mike

[David Usherwood]

We need to examine the calculation cache with 9.5.2. I don't believe TM1 retains it when data is changed - see
http://www.tm1forum.com/viewtopic.php?f=3&t=4446#p19635
but I've seen somewhere tweaks to manage it. May not help though.

[blackhawk]

I just find it odd that you have to tell it what the dependencies are when it seems that it should be intrinsically known. Especially when dealing with the cache; which is supposed to not clear the cache for non dependent cubes.

Which, as a general rule of thumb, we try to move towards no intercube rules (at least whenever possible). Intra-cube rules all day long, but if the more we can keep the cross cube rules out the better our performance, esp with very complex and large data models. Anyway that is another topic.

So now with parallel interaction I have to create dummy TIs to "warm" the dependencies and also now have to split metadata changes from data changes, so the number of processes are now growing more than I would like. Would be nice to have folders in processes so we can group these darn things.

Thanks for your postings and any other insight on parallel interaction is greatly appreciated.

[mattgoff]

I'm just digging into this myself, and after reading this thread and the Understanding Cube Dependency docs I'm not really getting it either. Aren't rules, pretty much by definition, a list of cube dependencies? Now, if automatic dependency generation was a particularly slow process and you could elect to disable it and manually identify dependencies (a la skipcheck/feeders) I could see the value. Or is that what this is, and I'm missing the other half of the mechanism?

Matt

[David Usherwood]

I agree, it's a bit weird. In my testing, putting a small set of data through a monstrous calculation fires _almost all_ of the dependencies, but overlooks a couple - don't know why.
Haven't had time to pursue this much, d&*^ client work keeps getting in the way.

[lotsaram]

The processing a small portion idea is interesting. I just scanned the entire tm1server.log for "TM1.Cube.Dependency" and storred the results in a 2D string cube (dimensions dependency index with members 1 ... 1000 and dependency_m with members source cube and dependent cube). After a server reload we just process a view in the string cube to add back all the known dependencies