parallel 2 TI process

[RabbitSlava]

All greetings.
Prompt please.
How to start simultaneously in parallel 2 TI process.
It is possible?
Thanks.

[paulsimon]

It should be, but only if they are updating two different cubes, that are not linked in any way by rules or feeders. You can either start them from different logins, or have one started from a scheduled chore, and then run the other from a login.

Regards


Paul Simon

[lotsaram]

As of 9.4 chores can be scheduled simultaneously and will run concurrently as opposed to queuing. (Provided of course that the chores contain independent processes updating independent cubes).

You can also run multiple processes by calling externally via a 3rd party app such as TM1ProcessExecute.exe

The main thing is to make sure that the processes that you want to run independently do not both try and update the same object, if that happens you will get locking and queuing ...

All loading in parallel, when one considers the speed of TI and how many records per second can be handled this is a beautiful thing ...

top.jpg (81.37 KiB) Viewed 16274 times

[PlanningDev]

Not to drag this back up but just to clarify here

If I have 2 cubes that have 2 shared dimensions, but Im not updating the shared dimensions will I get a locking issue? At the moment it appears that through the TM1ChoreExecute API I can start 2 chores that update unique dimensions in these two cubes but when one of the processes gets to the point of writing data into the one cube, the other process goes to wait and then appears to roll back changes after the first data update finishes.

[lotsaram]

TM1 can process data on independent threads as long as each process is updating different cubes. (You can parallel thread loads to the same cube if you use batch updates but then you can't accumulate.) Meta data updates however impose a much broader lock and should be performed single threaded.

As long as you are not updating dimensions then there is no reason why you can't write to 2 unlinked cubes or a dozen cubes at the same time regardless of whether the cubes have common dimensions or not. You just need to make sure than no common objects are being updated. Where most people come unstuck is failing to remove the commands to toggle data logging for the cubes being written to (as this action writes to a common object namely the }CubeProperties cube). If 2 processes attempt to write to the same object then the one that established the object lock first "wins" and the latecomer must must wait until the lock is released before continuing (at best or rollback at worst.)

[David Usherwood]

I looked into the 9.4+ parallelism recently, not for data loads from external sources (which I'm entirely happy to believe works as posters describe) but for parallel calculation (of rather complex models) and output to TM1 cubes. Based on pretty careful testing, it didn't work - the second process waited for the first to finish. In this case there were two models which shared some data which was readonly during the calculations, but were otherwise totally separate.
My conclusion is that the calculation cache remains single threaded, and that the only workable way to do parallel calculation is to have separate servers.
I have also tested the new slice locking capabilities of 9.5.2 and established that the locking doesn't hold the calculation cache.

[lotsaram]

I looked into the 9.4+ parallelism recently, not for data loads from external sources (which I'm entirely happy to believe works as posters describe) but for parallel calculation (of rather complex models) and output to TM1 cubes. Based on pretty careful testing, it didn't work - the second process waited for the first to finish. In this case there were two models which shared some data which was readonly during the calculations, but were otherwise totally separate.
My conclusion is that the calculation cache remains single threaded, and that the only workable way to do parallel calculation is to have separate servers.
I have also tested the new slice locking capabilities of 9.5.2 and established that the locking doesn't hold the calculation cache.

Hi David - can you explain what you mean by the last sentance? Is calculation still single threaded in 9.5.2 or are you saying it doesn't lock? (i.e. your prayers answered or not?)

[PlanningDev]

Im having a bit of trouble with parallel loading at the moment.

I have loaded up a test server.

I have 2 cubes, and they share 2 dimensions each. I have removed all rule files from both cubes. I then have 2 processes to pull data from the same ODBC source, into the cubes. Cube logging is not checked here. There are two chores each with one of the processes. I run TM1ChoreExecute in the eplilog of a process to execute both chores at the same time and one of them is still going into wait.

I don't know what else to even look for. I feel like I have made this as simple as possible. Two cubes, no rules, no feeders, no logging, only 2 shared dimensions and I still can't get parallel cube loading to work. Do I need to use BatchUpdate or Bulkload here?

[moby91]

I then have 2 processes to pull data from the same ODBC source, into the cubes. Cube logging is not checked here. There are two chores each with one of the processes. I run TM1ChoreExecute in the eplilog of a process to execute both chores at the same time and one of them is still going into wait.

It looks as if you try to execute 2 TI processes in parallel by 1 user thread. This will not work. Every TI process you intend to run in parallel must be executed by it's own user thread.

Try that:
- Start the TM1Top tool to monitor user thread activity.
- Start the first TM1 Architect session, thus starting the first user thread.
- Start the second TM1 Architect session, thus starting the second user thread.
- Simultaneously, in the first TM1 Architect session start the first TI process, and in the second TM1 Architect session start the second TI process.
- The TM1Top tool should display both user threads in a run state, both accumulating CPU time.

[moby91]

My conclusion is that the calculation cache remains single threaded, and that the only workable way to do parallel calculation is to have separate servers.

I assume one or both of your cubes had rules.

Writing rules calculated values into the calculation cache is a global exclusive operation: Only 1 user thread can write into the calculation cache, setting a global write lock which prevents other user threads from writing into or reading from the calculation cache.

By default TM1 is caching rules calculated values to reuse them. That is the best policy for most TM1 applications, but not for all. Thus there is an option to not cache rules calculated values, but instead to calculate them on demand every time they are requested. To disable the default policy of calculate, cache and reuse rules calculated values and instead use the optional policy of calculate, dump and calculate rules calculated values add the optional parameter "InfiniteThresholdForSingleCellStorage=T" to you TM1 server configuration file tm1s.cfg, then restart and test again.

[lotsaram]

Im having a bit of trouble with parallel loading at the moment.

I have loaded up a test server.

I have 2 cubes, and they share 2 dimensions each. I have removed all rule files from both cubes. I then have 2 processes to pull data from the same ODBC source, into the cubes. Cube logging is not checked here. There are two chores each with one of the processes. I run TM1ChoreExecute in the eplilog of a process to execute both chores at the same time and one of them is still going into wait.

I don't know what else to even look for. I feel like I have made this as simple as possible. Two cubes, no rules, no feeders, no logging, only 2 shared dimensions and I still can't get parallel cube loading to work. Do I need to use BatchUpdate or Bulkload here?

In terms of what you are doing right vs. wrong to establish parallel loading, well it's all wrong. Have you read this and other threads on parallel loading in detail?

To parallel load you have 2 options, either open multiple client sessions in different windows and kick off the independent process manually (as per the advice above.) This is fine for proving the concept or maybe for a proof of concept but not a solution for a production environment. The other alternative is to trigger the processes externally such as TM1ProcessExecute which you can download from this forum or RunTI which you may be able to obtain from IBM.

In theory you could schedule 2 or more chores to run at exactly the same time but in my experience external triggering is a much safer bet. You only need BatchUpdate to load in parallel to the SAME cube and you do not want BulkLoad as that forces the entire server into single threaded mode.

Also when you say "cube logging is not checked" I hope that does not mean you are using wizard generated code as this will still write code that writes to the cube properties cube. You need to know what every line of code is doing and ensure you have only what you need when parallel loading.

[blackhawk]

I run TM1ChoreExecute in the eplilog of a process to execute both chores at the same time and one of them is still going into wait.

I am not sure if TM1ChoreExecute will run parallel inside of a TI process. Try scheduling both to run at the same time rather than in the Epilog section of a process.

[PlanningDev]

Im having a bit of trouble with parallel loading at the moment.

I have loaded up a test server.

I have 2 cubes, and they share 2 dimensions each. I have removed all rule files from both cubes. I then have 2 processes to pull data from the same ODBC source, into the cubes. Cube logging is not checked here. There are two chores each with one of the processes. I run TM1ChoreExecute in the eplilog of a process to execute both chores at the same time and one of them is still going into wait.

I don't know what else to even look for. I feel like I have made this as simple as possible. Two cubes, no rules, no feeders, no logging, only 2 shared dimensions and I still can't get parallel cube loading to work. Do I need to use BatchUpdate or Bulkload here?

In terms of what you are doing right vs. wrong to establish parallel loading, well it's all wrong. Have you read this and other threads on parallel loading in detail?

To parallel load you have 2 options, either open multiple client sessions in different windows and kick off the independent process manually (as per the advice above.) This is fine for proving the concept or maybe for a proof of concept but not a solution for a production environment. The other alternative is to trigger the processes externally such as TM1ProcessExecute which you can download from this forum or RunTI which you may be able to obtain from IBM.

In theory you could schedule 2 or more chores to run at exactly the same time but in my experience external triggering is a much safer bet. You only need BatchUpdate to load in parallel to the SAME cube and you do not want BulkLoad as that forces the entire server into single threaded mode.

Also when you say "cube logging is not checked" I hope that does not mean you are using wizard generated code as this will still write code that writes to the cube properties cube. You need to know what every line of code is doing and ensure you have only what you need when parallel loading.

So I have read everything in detail and I am trying to accomplish 2 things. One, to see a parallel load occur under any circumstance, Two, trigger parallel loading from either TM1ChoreExecute or TM1ProcessExecute
1. I tried opening 2 different architect sessions with two different users and executing both chores
2. I tried writing a TI process that uses ExecuteCommand('location\TM1ChoreExcute.exe',0) in the epilog twice (1 for each chore) and running this from one user session
3. I have unchecked cube logging in the TI
4. The cellputN is generated in "Generated Statements" window.
#****Begin: Generated Statements***
OldCubeLogChanges = CUBEGETLOGCHANGES('Cube1');
CUBESETLOGCHANGES('Cube1', 0);
#****End: Generated Statements****

Are you saying that this code is an issue for locking? I haven't seen a thread that talks about this, just that you should make sure logging is not checked. It seems as if you are saying that the CUBESETLOGCHANGES reads the properties cube which causes the lock and ultimately the wait I am seeing.

[moby91]

Are you saying that this code is an issue for locking? I haven't seen a thread that talks about this, just that you should make sure logging is not checked. It seems as if you are saying that the CUBESETLOGCHANGES reads the properties cube which causes the lock and ultimately the wait I am seeing.

Let is exactly what lotsaram is saying: Do not execute the TI fcts CUBEGETLOGCHANGES and CUBESETLOGCHANGES, as these set a read lock respectively a write lock on the system cube }CubeProperties, thus serializing TI processes. If you intend to serialize TI processes, that is great. If you intend to run TI processes in parallel, that is bad.

One option to not execute the TI fcts CUBEGETLOGCHANGES and CUBESETLOGCHANGES:
Encapsulate the block of generated statements with an If expression whose condition is never meet, for instance

If (1>2);
#****Begin: Generated Statements***
OldCubeLogChanges = CUBEGETLOGCHANGES('Cube1');
CUBESETLOGCHANGES('Cube1', 0);
#****End: Generated Statements****
EndIf;

Copy the generated code you need from the non-editable block of generated statements into the manually editable block below.

[PlanningDev]

Thanks for the info!! We finally got it working.

Sorry for my ignorance on exactly what Control Objects are locked based on what TI functions are run but the TM1 documentation on cube locking is not good at best and it is very hard to track down what objects specifically are locked during processes.

[David Usherwood]

I'd like to revive this thread.
I'm still worrying away at how to get more calculations through a very large model.
@moby91, what would you see as the _benefit_ of setting InfiniteThresholdforSingleCellStorage=T ?
@lotsaram, if you are using TM1ProcessExecute to parallel run TIs, isn't that rather messy to set up and maintain?
Does _anybody_ have an idea as to how to use a modern multicore server to get more _calculations_ through TM1?
Does anybody see any of the geewhiz, shiny new features of 9.5.2 which will help here?
Why do the engineers care so much about data input and not about calculation?
Why do I care?

[lotsaram]

Using TM1ProcessExecute.exe for parallel loads is relatively simple and not that messy, it just requires a separate bat file for each parallel thread. If you keep everything within TM1 by calling the batch files with ExecuteCommand with no wait then I would say it is not messy at all. -You just need to ensure that TIProcessExecute logs on with a separate user than the calling shell process to avoid any possibility of deadlock, and unless you want to parametize the client and password in the bat file and pass it in through the TI then it is also a good idea from a compliance perspective to make sure the batch files are somewhere nice and safe.

If an external enterprise scheduling tool like Control-M or Tivoli is used to trigger the batch files then yes it can get messy but this needs to be weighed up against the benefits of external triggering of loads versus relying on the chore scheduler which as we all know is quite limited.

I'm also quite interested in InfiniteThresholdforSingleCellStorage=T. If the calculation cache never gets written to then there will be no locking from calculation actions, one woudl think that this should mean that concurrent calculation threads would then be possible, but I think you would need to do the experiment and see if this works in practice. (Of course NOT caching any calculations is itself a significant drawback.) I haven't had a chance to test 9.5.2 but presumably parallel interaction should allow for concurrent calcs as it allows for concurrent writes. Maybe this is finally the version for you ...

[David Usherwood]

I've only done a bit of testing (out on the road rather a lot at present) but see
http://www.tm1forum.com/viewtopic.php?f=3&t=4446#p19635
I had hopes that Data Reservation would lock the calculation cache. It doesn't seem to.
But maybe I could parallel process my calculations into the (more or less unruled) output cube. Worth a go.