Locking issue on cubes while running TI processes parallely

[ttesla]

Hi All,
Our TM1 Model (size 31 GB) experiencing performance issue while running TI processes parallely. They would be around 30 TI processes running for around 12 hours to process data like calculations, copying. Our intention is to reduce TI processing to reasonable time frame (around 5-6 hours).
TI code in model will create cubes on fly (creates, uses, deletes) to improve TI processing speed but still performance is not up-to expected levels. To improve processing time started testing TI processes by running parallely from TM1RunTI command-line tool so that TI processes can utilizes multiple cores on server (CPU has 16 cores) but processes( each process runs different sections of cells in same cube) are locking cube and waiting for others to complete.

Any pointers or approach to improve TI processing speed by utilizing multiple cores is appreciated. Looking for help on best practices to be followed while running TI processes parallel & using multi cores by firing TI’s from TM1Runti commandline tool.
Note:
1. TM1 server instance on TM1 cloud 10.2.2 with data tier having RAM Size 256 GB
2. Each parallel process runs for different sections of cells for same cube
3. Creates, uses, deletes temporary cubes while running TI process to improve speed
4. Business runs these processes every week
Thanks

[declanr]

I do things like this regularly and there is often a bit of trial and error to it; stick logging on and check where the locks are then remove one by one.

As a general rule of thumb though I always create all views, subsets etcs in 1 master process before kicking off the data transfer sub processes that use them.

Your model isn't really that big so I would be interested as to what you are actually doing for it to take that long; even your target time seems very high to me. Of course there are some valid circumstances that would lead to that but I would imagine you could probably cut it down more.

[ttesla]

Thanks for your response declanr.

Even tough logging mechanism placed in TI Code(AsciiOutPut) but not able to figure out at what line of code locking occurs, even Operations Console is not helping here it just shows "wait" mode but not convey functions etc.

Generally (our way of implementation), each subprocess(kicked off from Master process) responsibility to create,use, delete all views, subsets, temporary cubes, its applicable even for processes running for same cube with different sections of cells.

As business gives us data at consolidation level, code in TI will drill down to each leaf level and do calculations like averages, mean etc on data.

I would really appriecate if you can share your experience in identifying locks in TI code viz. probably some functions in TM1 TI APIs where chances of locking are more, order of processing data, etc.

Thanks

[PavoGa]

This might help with some ideas, it is a recent discussion on this very topic:

http://www.tm1forum.com/viewtopic.php?f=3&t=12552

Additionally, are you writing to the consolidations with CELLPUTPROPORTIONALSPREAD? Did not sound like it, but thought it worth asking. Have you turned on log4j.logger.TM1.Lock.Exception=DEBUG?

Declan is absolutely correct, move the metadata updates to the beginning and best if through TM1RunTI.

Additionally, are you doing cube-to-cube and/or intra-cube reading and writing? If so, it can certainly crush performance. I recently had a process that, by writing to text files and reading the text files back in dropped the time from over 25 minutes to less than one.

[ttesla]

Thanks for your response PavoGa.

Yes "log4j.logger.TM1.Lock.Exception=DEBUG" added to see additional logs. Output of it thrown lots of data but does not help us to nail down locking issue.
Definitely not writing to the consolidations with CELLPUTPROPORTIONALSPREAD.
Most of TI's which do calculations are designed to use data source as Cube views (in TI wizard select Data source as cube view). If using temporary cubes in process ( to store calculated values like gross revenue etc and use it in further processing) is called intra processes then yes we do. we feel that instead of storing in temporary cube(lookup cubes created then our system will have lot of cubes, ultimately increases maintenance) if we write to file and load back to cube for lookup its going to take much time because data tab has to process again same number of records.

One more thing observed is that locking starts to happen after 35 to 45 secs after processes starts, so we concluded that its data tab that is causing issue. more over our temporary cubes differ by parameter passed during kickoff time.

Thanks.

[BariAbdul]

Most of TI's which do calculations are designed to use data source as Cube views (in TI wizard select Data source as cube view). If using temporary cubes in process ( to store calculated values like gross revenue etc and use it in further processing) is called intra processes then yes we do. we feel that instead of storing in temporary cube(lookup cubes created then our system will have lot of cubes, ultimately increases maintenance) if we write to file and load back to cube for lookup its going to take much time because data tab has to process again same number of records

I think,This could be your potential cause of your model slowness as cube to cube read data is bit slow. Couldn't you export them to text files and have them as source file? Thanks

[PavoGa]

One more thing observed is that locking starts to happen after 35 to 45 secs after processes starts, so we concluded that its data tab that is causing issue. more over our temporary cubes differ by parameter passed during kickoff time.

How do you know for certain it is locking and not just a slowing down of reading the source cube? To also clarify my earlier question, inter-cube is reading from one cube and writing to another. Intra-cube would be reading from and writing to the same cube in a single TI. What Bari and I are trying to tell you is that can run into some serious performance issues reading from and writing to the same cube in a single TI. It has to do with the write operation resetting the calculation cache on the cube and the server is recalculating rules it previously resolved. Writing to a file (or a temporary cube in some cases) alleviates that issue.

[ttesla]

Thanks for response PavoGa.

In TM1 Operations console/TM1 TOP we can see mode as WAIT:IXC.
As informed in thread, separated Reference structure (Cube, views, subsets) creation to different processes, now each processes are not locking each other (processing same cube with different cells) able to achieve parallelism.

Thanks for declanr, PavoGa, BariAbdul for the inputs shared.