TM1 Forum

Posted: **Mon Nov 11, 2013 10:21 pm**

Hi Guys,

I am dealing with locking issues when I try to run TI processes in parallel in TM1 10.1
Any kind of view or subset manipulation seems to be putting a lock, including SUBSETDELETEALLELEMENTS , SUBSETELEMENTINSERT or VIEWSUBSETASSIGN.

I had implemented a TM1 project in 9.5.2 once the PI feature was available and i was able back then to avoid locking. I know that you cannot use MDX based subsets in your views and also you cannot change the metadata in the process. But I am not doing any of these. All I am trying to do is create a view as data source ( I am not even dropping the view since it is there ). I need to refresh some STATIC subsets and assign them to the view but it seems to be impossible. I have checked the tm1s.cfg and Parallel Interaction is ON.

Have you ever experienced something like this?

Thank you
Ardi

Posted: **Mon Nov 11, 2013 10:43 pm**

Hi Ardi,

Besides metadata changes and typical subset manipulation functions (like you said subsetdestroy/create etc) there are other TM1 task that can overshadow benefits of PI, some of them are:
Establishing cube dependencies, ViewConstruct, Updating aliases.

Please have a look at 'Best practice use of Parallel Interaction', which explains all of the above and more in details:

http://pic.dhe.ibm.com/infocenter/cbi/v ... ction.html

Regards

Evgeny

Posted: **Tue Nov 12, 2013 3:57 pm**

Thank you Evgeny,

I have checked that document about best practices, and in my TI, I am not doing any of the operations that are mentioned there.

In fact I have a TI where I establish Cube Dependencies and I run it before I run my TI which is causing the IX Lock.

This is driving my crazy since I cannot find a way to avoid cube locking

Posted: **Tue Nov 12, 2013 4:14 pm**

Hi Ardi,

Have you considered that maybe your processes are both updating a shared control cube? I once tried to write some processes that would run in parallel and came into locking issue. The issue ended up being that I was disabling cube logging (in different cubes) in both processes. Because both processes were trying to update the CubeProperties control cube, the processes would lock even though I wasn't doing any metadata changes.

brad

Posted: **Tue Nov 12, 2013 4:40 pm**

Hi,

Actually I am trying to RUN the same TI Process twice in parallel ( I have created 2 chores that execute the same TI by applying different Parameters, one of the parameter is the Fiscal Period ) . SO basically I am trying to get use of multithreading while running a process that takes long time in a single thread.

I don't have any CUBESETLOGCHANGES statement ( Actually I had it until yesterday and I removed it ). I have also removed many other statements and also disabled some rules and my IX Lock number went down from 11 to 2, but I still have locking issues so basically it seems I cannot run the same process twice in parallel.

Do you guys use multithreading in TM1 and if so, how do you implement it

Thank you

Posted: **Tue Nov 12, 2013 5:53 pm**

Brad, I can't find any documentation, but if memory serves, the lock on CUBESETLOGCHANGES had been alleviated in 10.1 (with PI enabled.)

Ardi, you might consider enabling lock contention logging (TM1.Lock.Exception,) to track down your remaining locks.

Posted: **Tue Nov 19, 2013 8:22 pm**

I have an update regarding the locking. As I said earlier I was trying to run the same process ( by applying different values for parameters ) in parallel. I was experiencing locking and I found the reason of the lock. The process was updating a view and related subsets to be used as datasource, and the name of some of subsets were the same, so after changing the name of subsets by contatenating it with the Parameter Value, there is no more locking.

Thanks
Ardi

Posted: **Thu Feb 27, 2014 7:30 pm**

Hi,

We are still stragling with locking issues. We have built all our data_load and data_calculate TIs with parameters ( Region and Fiscal Period ) so there is no overlapping in the cells that they update. So we have included many TIs into a batch, we are able to run that batch many times in parallel, by providing different parameter values so each thread loads/calculates/refresheses reporting cube for the Region and Period provided into parameters.

This is working fine and no locking happens so the whole batch can be executed multiple times in parallel and it completes sucessfully.

But if we restart the service, if we try to run the batch in parallel then we experience locking. As i mentioned, the batch has 4 main TIs

1. Load Data into Staging Cube ( CSV is the source)
2. Copy Data from Staging Cube to Calc Cube ( The source is a View from Stage Cube )
3. Re-run Calculations in Calc Cube
4. Refresh Reporting Cube

and the locking happens at step 2. So loading data in parallel works, but copying from one cube to another does not work

As I said, this happens only the first time after restart.

We are using TM1.1.1 and we have Parallel Interaction ON, also we are establishing cube dependencies.

What else can be the reason of this locking?

Thanks
Ardi

Posted: **Thu Feb 27, 2014 7:55 pm**

What type of wait is shown in top? IXC? Did you give TM1.Lock.Exception logging a go?

Posted: **Thu Feb 27, 2014 8:04 pm**

As far as I was aware (please correct me if I'm wrong) but within TI any kind of subset update is regarded as meta change and will lock the cube,

Jim.

Posted: **Thu Feb 27, 2014 9:50 pm**

Updating subsets and views seems to be OK, does not create lock, as long as you dont have 2 or more TIs trying to update the same subset in the same time. So to avoid this scenario, we name the subsets and views in a way that their name is unique ( by concatenating with Region and Period )

The only time we see the lock which is a ICX lock is when we execute them in parallel after a service restart.

Part of the batch is the establishment of Cube Dependencies, so this does not seem to be the reason.

Thanks

Posted: **Fri Feb 28, 2014 3:33 pm**

this happens only the first time after restart

There was/is a "dirty flag" on the cube that signaled updated cubes that needed to be flushed to disk in SaveDataAll/shutdown. That flag was "metadata" and required an IX lock. It would be set the first time a cube was written to since being loaded from disk. Could be this, or something similar.

In your preamble TI which establishes dependencies, you could also update some dummy cell in the Calc cube to ensure that dirty flags are already set. And, just double checking, make sure this TI is run separately (i.e. it commits before any of the parallel TIs kick off.)

If you haven't tried TM1.Lock.Exception logging, I'll suggest it again. You'd add the following line into tm1s-log.properties:

Code: Select all

log4j.logger.TM1.Lock.Exception=DEBUG

And on the lock conflict that precedes the IXC wait, you'd see a message like the following.

Code: Select all

17832   [4]   DEBUG   2014-02-28 15:14:31.730   TM1.Lock.Exception   Lock conflict acquiring 'IX' lock. Thread '17152' holds 'READONLY' lock. Object 'SalesCube' , type 'Cube'.

Knowing which object is locked/requested, and in which modes, can be useful since TM1 is very dynamic in nature - you may be surprised by which objects are involved in an operation. On the other hand, it could point to a very common object and not tell you much at all. YMMV.

Posted: **Fri Feb 28, 2014 5:04 pm**

Took me a while to find it, but, you can also see this info in Ops Console these days http://www-01.ibm.com/support/docview.w ... wg27035835.

Posted: **Tue Mar 04, 2014 8:03 pm**

Thanks,

I have not enabled the TM1.Lock.Exception logging yet since I dont have access to make these changes in the server.

But now i have another issue.
The way I am launching the TI to be executed in Parallel multiple times, is by executing a EXECUTECOMMAND statement,which calls a Batch containg TM1RunTI, which on it own executes the TI with different parameters.

So when the EXECUTE command is executed, i can see in TM1Top many threads showing up, for example I have 24 threads which login one after the other. So it takes around 15 seconds for them to change the status to RUN in TM1Top

If for some reason, one of the threads finishes before quickly, the threads that have not started yet, are put on wait. So the Parallel Execution does not work.

Do you know is there is a way to fix this?

Thanks again

Posted: **Tue Mar 04, 2014 8:44 pm**

The username/login is another common point of contention. Do your TIs happen to use the same credentials to connect (e.g. "Admin"?) If so, you could try giving each their own username (with admin privileges/group membership.)

Posted: **Mon Mar 17, 2014 5:39 pm**

Thank you guys, I was off for a couple of days, and now I'm back to this issue.

We cannot have dedicated users for each thread because we use IntegratedSecurityMode=5 and we cannot create users in TM1.

It looks like this PI, with all these limitations is almost useless.

Posted: **Mon Mar 17, 2014 11:55 pm**

Ardi

What user is the TM1 instance running under within the Services section of Administrator tools. I would expect it is a domain service account created specifically for the task of running TM1. This account will have a static password. Make sure that this account is an admin on your instance. This user can then be used to run the processes in parallel.

Assuming you have an it department that doesn't like handing out the service account passwords, you can use the command tm1crypt to encrypt the password and create a cipher which can then be used to logon to TM1. Get the windows admins to run the following:

Open up a command prompt on the TM1Server
Execute the following command replacing the word password with the password of the TM1 ServiceAccount

Code: Select all

"e:\Program Files\ibm\cognos\tm1_64\bin64\tm1crypt.exe" -pwd password -keyfile "e:\Program Files\ibm\cognos\tm1_64\bin64\tm1keySVC.dat" -outfile "e:\Program Files\ibm\cognos\tm1_64\bin64\tm1cipherSVC.dat" –validate

The from with TM1 you will be able to use these files in combination with the TM1RunTI.exe in the following way:

Code: Select all

sRunTIPath ='The path and filename of the TM1RunTI.exe executable';
sProcess = 'The process you want to run in parallel';
sAdminhost = 'Windows Server Machine Name';
sInstance = 'The name of the TM1 instance that you want to run the process on';
sNameSpace = 'The CAM namespace that you users authenticate against';
sUser = 'The Service account';
sPasswordFile = 'The loaction of the password file created above';
sPasswordKeyFile ='The loaction of the password key file created above';
sParameters = 'Any parameters that the process needs to run';
nWait = 0;

sCommand = sRunTIPath |  ' -Process ' |   sProcess | ' -Adminhost ' | sAdminHost | ' -Server ' | sInstance | ' -CAMNamespace ' | sNameSpace |  ' -User ' | sUser | ' -passwordfile ' | sPasswordFile | ' -passwordkeyfile ' | sPasswordKeyFile | ' ' | sParameters ;

ExecuteCommand(sCommand , nWait );

NOTE: you may need to put the file location and other variables in double quotes to get this to run , use the following:

Code: Select all

sQuote = Char(34);
sPasswordKeyFile =  sQuote | 'E:\Program Files\ibm\Cognos\tm1_64\bin64\tm1keySVC.dat' | sQuote;

Posted: **Tue Mar 18, 2014 4:08 pm**

Thank you failurehappening,

This is exactly what we are doing. We use the TM1 Service Account with ebcrypted password to execute a TI Process ( which is actually a batch TI ) with different parameter values. Let's say we have 3 Fiscal Periods and 4 Regions to load/calculate data for , so we launch 12 times the same batch TI.

So the way we launch them in exactly the way you explained, so we use an account called TM1ServiceAccount_Dev, with TM1PasswordFile and TM1PasswordKeyFile .

So when we run them, in TM1Top we can see that they show up in 12 threads, but if for some reason one of the threads is quick enough so it finishes before the other ones have connected and started executing, then the threads that have not started yet are put on hold and they kick in only one by one.

Thanks again

Posted: **Tue Mar 18, 2014 10:29 pm**

Ardi

Are you performing ANY meta data updates, changing dimension structure, creating subsets, creating views, updating dimension attributes? These kind of operations can't be performed in parallel and will make all processes wait to execute.

I would suggest taking all non cube load activities (including building subests and views) out in to a separate process, execute all that first, and then run the cube load. This will then execute in parallel.

Posted: **Tue Mar 18, 2014 10:57 pm**

Two things we do at the end of the serial sequence of TIs that prepares the environment for PI TI launch is
A. Set the cube dependencies, looking back over this thread you seem to have a handle on this but you should be able to see any new ones that are missed from your TI in the message log. Don't forget the cube dependencies list in the TI may need to be maintained as new relationships are introduced as the system evolves.
B. We "poke" all the dimensions on the cubes concerned by creating and deleting a subset vs them, post all the dimension maintenance work. We found that this cleared an erratic issue where one of the parallel export jobs would take a lock on a dimension and force all the other parallel TIs into serial execution.

One other thing we did when the we refined the implementation was to ensure that all the parallel exports had completed before we launched the parallel import jobs. (We are pre-calculating and exporting data from one cube and writing to another static cube for BI to reference). The nature of how we dice our slice means that some of the exports finish before others and we found that (erratically) if we let the import threads start before all the exports complete we would switch into serial.

Keep trying with the PI, it can be hard going to reach a stable solution but once there it works very well (touch wood!).

Cheers,
Steve

TM1 Forum

Parallel Interaction in 10.1

Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1

Re: Parallel Interaction in 10.1