Executing processes from a Master Process

[PavoGa]

I have for several years used master processes in conjunction with a control cube to manage the execution of processes. The basic structure of the system would be:

* a master process reads the control cube for processes to execute along with parameters
* builds a datasource, typically a dimension subset
* the processes are executed via ExecuteProcess function in DATA tab.

This year I've started using TM1RunTI in order to minimize locking issues but have now run into a new problem. And maybe I just need some guidance on what one can/cannot or should do.

Because of the aggravation of migration policy to production environments, we have TI processes built to perform maintenance issues so it does not take four days to migration a dimension/attribute/etc change to production (as long as we use TI processes, we can make all the changes we want). The current problem is a TI process that does nothing but insert an attribute (an alias in this case), locks the server up (contention lock with }Dimensions) and the service has to be stopped and restarted. This particular process runs without a lock contention from the same master process if ExecuteProcess is used instead of TM1RunTI.

So a couple of questions:

1) Is there any problem with executing processes from the DATA section instead of within the PROLOG section?
2) The attribute insert will not run through TM1RunTI without locking up the server regardless of attribute type. Is it because a dimension is being read as the datasource? If so, is there anyway around it other than using ExecuteProcess or ATTRInsert in the master process?

And before someone says something about the migration policy, I know. It is ridiculous, but it is not changing. Instead of frontal assaults on the policy, I've pretty much bypassed it on anything but actual TI process changes.

Ty

[Alan Kirk]

I have for several years used master processes in conjunction with a control cube to manage the execution of processes. The basic structure of the system would be:

* a master process reads the control cube for processes to execute along with parameters
* builds a datasource, typically a dimension subset
* the processes are executed via ExecuteProcess function in DATA tab.

This year I've started using TM1RunTI in order to minimize locking issues but have now run into a new problem. And maybe I just need some guidance on what one can/cannot or should do.

Because of the aggravation of migration policy to production environments, we have TI processes built to perform maintenance issues so it does not take four days to migration a dimension/attribute/etc change to production (as long as we use TI processes, we can make all the changes we want). The current problem is a TI process that does nothing but insert an attribute (an alias in this case), locks the server up (contention lock with }Dimensions) and the service has to be stopped and restarted. This particular process runs without a lock contention from the same master process if ExecuteProcess is used instead of TM1RunTI.

So a couple of questions:

1) Is there any problem with executing processes from the DATA section instead of within the PROLOG section?
2) The attribute insert will not run through TM1RunTI without locking up the server regardless of attribute type. Is it because a dimension is being read as the datasource? If so, is there anyway around it other than using ExecuteProcess or ATTRInsert in the master process?

In answer to 1) there are two problems that I can think of. The first is that the process will be executed for every single row of the data source. I know that there are situations where you may not be able to avoid that, but it has the potential for problems. The bigger problem is that metadata updates should be done no later than the metadata tab. Excluding the new "direct" functions, the metadata tab is where metadata changes (such as attribute changes) occur.

However I suspect that your locking issue is coming about from something that is pretty easy to overlook.

A string of calls to processes acts in the same way as a chore does. With a chore, some things are not committed until the end of the chore. Also the chore keeps track (for want of a better term) of all of the changes that it's making and therefore doesn't generally generate lock contentions because it knows "Oh, right, I'm the one who's doing that, that's fine".

The problem is that when you call RunTI you are creating something which is not part of the chore or set of called processes, as the case may be. Sure, you are calling it from a process BUT RunTI creates its own connection to the server, and then does its own updates. It's really no different to another user logging on and trying to do changes. The problem is that they system knows that the first user (your original string of processes) is already making changes to the metadata and hasn't finished yet. Then this new guy in the form of the RunTI connection to the server comes along and tries to do the same. Result? Contention lock.

Put simply, stick to doing the change by ExecuteProcess. I understand you wanting to split out the workload to run in parallel, and that's fine for data loads... but it's not a good idea for metadata changes.

[PavoGa]

Thank you for confirming something I suspected was the problem. In this case, it looks like just reading the dimension as a datasource is causing the lock contention. I'm going to do some further testing to see if I can get around that issue. I'll post the results.

You're right that the master process executes processes for every record. Using a counter dimension (1-...), those are processed in order, each record containing what is needed as to which process to execute, parameters to pass, etc. The parameters needed are fetched by CELLGET.

Thank you again.

[BrianL]

Just to add to what Alan said -

The problem is that when you call RunTI you are creating something which is not part of the chore or set of called processes, as the case may be. Sure, you are calling it from a process BUT RunTI creates its own connection to the server, and then does its own updates. It's really no different to another user logging on and trying to do changes. The problem is that they system knows that the first user (your original string of processes) is already making changes to the metadata and hasn't finished yet. Then this new guy in the form of the RunTI connection to the server comes along and tries to do the same. Result? Contention lock.

It can actually get worse than this. If you use ExecuteCommand to call TM1RunTI and set the wait parameter to one, you can completely deadlock TM1. The master process can be waiting for the runti command to finish while that process hit contention with the master process. This kind of deadlock requires manual intervention to cancel (at least) one of the processes.

The Execute command takes two parameters; the second one describes whether to have a synchronous call or an asynchronous call. Cognos TM1 tools should only be called asynchronously (Parameter 0) to avoid server deadlocks if the system is waiting for a lock held by the TurboIntegrator process and the process is waiting for the utility. The same advice applies to any executables called by ExecuteCommand if they login to Cognos TM1 .
Note: Never use a synchronous call if the tool logs into Cognos TM1 .

Source: http://www.ibm.com/support/knowledgecen ... runti.html

[PavoGa]

It can actually get worse than this. If you use ExecuteCommand to call TM1RunTI and set the wait parameter to one, you can completely deadlock TM1. The master process can be waiting for the runti command to finish while that process hit contention with the master process. This kind of deadlock requires manual intervention to cancel (at least) one of the processes.

LOL! Brian, it DID get worse than that. Ran into what you described like a brick wall.

Looks like the primary problem was cleared up by moving the entire DATA code into the PROLOG section and simply looping through a subset instead of reading it as a data source. It now runs fine, merrily updating dimensions, attributes, creating cubes, copying data from here to yon, etc. with no hang ups or lock contentions.

Unfortunately, you rightly point out something that has bothered me and one of our other developers as well for quite some time. We have trouble using the Synchronize function or any other method to control the processes to run synchronously. Every time we have tried, our server crashes. Nothing in the server log to describe the problem. It has to be something we do not understand about the mechanics of it.

Thanks again!

[BrianL]

That's the first time I've heard of the Synchronized() function crashing. It's been my experience that IBM takes crash issues seriously but YMMV.

[PavoGa]

Agreed. That is why I think we are doing something wrong. I'm going to look at it again and post back if still having issues. Thanks, Brian.

[PavoGa]

That's the first time I've heard of the Synchronized() function crashing. It's been my experience that IBM takes crash issues seriously but YMMV

Brian, I have gone back and tried the Synchronized function again. Just as soon as the second process starts, the server crashes. The process is called using TM1RunTI in a batch file. We must be doing something seriously wrong. Here is the PROLOG until the Synchronized function:

Code: Select all

###################
# parameters
# pDebug = debug flag  DEFAULT to 0
# pCube = the cube to load the rules for.
####################

NumericGlobalVariable('ProcessReturnCode');
NumericGlobalVariable('DataMinorErrorCount');

### Inititalize standard variables
strProcessName = GetProcessName();
tNow = NOW;
strNow = TIMST(NOW, '\Y\m\d\h\i\s');
strEnvironment = '';
strErrLogFolder = CELLGETS('Control Globals', 'Value', 'OutputPath');
strPrologErrLogFile = strErrLogFolder | strProcessName | '_PRO_err.log';
strMetaErrLogFile = strErrLogFolder | strProcessName | '_MET_err.log';
strDataErrLogFile = strErrLogFolder | strProcessName | '_DAT_err.log';
strEpilogErrLogFile = strErrLogFolder | strProcessName | '_EPI_err.log';
strDataFolder = CELLGETS('Control Globals', 'Value', 'TestOutputPath');
strTmp = 'tmp_' | strProcessName | strNow;
strEmailProcess = 'eMail - Ctrl';
strUser = TM1User();
cubEmail = 'eMail - Configuration';
nErrorFlag = 0;
nRecordsProcessed = 0;
nDataSourceCount = 0;

### Initialize other variables

cubRuleManagement = '}System - Rule Management';

Synchronized(cubRuleManagement);

This process reads this cube and writes the rule for a given cube. Works just fine without the Synchronized function. ParallelInteraction=T. Here are the last lines in the server log:

4576 [30] INFO 2016-06-08 22:04:03.873 TM1.Process Process "adm_Migration Management" executed by user "admin_ty"
6076 [33] INFO 2016-06-08 22:04:05.355 TM1.Process Process "adm_Load Rule from Rule Cube" executed by user "admin_ty"
6076 [33] DEBUG 2016-06-08 22:04:05.433 TM1.Lock.Exception Contention encountered attempting to acquire lock (0x000000000792A938) on object [Cube "Control Globals", addr=0x000000000792A010, index=R-1290] in IX mode at ..\tm1_r7s\TM1CubeImpl.cpp:19101 during function 'ProcessExecuteEx'.
Entering wait state 'IXCur'.
Blocked by the following 0 threads:

3132 [35] INFO 2016-06-08 22:04:06.416 TM1.Process Process "adm_Load Rule from Rule Cube" executed by user "admin_ty"

Have I completely missed understanding how Synchronized works?

Regardless, I am able to synchronize the processes, using a TI process in the PROLOG and EPILOG to effect a lock on a dimension to synchronize. But it would be nice to use the function as intended.

[BrianL]

I don't see anything wrong with this code.

One note to make is that Synchronized() takes an arbitrary string name, and doesn't require the name of an existing object on the server.

While I can reproduce with 10.2.2 FP5, There's a chance this is related to the combination of Lock.Exception logging and use of the Synchronized() function. If so, it's probably related to http://www-01.ibm.com/support/docview.w ... wg1PI42411

[PavoGa]

Thanks, Brian.

We do have the lock exception logging turned on and I'll try that later. With regards to the argument for the Synchronized function, I tried it both ways, with an existing object name and just a string value, both with the same result. Should have mentioned that.

Thank again. If that was the problem, I'll report back on it.

[PavoGa]

Brian,

Finally got around to testing and reproduced the problem with the server crashing when using SYNCHRONIZED and Lock.Exception logging turned on. Turned logging off and SYNCHRONIZED worked perfectly.

Thanks!