TM1 Process Self Triggering

[guru1988fortm1]

Hi All,

I am having a Master Process which calls many other sub processes. These sub processes are written in both prolog and Epilog

The total time to run this process must be around 9 hours (adding up the times of the individual processes)

Now my problem is when the last sub process in the epilog is completed, i am not getting the 'Process getting sucessfully' or 'Finished with erros' etc.,

Instead, my master process is triggered again. When i check in logs , it says the process is executed by me

28960 [1] INFO 2020-05-30 17:52:17.466 TM1.Process Process "Sys.TI.Tracker": finished executing normally, elapsed time 0.00 seconds
26888 [] WARN 2020-05-30 17:52:17.560 Event.Thread Thread[28960] is blocking 72 other threads, and is in state [Run].
14444 [a] INFO 2020-05-30 17:52:18.072 TM1.Process Process "GP.Cub.Master.Load" executed by user "Admin"

Note : In the last process of the epilog , i am using run process to run multiple other TIs in paralle, even after completion, the threads stay in the top .



Appreciate your thoughts on this

Thanks,
Regards
Guru Madheshwaran

[declanr]

Before the process has finished; a chore etc has kicked off another process that has caused contention (often a savedatall function) which then means that all transactions from your Master Process and its Sub Processes have rolled back and then started again when it has a clean window to do so.

[guru1988fortm1]

But i dont have any chores scheduled in the instance

[Alan Kirk]

But i dont have any chores scheduled in the instance

Do you happen to have a SaveTime parameter set in your server config file, which could amount to the same thing? (Which is why Declan said "ecc"; he didn't say it was the only possibility, just one possibility. As is this.)

[guru1988fortm1]

Thanks Declan and Alan!!!

I was just scanning through all the sub processes if there is a save data in them .

But Thank you for making me realize that my process is rolling back which i had no idea earlier, i was just wondering why is it running again and again

Also, i am using Run process in two of the process in the epilog

Epilog of master process

Process 1 ---- uses Run process to run 80 processes (20 in parallel )

Process 2
Process 3
Process 4
Process 5

Process 6 --- uses run process to run 500 processes (20 in parallel ) (It has save data over one particular cube)

i will remove the save data of that particular cube and try running again.

Also, if the process rollback, will there be any logs or something where it rolls back because i had no idea until you told me

Thanks,
Regards
Guru Madheshwaran

[paulsimon]

Hi

Just as a gut feel, a process that runs for 9 hours seems excessive. Are the processes perhaps reading from a SQL data source that is delivering data very slowly? I have certainly seen cases where concentrating on the data source has brought more benefits than any tinkering with TM1. Are the SQL queries running efficiently? Have appropriate indexes been created, etc? If nothing can be done about the data source, I have certainly seen cases where landing the data into CSV files and then processing these has improved overall performance because TM1 is not waiting around for data to be delivered to it.

If there is a lot of contention, due to meta data updates, etc, which cannot be run in parallel, you might find that you aren't really getting any benefit from running processes in parallel. Your profile still indicates that you are on version 9.5.2. If you are actually on PA, then using temporary subsets and views for things like clearing existing data can do a lot to avoid contention.

Similarly, you will typically only be making minor changes to dimensions. Having one pass over the data on the data tab and using the Direct versions of TI dimension statements to insert the occasional new element can be much faster than having two passes over the data on both metadata and data tabs, with a full dimension rebuild occurring at the end of the metadata.

Check processes that are only loading data to make sure there are no statements on metadata tab otherwise you will be needlessly be doing two passes over the data source.

Look for cases where processes are loading data into a cube from a data source, and then another process is reading data from that cube and writing it to another cube, etc. It can be much more efficient to load both cubes directly from the data source, because you eliminate the pass over the intermediate cube.

Regards

Paul SImon

[ykud]

I'd also look into this:
Event.Thread Thread[28960] is blocking 72 other threads

Something is taking a lock and holding it forcing your main process to roll back and restart. SaveCubeData is server side lock, as is SecurityRefresh, ViewConstruct and a few dimension change functions, so I'd scan your subprocesses for those.

Are you running sub-processes with ExecuteProcess? This accumulates the locks in the main process, it might be worthwhile to run the subprocesses as separate runti calls so that each 'commits' it's lock cleanly.
Overall I'd also check if you can convert this process into a multi-commit mode chore -- it'll be less locking & rollback prone this way.

Cheers,
Y

[guru1988fortm1]

Hi

Just as a gut feel, a process that runs for 9 hours seems excessive. Are the processes perhaps reading from a SQL data source that is delivering data very slowly? I have certainly seen cases where concentrating on the data source has brought more benefits than any tinkering with TM1. Are the SQL queries running efficiently? Have appropriate indexes been created, etc? If nothing can be done about the data source, I have certainly seen cases where landing the data into CSV files and then processing these has improved overall performance because TM1 is not waiting around for data to be delivered to it.

If there is a lot of contention, due to meta data updates, etc, which cannot be run in parallel, you might find that you aren't really getting any benefit from running processes in parallel. Your profile still indicates that you are on version 9.5.2. If you are actually on PA, then using temporary subsets and views for things like clearing existing data can do a lot to avoid contention.

Similarly, you will typically only be making minor changes to dimensions. Having one pass over the data on the data tab and using the Direct versions of TI dimension statements to insert the occasional new element can be much faster than having two passes over the data on both metadata and data tabs, with a full dimension rebuild occurring at the end of the metadata.

Check processes that are only loading data to make sure there are no statements on metadata tab otherwise you will be needlessly be doing two passes over the data source.

Look for cases where processes are loading data into a cube from a data source, and then another process is reading data from that cube and writing it to another cube, etc. It can be much more efficient to load both cubes directly from the data source, because you eliminate the pass over the intermediate cube.

Regards

Paul SImon

Thanks Paul!!

Yes, we are loading data from EDW data sources . Almost 25 tables are read for both metadata and data loads
Also, 9 hours hasnt been a big problem because this is an yearly submission job which loads the whole year's data into TM1
Its a master process that loads data + does some inter cube calculations etc.,
Sorry about the verison, we are using PA version, will update my profile

Thanks,
regards
Guru Madhesh

[guru1988fortm1]

I'd also look into this:
Event.Thread Thread[28960] is blocking 72 other threads

Something is taking a lock and holding it forcing your main process to roll back and restart. SaveCubeData is server side lock, as is SecurityRefresh, ViewConstruct and a few dimension change functions, so I'd scan your subprocesses for those.

Are you running sub-processes with ExecuteProcess? This accumulates the locks in the main process, it might be worthwhile to run the subprocesses as separate runti calls so that each 'commits' it's lock cleanly.
Overall I'd also check if you can convert this process into a multi-commit mode chore -- it'll be less locking & rollback prone this way.

Cheers,
Y

Thanks ykud!!

As you and declanr said, the reason for rollback is that one of the sub processes was having a cubesavedata over a particular cube . That was the reason for roll back , i removed it but facing another issue which was the reason for above lock you had mentioned

I am having a sub process that has a run process script in it as below

maxsize is the size of dimension containing around 150 elements

WHILE( i<=MaxSize );
sElement=DIMNM(dimension, index)
RunProcess(Element passed as parameter and a process is called );

# Only a series of 10 should be run in parallel
IF(MOD(i,10)=0);
Sleep(360000);
ENDIF;
i = i + 1;
END;

When i run this subprocess separately , it runs perfectly fine. 10 threads run at a time and gets completed. After 6 mins, the next 10 runs and so on

but when i put this subprocess containing the above code into epilog of my master process or in any other process,
All the 10 threads go on wait when the sub process is triggered and after 6 mins, the next 10 threads start and go on wait , so in the end all 150 threads are all on wait.

Any idea about this run process stuff!!!

Also, thought of putting this in a chore with multi commit mode but it crashes the server as the subprocess is using run process in it

https://www.tm1forum.com/viewtopic.php?f=3&t=15168


Thanks
Regards
Guru Madhesh

[ykud]

As you and declanr said, the reason for rollback is that one of the sub processes was having a cubesavedata over a particular cube . That was the reason for roll back , i removed it but facing another issue which was the reason for above lock you had mentioned

Try adding log4j.logger.TM1.Lock.Exception=DEBUG to your tm1s-properties properties and see what object is being locked?
A dimension or a cube?

Also set Reducecubelockingondimensionupdate to T if you haven't already.

Cheers,
Y

[PavoGa]

Thanks ykud!!

As you and declanr said, the reason for rollback is that one of the sub processes was having a cubesavedata over a particular cube . That was the reason for roll back , i removed it but facing another issue which was the reason for above lock you had mentioned

I am having a sub process that has a run process script in it as below

maxsize is the size of dimension containing around 150 elements

Code: Select all

WHILE( i<=MaxSize );
	        sElement=DIMNM(dimension, index)
		RunProcess(Element passed as parameter and a process is called );

		# Only a series of 10 should be run in parallel 
		IF(MOD(i,10)=0);
			Sleep(360000);
		ENDIF;
		i = i + 1;
	END;

When i run this subprocess separately , it runs perfectly fine. 10 threads run at a time and gets completed. After 6 mins, the next 10 runs and so on

but when i put this subprocess containing the above code into epilog of my master process or in any other process,
All the 10 threads go on wait when the sub process is triggered and after 6 mins, the next 10 threads start and go on wait , so in the end all 150 threads are all on wait.


Thanks
Regards
Guru Madhesh

The DIMIX and DIMNM functions put a read lock on the dimension. I do not know what is going on in the subprocess, but you may want to try building a temporary subset which avoids the lock issue and looping through it.