MTCubeLoad

Post Reply
User avatar
mce
Community Contributor
Posts: 352
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

MTCubeLoad

Post by mce »

Hi,

We added MTCubeLoad=T and MTQ=15. Then after few days, by chance, a user realized that some numbers in a cube have disappeared.
We loaded a backup of few days old in a separate environment and we saw, the numbers are there in the cube.
When I checked thed CubeName.cub file, in both the backup and the live environment, we have exactly the same file, but TM1 displays different numbers. Then after we remove the MTCubeLoad=T parameter in the live system, we started seeing the numbers normally as in the backup model.

I just ralized the following note in IBM page at https://www.ibm.com/support/knowledgece ... eload.html.

Code: Select all

"Setting MTCubeLoad=T does not work in all cases. When issues are detected, you must disable the multi-threaded loading of individual cubes."
So IBM suggestion is to check each and every cube to make sure there are no issues after adding this parameter. I have more than 160 cube in this TM1 server. So how can we check this and make sure? What guarantees that this problem will not happen again in a new cube? IBM's suggestion is complete non-sense. At least, there must information explaining the conditions that the issue may happen so that we check those cases.

I would expect this issue to be resolved and this note must then be removed. This is an obvious DEFECT! Hope someone from IBM hears about this comment.

Regards,
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: MTCubeLoad

Post by Alan Kirk »

mce wrote: Sat May 04, 2019 9:45 am Hi,

We added MTCubeLoad=T and MTQ=15. Then after few days, by chance, a user realized that some numbers in a cube have disappeared.
We loaded a backup of few days old in a separate environment and we saw, the numbers are there in the cube.
When I checked thed CubeName.cub file, in both the backup and the live environment, we have exactly the same file, but TM1 displays different numbers. Then after we remove the MTCubeLoad=T parameter in the live system, we started seeing the numbers normally as in the backup model.

I just ralized the following note in IBM page at https://www.ibm.com/support/knowledgece ... eload.html.

Code: Select all

"Setting MTCubeLoad=T does not work in all cases. When issues are detected, you must disable the multi-threaded loading of individual cubes."
So IBM suggestion is to check each and every cube to make sure there are no issues after adding this parameter. I have more than 160 cube in this TM1 server. So how can we check this and make sure? What guarantees that this problem will not happen again in a new cube? IBM's suggestion is complete non-sense. At least, there must information explaining the conditions that the issue may happen so that we check those cases.

I would expect this issue to be resolved and this note must then be removed. This is an obvious DEFECT! Hope someone from IBM hears about this comment.

Regards,
Do you use conditional feeders at all? Multi-threaded loading is something which is best avoided if you do, because you can end up with exactly that result.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
User avatar
mce
Community Contributor
Posts: 352
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: MTCubeLoad

Post by mce »

Alan Kirk wrote: Sat May 04, 2019 6:30 pm Do you use conditional feeders at all? Multi-threaded loading is something which is best avoided if you do, because you can end up with exactly that result.
This issue has nothing to do with feeders. Some of the input data does not appear with MTCubeLoad=T.
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: MTCubeLoad

Post by Alan Kirk »

mce wrote: Sat May 04, 2019 10:06 pm
Alan Kirk wrote: Sat May 04, 2019 6:30 pm Do you use conditional feeders at all? Multi-threaded loading is something which is best avoided if you do, because you can end up with exactly that result.
This issue has nothing to do with feeders.
Some of the input data does not appear with MTCubeLoad=T.
Well excuse the hell out of me for raising a thing that is known to cause issues with multi threaded loads, especially as you made it clear in the original post that you were talking about input values rather than calculated values... as you didn't.

Granted I should have read through the article that you linked to first since this would seem to be a rather different animal to the older method but, at the risk of eliciting another dose of tone, IBM does also raise the option of turning it off.

So in answer to the question of how you can know whether it will happen with a future cube, you don't. Therefore the most logical solution would be "don't use it until or unless IBM can remove the proviso."
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
User avatar
mce
Community Contributor
Posts: 352
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: MTCubeLoad

Post by mce »

Alan Kirk wrote: Sat May 04, 2019 11:18 pm Well excuse the hell out of me for raising a thing that is known to cause issues with multi threaded loads, especially as you made it clear in the original post that you were talking about input values rather than calculated values... as you didn't.
Thanks for bringing the conditional feeder issue to my attention. In fact, I use conditional feeders a lot. Today I am upgrading a model with many conditional feeders into 2.0.7 and adding MTFeeders=T. I expect the feeder processing time to improve especially with CubeProcessFeeders command. But I am keeping MTFeeders.AtStartup=F due to potential issues with conditional feeders.

I was excited about MTCubeLoad and MTFeeders.AtStartup=F after seeing how fast server load time improved. But unfortunately not able to use these due to these issues.

Regards,
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: MTCubeLoad

Post by Alan Kirk »

mce wrote: Sat May 04, 2019 11:39 pm I was excited about MTCubeLoad and MTFeeders.AtStartup=F after seeing how fast server load time improved. But unfortunately not able to use these due to these issues.

Regards,
Understandable. However I can't for the life of me understand why IBM would turn something like this loose in the wild when (based on the comment in their document) they know full well that it's not 100% reliable.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: MTCubeLoad

Post by Alan Kirk »

I've been thinking about this further.

What I said earlier still holds; if IBM can't be clear about when the functionality can fail, and indicate that it CAN fail, I'd just turn it off and stay away from it.

However... if issues like this go unreported they'll never get fixed. And for it to get fixed it would be necessary to know EXACTLY what is failing.

Here is what I would do if time allowed.
- Recreate what you had done earlier; having the setting on in one environment and off in another from exactly the same cube files.
- Create a TI with a data source of my cubes dimension;
- For each cube, call the Bedrock.Cube.Data.Export function. (If you use Bedrock. If not, it's probably the easiest way to get from A to B in a case like this.)
- That process can exclude consolidations and rule derived values. It's a generic process that can work on any cube up to 27 dimensions.
- Have the output files from one environment go to one folder, the output files from the other to a different one.
- Use Winmerge to look for differences in the folders. If there are differences in the file sizes from the two environments, then you'll know that there are differences in the cube concerned.
- You can then use Winmerge to load up those two files and see how many values differ.
- If it turned out that there are too many variances to manually analyse, I'd probably load the files into either a database or even back to a TM1 server and run a query / process as the case may be to identify variances.

It MAY be that a pattern emerges, or it may not.

Either way, it gives you hard evidence of not only variances but WHERE those variances occur, which can then be raised as a support ticket.

The next bit may get a bit tricky. I'd bet good money that IBM support will ask for your entire model. And I'll bet that your company's response will be "no way!", NDA or not. And at that point the request may reach an impasse. But if a GOOD support person gets it, they may be able to figure out what's going on from the data you've supplied.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
User avatar
mce
Community Contributor
Posts: 352
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: MTCubeLoad

Post by mce »

Alan Kirk wrote: Sun May 05, 2019 9:57 am I've been thinking about this further.

What I said earlier still holds; if IBM can't be clear about when the functionality can fail, and indicate that it CAN fail, I'd just turn it off and stay away from it.

However... if issues like this go unreported they'll never get fixed. And for it to get fixed it would be necessary to know EXACTLY what is failing.

Here is what I would do if time allowed.
- Recreate what you had done earlier; having the setting on in one environment and off in another from exactly the same cube files.
- Create a TI with a data source of my cubes dimension;
- For each cube, call the Bedrock.Cube.Data.Export function. (If you use Bedrock. If not, it's probably the easiest way to get from A to B in a case like this.)
- That process can exclude consolidations and rule derived values. It's a generic process that can work on any cube up to 27 dimensions.
- Have the output files from one environment go to one folder, the output files from the other to a different one.
- Use Winmerge to look for differences in the folders. If there are differences in the file sizes from the two environments, then you'll know that there are differences in the cube concerned.
- You can then use Winmerge to load up those two files and see how many values differ.
- If it turned out that there are too many variances to manually analyse, I'd probably load the files into either a database or even back to a TM1 server and run a query / process as the case may be to identify variances.

It MAY be that a pattern emerges, or it may not.

Either way, it gives you hard evidence of not only variances but WHERE those variances occur, which can then be raised as a support ticket.

The next bit may get a bit tricky. I'd bet good money that IBM support will ask for your entire model. And I'll bet that your company's response will be "no way!", NDA or not. And at that point the request may reach an impasse. But if a GOOD support person gets it, they may be able to figure out what's going on from the data you've supplied.

The user realized the issue in a very small plan cube. The variances seem to be random, not based on any specific dimension element.
If time allows, I will dig into details of this issue and open a support ticket. Thanks for the comments.
EvgenyT
Community Contributor
Posts: 324
Joined: Mon Jul 02, 2012 9:39 pm
OLAP Product: TM1
Version: PAL 2.0.8
Excel Version: 2016
Location: Sydney, Australia

Re: MTCubeLoad

Post by EvgenyT »

Hi,

2 Cents, MTCubeLoad does not affect feeder processing at all, MTFeeders do.

Please look at this thread where this topic has been discussed in details:

https://www.tm1forum.com/viewtopic.php? ... 878#p71878

Thanks

Evgeny
Post Reply