Transfer of data between servers

Suggest and discuss enhancements for TM1
Post Reply
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Transfer of data between servers

Post by Alan Kirk »

This isn't one of mine, but one that I just noticed in this week's incarnation of the RFE system:
TM1 does not a user friendly and easy mechanism to transfer data between different TM1 servers. Currently there are 2 options, but none of them can be properly used:

1 - Use Replication to transfer and synchronize data from different TM1 servers: This option is not useful if data is not manually entered in TM1 and recorded via transaction logs. If source data is calculated via rules and consolidations, then it does not work. If source data is transferred to TM1 cube via TI process without logging, then it does not work. Therefore this option is not useful in 90% of the cases.

2 - Using ODBO as data source in a TI process. This option is not user friendly at all because it works via MDX queries. It never worked for our requirements.
I used to use Replication during the last Ice Age. I don't know how it is for anyone else but I found it to be as "reliable" as any given system for picking horse race winners; which is to say, "not". It would randomly stop working for no obvious reason and you'd have to unpick the replication and try it again.... until the next time it randomly stopped working, lather, rinse and repeat until you finally said "You know what, stuff this" and wrote TIs to export the structure for reimport.
As a solution to this problem:

a new feature can be added in either of the following 2 options:

1 - Add an option to replication/synchronization definition in a way not use transaction log files but read the source cube itself when doing the synchronization every time. Moreover replication feature should also have the capability to read source cube views that are calculated via rules or dimension consolidations so that a high level summary of a source cube can be transferred to a target cube with less dimensions.

2 - In a TI process, when using IBM Cognos TM1 as data source, it should allow, not only using the cubes in the same TM1 server, but it should also allow connecting to a different TM1 server and using its cubes in the same way. In this option, it should allow using existing cubes views or subsets in source TM1 server.
I'm in agreement with option 2. Given the new REST API, this should not be an impossible challenge.

It has 8 votes at the moment and is still in the "needs review" stage. If you like the idea, I encourage voting early and voting often via the link in the opening paragraph.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
User avatar
mce
Community Contributor
Posts: 352
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: Transfer of data between servers

Post by mce »

I am also in favor of option 2. I do not understand how come IBMers can avoid adding this type of a very basic and must have functionality.
Such a tool that is not able to integrate its own data from its different databases. TM1 server A cannot read data from TM1 server B in a very simple and straightforward mechanism. That is a shame.
User avatar
mce
Community Contributor
Posts: 352
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: Transfer of data between servers

Post by mce »

mce wrote: Tue Jun 02, 2020 11:43 pm Such a tool that is not able to integrate its own data from its different databases. TM1 server A cannot read data from TM1 server B in a very simple and straightforward mechanism. That is a shame.
In between... :)
User avatar
scrumthing
Posts: 81
Joined: Tue Jan 26, 2016 4:18 pm
OLAP Product: TM1
Version: 11.x
Excel Version: MS365

Re: Transfer of data between servers

Post by scrumthing »

I don't think something like this will happen. They want to get rid of TI and not improve it...
You can do it relatively simple with the rest api and for example tm1py. That offers in any case much more flexibility and a better performance than TI would, I think.
There is no OLAP database besides TM1!
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: Transfer of data between servers

Post by Alan Kirk »

scrumthing wrote: Wed Jun 03, 2020 8:52 am I don't think something like this will happen. They want to get rid of TI and not improve it...
In the interview that I did with Hubert he stated straight out that that won't be happening in the next version; it's something further down the road. In any case, this isn't really about TI as such, it's about the ease of creating a data source which will just happen to be used in TI at present. In an ideal world it should be no more difficult to use an object on another server as a data source than it is to use an object on the current server, without the need to write screeds of code or query text.

Much as I have advocated strongly for users to learn MDX and the REST API, and continue to do so, I recognise that there will be quite a few people who simply don't want to do that through either lack of time or lack of inclination.
scrumthing wrote: Wed Jun 03, 2020 8:52 am You can do it relatively simple with the rest api and for example tm1py. That offers in any case much more flexibility and a better performance than TI would, I think.
That would require something external to the server, which is in my view something that should be a last resort. Second, if the TI data source were to use the REST API to pull from the other server then by definition there shouldn't really be any difference in performance.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
Wim Gielis
MVP
Posts: 3105
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1, Jedox
Version: PAL 2.0.9.18
Excel Version: Microsoft 365
Location: Brussels, Belgium
Contact:

Re: Transfer of data between servers

Post by Wim Gielis »

Alan Kirk wrote: Tue Jun 02, 2020 10:41 pmIt has 8 votes at the moment
11 now, after having added mine.

Just to add that in tools such as Jedox, this is all standard. You would have different Integrator projects, all of which can interact with multiple Modeler projects.
Last edited by Wim Gielis on Fri Jun 12, 2020 9:56 pm, edited 1 time in total.
Best regards,

Wim Gielis

IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
User avatar
Ajay
Regular Participant
Posts: 183
Joined: Wed May 14, 2008 8:27 am
OLAP Product: TM1
Version: 10.2.0, PA 2.0.9
Excel Version: 2016
Location: London

Re: Transfer of data between servers

Post by Ajay »

Totally agree, and have added my vote to this too.

I'm accomplishing this at the moment, with some TI code and batch files, as I am restricted to output what I need from server A, in *.txt files, then move it via batch code to another location, on server B and process the files there using TI.

It is a bit of a "faff", but I agree that a datasource would be the way to go with this.

Ajay
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: Transfer of data between servers

Post by Alan Kirk »

At least this one has also now moved to the "Future Consideration" pile.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
User avatar
gtonkin
MVP
Posts: 1192
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: Latest and greatest
Excel Version: Office 365 64-bit
Location: JHB, South Africa
Contact:

Re: Transfer of data between servers

Post by gtonkin »

Just saw that too - no doubt it will be via the RestAPI but fine as long as it is native, easily configurable and lightning fast with error-handling etc. etc.
User avatar
mce
Community Contributor
Posts: 352
Joined: Tue Jul 20, 2010 5:01 pm
OLAP Product: Cognos TM1
Version: Planning Analytics Local 2.0.x
Excel Version: 2013 2016
Location: Istanbul, Turkey

Re: Transfer of data between servers

Post by mce »

This RFE has 21 votes now. Hope to get more votes on this so that it can be prioritized by IBMers.
https://ibm-data-and-ai.ideas.aha.io/ideas/PAOC-I-288
Post Reply