Page 1 of 1

Transfer of data between servers

Posted: Tue Jun 02, 2020 10:41 pm
by Alan Kirk
This isn't one of mine, but one that I just noticed in this week's incarnation of the RFE system:
TM1 does not a user friendly and easy mechanism to transfer data between different TM1 servers. Currently there are 2 options, but none of them can be properly used:

1 - Use Replication to transfer and synchronize data from different TM1 servers: This option is not useful if data is not manually entered in TM1 and recorded via transaction logs. If source data is calculated via rules and consolidations, then it does not work. If source data is transferred to TM1 cube via TI process without logging, then it does not work. Therefore this option is not useful in 90% of the cases.

2 - Using ODBO as data source in a TI process. This option is not user friendly at all because it works via MDX queries. It never worked for our requirements.
I used to use Replication during the last Ice Age. I don't know how it is for anyone else but I found it to be as "reliable" as any given system for picking horse race winners; which is to say, "not". It would randomly stop working for no obvious reason and you'd have to unpick the replication and try it again.... until the next time it randomly stopped working, lather, rinse and repeat until you finally said "You know what, stuff this" and wrote TIs to export the structure for reimport.
As a solution to this problem:

a new feature can be added in either of the following 2 options:

1 - Add an option to replication/synchronization definition in a way not use transaction log files but read the source cube itself when doing the synchronization every time. Moreover replication feature should also have the capability to read source cube views that are calculated via rules or dimension consolidations so that a high level summary of a source cube can be transferred to a target cube with less dimensions.

2 - In a TI process, when using IBM Cognos TM1 as data source, it should allow, not only using the cubes in the same TM1 server, but it should also allow connecting to a different TM1 server and using its cubes in the same way. In this option, it should allow using existing cubes views or subsets in source TM1 server.
I'm in agreement with option 2. Given the new REST API, this should not be an impossible challenge.

It has 8 votes at the moment and is still in the "needs review" stage. If you like the idea, I encourage voting early and voting often via the link in the opening paragraph.

Re: Transfer of data between servers

Posted: Tue Jun 02, 2020 11:43 pm
by mce
I am also in favor of option 2. I do not understand how come IBMers can avoid adding this type of a very basic and must have functionality.
Such a tool that is not able to integrate its own data from its different databases. TM1 server A cannot read data from TM1 server B in a very simple and straightforward mechanism. That is a shame.

Re: Transfer of data between servers

Posted: Tue Jun 02, 2020 11:45 pm
by mce
mce wrote: Tue Jun 02, 2020 11:43 pm Such a tool that is not able to integrate its own data from its different databases. TM1 server A cannot read data from TM1 server B in a very simple and straightforward mechanism. That is a shame.
In between... :)

Re: Transfer of data between servers

Posted: Wed Jun 03, 2020 8:52 am
by scrumthing
I don't think something like this will happen. They want to get rid of TI and not improve it...
You can do it relatively simple with the rest api and for example tm1py. That offers in any case much more flexibility and a better performance than TI would, I think.

Re: Transfer of data between servers

Posted: Wed Jun 03, 2020 10:37 pm
by Alan Kirk
scrumthing wrote: Wed Jun 03, 2020 8:52 am I don't think something like this will happen. They want to get rid of TI and not improve it...
In the interview that I did with Hubert he stated straight out that that won't be happening in the next version; it's something further down the road. In any case, this isn't really about TI as such, it's about the ease of creating a data source which will just happen to be used in TI at present. In an ideal world it should be no more difficult to use an object on another server as a data source than it is to use an object on the current server, without the need to write screeds of code or query text.

Much as I have advocated strongly for users to learn MDX and the REST API, and continue to do so, I recognise that there will be quite a few people who simply don't want to do that through either lack of time or lack of inclination.
scrumthing wrote: Wed Jun 03, 2020 8:52 am You can do it relatively simple with the rest api and for example tm1py. That offers in any case much more flexibility and a better performance than TI would, I think.
That would require something external to the server, which is in my view something that should be a last resort. Second, if the TI data source were to use the REST API to pull from the other server then by definition there shouldn't really be any difference in performance.

Re: Transfer of data between servers

Posted: Wed Jun 03, 2020 11:04 pm
by Wim Gielis
Alan Kirk wrote: Tue Jun 02, 2020 10:41 pmIt has 8 votes at the moment
11 now, after having added mine.

Just to add that in tools such as Jedox, this is all standard. You would have different Integrator projects, all of which can interact with multiple Modeler projects.

Re: Transfer of data between servers

Posted: Thu Jun 11, 2020 6:55 am
by Ajay
Totally agree, and have added my vote to this too.

I'm accomplishing this at the moment, with some TI code and batch files, as I am restricted to output what I need from server A, in *.txt files, then move it via batch code to another location, on server B and process the files there using TI.

It is a bit of a "faff", but I agree that a datasource would be the way to go with this.

Ajay

Re: Transfer of data between servers

Posted: Wed Sep 16, 2020 7:03 pm
by Alan Kirk
At least this one has also now moved to the "Future Consideration" pile.

Re: Transfer of data between servers

Posted: Wed Sep 16, 2020 7:12 pm
by gtonkin
Just saw that too - no doubt it will be via the RestAPI but fine as long as it is native, easily configurable and lightning fast with error-handling etc. etc.

Re: Transfer of data between servers

Posted: Thu Dec 03, 2020 11:56 am
by mce
This RFE has 21 votes now. Hope to get more votes on this so that it can be prioritized by IBMers.
https://ibm-data-and-ai.ideas.aha.io/ideas/PAOC-I-288