Replication - worth it ? or not ?

Ajay · Post by **Ajay** » Mon Jan 26, 2009 4:49 pm

Hi All,

We are having some performance issues on network speeds and MPLS issues for users in some our more distant countries.

As a result we were pushing ahead with some pretty expensive Citrix development when we did think that maybe a local server with replication may do the trick.

We definitely have the down time to do the replication to the UK server, but I wanted to know if any of you have acheived this and what were your experiences. Did you think it was a worthwhile exercise or not. Certainly from a cost perspective it certainly makes sense, but am i about to walk the plank with the idea ?

Let me know what to expect and what are the key do's and don'ts

Cheers
Ajay

x64bit 9.0 SP3 U7, w2003 OS Enterprise Edition, 200 users, 1TB HD, 72GB RAM

ScottW · Post by **ScottW** » Mon Jan 26, 2009 10:59 pm

If you are talking a pure "Star -> Planet" one way replication for a reporting only satellite server then replication works fine. But IMO if you're talking about only replicating reporting cubes locally for better performance then this can probably be just as easily achieved by scheduling an TI to dump text files of dimension and cube data, ftp pull them to the remote server, then TI them in. This will work fine if your cube data is not too huge. [If you do have a large data set then you can get TM1 to spit out a delta file instead to ftp if you set up appropriate versions.]

If the cubes require write-back in both locations then replication is the only way to achieve this. Generally replication works pretty well but can be tedious to set up and can require a fairly long batch window if datasets are large and/or there is heavy data entry. Although replication works OK in most instances, it can be very difficult to get it to work over poor networks, especially if latency and/or drop-outs is bad.

General: replication relies on logging to synchronise the data so you need to be careful about logging settings. I would recommend setting up a replication in your environment to test it out and see if you are happy with it.

Replication (or better phrased synchronisation between several servers) is definitely an area where TM1 needs to improve!

TJMurphy · Post by **TJMurphy** » Tue Jan 27, 2009 9:06 am

We're using replication here and it's been fine. However, one server is read-write, the other one is read only so there's no risk of conflicts. I've set up a couple of big reports to check the two servers are in synch and run these once a week. On the rare occassions we've had a problem it seems (iirc) that there had been network problems on our main server *or* someone had used the "backout" facility from the message log which didn't seem to create a log file entry. We've banned anyone (mostly me

) from using the backout facility for a few versions now so that might be fixed.

What we have found is that if a cube gets out of synch then you have to delete and recreate the synchronisation, I've never tried to copy the cubes over and see if that works instead.

Tony

Martin Ryan · Post by **Martin Ryan** » Tue Jan 27, 2009 9:18 am

I haven't done any replication, but have heard a few horror stories. If it were me, I'd try to find a TI workaround. For example, if you have two servers in two locations, and the locations look after their own departments, then it's easy enough to dump out only those departments and transfer them across to the other server.

I have only heard good things about Citrix, provided the person who set it up knew what they were doing.

Martin

Steve Vincent · Post by **Steve Vincent** » Tue Jan 27, 2009 9:29 am

TJMurphy wrote:...*or* someone had used the "backout" facility from the message log which didn't seem to create a log file entry. We've banned anyone (mostly me ) from using the backout facility for a few versions now so that might be fixed.

Tony, that's an interesting and somewhat worrying point. I tried replication in 8.1.8 in a test area (just to copy a dim) and it screwed up royally by deleting lots of things it shouldn't, so i've stayed well away from it since. Not sure which version you are on but 9.0 does not log transactions when they are backed out either, so if replication only uses logs to do its work then there is a massive and fatal flaw with it. What if you have massive cubes and don't want it all logged but replicate it once a week? If that's right then it's useless...

Would be interesting to know from someone with 9.4 if backing out a cube with logging turned on actually logs the reversals

Ajay · Post by **Ajay** » Tue Jan 27, 2009 7:37 pm

Thanks guys, all of your comments very useful. Are there any protocols or standards which would help the build of the replication environment and/or jobs ?

For example should the two servers be on the same WAN or can remote external IP addresses be OK ?

Any naming conventions to be used or indeed avoided ?

Cheers
Ajay

TJMurphy · Post by **TJMurphy** » Wed Jan 28, 2009 9:19 am

Steve Vincent wrote:
TJMurphy wrote:...*or* someone had used the "backout" facility from the message log which didn't seem to create a log file entry. We've banned anyone (mostly me ) from using the backout facility for a few versions now so that might be fixed.
Tony, that's an interesting and somewhat worrying point. I tried replication in 8.1.8 in a test area (just to copy a dim) and it screwed up royally by deleting lots of things it shouldn't, so i've stayed well away from it since. Not sure which version you are on but 9.0 does not log transactions when they are backed out either, so if replication only uses logs to do its work then there is a massive and fatal flaw with it. What if you have massive cubes and don't want it all logged but replicate it once a week? If that's right then it's useless...

Would be interesting to know from someone with 9.4 if backing out a cube with logging turned on actually logs the reversals

We're only replicating cubes on our system (which picks up the dimensions). I've never tried to replicate just a dimension - how did you set it up to do that?

Just tried a one line change on a cube now and backed it out using the transaction log and it created a log file entry. So, that one's fixed in our current version - 9.1 SP3. (Not sure I wanted to know that, I might be tempted to start using it again now!).

Tony

Alan Kirk · Post by **Alan Kirk** » Wed Jan 28, 2009 8:29 pm

TJMurphy wrote: We're only replicating cubes on our system (which picks up the dimensions). I've never tried to replicate just a dimension - how did you set it up to do that?

We only replicate dimensions, not cubes. However it's a slight cheat because you don't REALLY replicate the dimension alone; you replicate the attributes cube FOR the dimension.

(If you don't have attributes for that dimension, just create a dummy one then replicate the resulting cube.)

In 8.2.12 at least, this works well... except that even when set to replicate, the subsets often don't. This hasn't been a large enough problem for me to investigate in detail.

Mike Cowie · Post by **Mike Cowie** » Wed Jan 28, 2009 10:10 pm

One thing to watch out for with replication (which I, frankly, haven't dealt with directly in a long time) is that if you set Rules to synchronize along with a cube, it used to be the case that Synchronization would cause your Rules to be re-saved whenever a Dimension in the cube that has the rules is synchronized/updated. This might not seem like too much of a big deal, unless your rules take quite a long time to save. Quite a headache if you aren't looking out for it.

Hope that helps.

Regards,
Mike

mattgoff · Post by **mattgoff** » Fri Jan 30, 2009 11:58 am

Had a busy patch there, so just now catching up on the forums....

We use replication extensively. We have a master star server in San Francisco and eight planets scattered all over the world that synchronize with it. The number one thing I can tell you is that it is by NO MEANS a reliable synchronization mechanism. Although the database objects do seem to stay clean (elements, hierarchies, dims, cubes, rules), the data can become unsynchronized without warning. Since replication is incremental, if any update fails, it stays failed with no simple "force resynch" mechanism built into TM1. My workaround was to build a TI process to export cube data, ViewZeroOut, and import to force new logs to get written (and effectively force a resynch).

I've done as much as I can to get things as stable as possible, and after implementing the following guidelines I only have the occasional data issue (although since it still happens, vigilence is required):

SaveDataAll on both servers before every replication. Since replication is pull, you have to work out your timing and schedule a separate process on the star to occur before replication begins.
No overlapping replications. Since we have eight planets and replication duration can vary greatly, once I add in some buffer time I can't fully synchronize more than once every two hours. Normally replication only takes 2-3 minutes, but if I have to force a resynch (via my workaround) it can take a lot longer.
No rolling back transactions.

I have seriously considered limiting replication to only synchronize TM1 objects and moving data replication to my own TI and external scripting. Long term, I'm working with Cognos to try to resolve the replication issues, but as with most difficult-to-reproduce issues it's not moving fast....

Matt

Marcus Scherer · Post by **Marcus Scherer** » Fri Jan 30, 2009 1:08 pm

Hi,
very interesting answers from you Gurus. One more question to the subject: in older versions the replication of attributes was not supported. Does someone know if it is supported with V. 9?

TIA,

Marcus

mattgoff · Post by **mattgoff** » Fri Jan 30, 2009 3:52 pm

Yes. We started on 9.0.2 (now on 9.1.4), so I can't compare and contrast to pre-9.0 versions, but, our attributes are stored in cubes and replicate like any other cube.

Matt

mattgoff · Post by **mattgoff** » Sun Feb 08, 2009 12:04 pm

I've been struggling with a replication emergency all weekend and have discovered something very interesting. Looking through the replication logs, it appears that every object to be replicated requires a minimum of one second to process over our WAN. Why is this a big deal? Well, objects include cubes, rules, views, and subsets! In our case, I had originally set up replication using default parameters for each cube, so seven cubes was taking 180ish seconds to replicate WHEN ALL DATA WAS ALREADY SYNCHRONIZED! If I've done a refresh from Oracle on the master, replication is 180 secs + a lot more time to sync the data. This is a big problem, esp during close, since replication sets a global lock (on the planet) while it runs (at least on 9.1.4).

This all goes back to my long-standing complaint about how TM1 handles network operations: nothing is batched. It's written like an interprocess comm where latency is zero instead of planning for 10k+ km WAN latencies. I'll post more in the other thread on this, but I wanted to make sure everyone who uses replication knows to set it up w/o synchronizing any views or subsets. It's a bit of a pain, but I'm writing a process to create the views and subsets on each planet. Since we replicate every two hours, the time savings is significant.

Matt

Steve Vincent · Post by **Steve Vincent** » Mon Mar 09, 2009 3:51 pm

TJMurphy wrote: We're only replicating cubes on our system (which picks up the dimensions). I've never tried to replicate just a dimension - how did you set it up to do that?

Sorry for the delay Tony, it was something i tried ages ago and not sure how i did now, but a project i'm about to start might (or not!) benefit from replicated dims so i've been reading up on it again. Its possible i set a replication up to copy a "dummy" cube. eg. i created a cube consisting of just the dims i wanted to replicate but with no data in it, then used that cube to do the replication. Sounds like it should have worked but i can't test it just now.

Anyway, something i just noticed in the admin guide has me worried. I'm hoping i'm reading it wrong but i fear i'm not;

Admin Server Considerations
when you establish a replication connection, both the source server and the target server must be registered on the same admin server. if the 2 servers do not share the same admin server, the replication and synchronization results are unpredictable

So if a server in London, UK is replicating with another one in Canberra, Aus, are they seriously suggesting that they both have to be using the same admin server for it to work? I'm sorry but a server having to use the admin host 10,000+ miles away is ludicrous. If I have read that right then Iâ€™m not surprised a lot of people have had dodgy results with itâ€¦

Steve Rowe · Post by **Steve Rowe** » Mon Mar 09, 2009 6:28 pm

Anyway, something i just noticed in the admin guide has me worried. I'm hoping i'm reading it wrong but i fear i'm not;

Admin Server Considerations
when you establish a replication connection, both the source server and the target server must be registered on the same admin server. if the 2 servers do not share the same admin server, the replication and synchronization results are unpredictable

So if a server in London, UK is replicating with another one in Canberra, Aus, are they seriously suggesting that they both have to be using the same admin server for it to work? I'm sorry but a server having to use the admin host 10,000+ miles away is ludicrous. If I have read that right then Iâ€™m not surprised a lot of people have had dodgy results with itâ€¦
TJMurphy wrote: We're only replicating cubes on our system (which picks up the dimensions). I've never tried to replicate just a dimension - how did you set it up to do that?
Sorry for the delay Tony, it was something i tried ages ago and not sure how i did now, but a project i'm about to start might (or not!) benefit from replicated dims so i've been reading up on it again. Its possible i set a replication up to copy a "dummy" cube. eg. i created a cube consisting of just the dims i wanted to replicate but with no data in it, then used that cube to do the replication. Sounds like it should have worked but i can't test it just now.

Anyway, something i just noticed in the admin guide has me worried. I'm hoping i'm reading it wrong but i fear i'm not;

Admin Server Considerations
when you establish a replication connection, both the source server and the target server must be registered on the same admin server. if the 2 servers do not share the same admin server, the replication and synchronization results are unpredictable
So if a server in London, UK is replicating with another one in Canberra, Aus, are they seriously suggesting that they both have to be using the same admin server for it to work? I'm sorry but a server having to use the admin host 10,000+ miles away is ludicrous. If I have read that right then Iâ€™m not surprised a lot of people have had dodgy results with itâ€¦

Not something I have done for a while it's true, but you can have the server register itself on multiple adminhosts. The local one the clients point at and the "central" one for replication.

David Usherwood · Post by **David Usherwood** » Mon Mar 09, 2009 8:55 pm

This thread has reminded me of something I planned to look at some time ago, when we tried replication and (as Scott commented) found it only of value in very specific, limited circumstances. There's a tool called Rsync, which was written by Andrew Tridgell, the co creator of Samba, as his PhD project. It does two way (or one way if you wish) synchronisation of any data, sending only the changes back and forth - so very efficient. Clearly without the extensive integrity checking of our favourite OLAP server, you'd have to take some care, but with that care, I think it would work, especially since TM1 doesn't lock the server directory. (You'd need a restart after a sync, of course.) Next time I have some time I'll try it out. Don't know when that's going to be though.

mattgoff · Post by **mattgoff** » Tue Mar 10, 2009 9:48 am

Steve Rowe wrote:
Anyway, something i just noticed in the admin guide has me worried. I'm hoping i'm reading it wrong but i fear i'm not;

Admin Server Considerations
when you establish a replication connection, both the source server and the target server must be registered on the same admin server. if the 2 servers do not share the same admin server, the replication and synchronization results are unpredictable
So if a server in London, UK is replicating with another one in Canberra, Aus, are they seriously suggesting that they both have to be using the same admin server for it to work? I'm sorry but a server having to use the admin host 10,000+ miles away is ludicrous. If I have read that right then Iâ€™m not surprised a lot of people have had dodgy results with itâ€¦
Not something I have done for a while it's true, but you can have the server register itself on multiple adminhosts. The local one the clients point at and the "central" one for replication.

Yes, this is how we do it. Each server runs a local admin which is the first admin server entry in the cfg file. The second entry is the admin server on our star. This allows replication plus all users only see their local server in Perspectives.

As I work with replication more, I'm getting the impression that when set up EXACTLY RIGHT, it's reasonably stable. By set up, I mean establishing the replications and replication account permissions, both of which require a lot of experience to get right. This is quite a feat, but really not any more complex than getting your feeders right (!). The added difficulty is that TM1's WAN communication is slow and you're bouncing between multiple servers during the process. TM1's (lack of good) documentation doesn't help either.

Matt

Jessy · Post by **Jessy** » Wed Aug 05, 2009 9:05 am

Hi,all

I'm coming across with similar problem

Mike Cowie wrote: One thing to watch out for with replication (which I, frankly, haven't dealt with directly in a long time) is that if you set Rules to synchronize along with a cube, it used to be the case that Synchronization would cause your Rules to be re-saved whenever a Dimension in the cube that has the rules is synchronized/updated. This might not seem like too much of a big deal, unless your rules take quite a long time to save. Quite a headache if you aren't looking out for it.

even if there is no rule, when I modify a dimension of the cube, let's say I insert some elements into it(in the target server), this update will not be reflectd in the source server, instead dimension in the target server will be covered with that in the source server.

Jessy · Post by **Jessy** » Wed Aug 05, 2009 9:16 am

Marcus Scherer wrote:Hi,
very interesting answers from you Gurus. One more question to the subject: in older versions the replication of attributes was not supported. Does someone know if it is supported with V. 9?

TIA,

Marcus

Hi, I've tried in 9.4.1, attributes are not copied.

And I haven't seen it mensioned in the operation guide.

Let me know if it is supported. Thanks in advance

Steve Vincent · Post by **Steve Vincent** » Wed Aug 05, 2009 2:44 pm

It *is* in the admin guide, its just not that obvious to someone new to TM1

There is a small section that mentions replicating "control" cubes. To copy the attributes you need to seperatly replicate the control cube for them as well as the dimension. So if you had a dim call "Dept" you'll need to replicate the cube "}ElementAttributes_Dept" as well.

TM1 Forum

Replication - worth it ? or not ?

Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?

Re: Replication - worth it ? or not ?