Is it possible to achieve real-time data backup in TM1?

[solverxyz]

Hi, all.

Our customers ask a special requirement:
How to achieve real-time data backup?

What they care most is data lost ( data in .cub ).

We already provide a 30 mins backup TI( SaveData, \Data\*.* -> .ZIP ), but it still cannot satisfy them.
Does anyone have solutions?


In my thought, it seems to against the main tech. used in TM1 - Memery Base MOLAP.
TM1 cannot save data all the time( or performance will be ulgy ?? @@ ), am I right??

[Alan Kirk]

Hi, all.

Our customers ask a special requirement:
How to achieve real-time data backup?

What they care most is data lost ( data in .cub ).

We already provide a 30 mins backup TI( SaveData, \Data\*.* -> .ZIP ), but it still cannot satisfy them.
Does anyone have solutions?

In my thought, it seems to against the main tech. used in TM1 - Memery Base MOLAP.
TM1 cannot save data all the time( or performance will be ulgy ?? @@ ), am I right??

Yeah, performance will be ugly.

TM1 does have real time backup already. Unless you intentionally turn off cube logging, which you would normally only do for bulk clear and reload operations from external systems which could be replicated in the event of a server crash anyway, it's called tm1s.log. With that, there is no need, at all, for the .cub file to be constantly hammered into the hard drive.

You have explained this to your client, right?

[solverxyz]

Hi, all.

Our customers ask a special requirement:
How to achieve real-time data backup?

What they care most is data lost ( data in .cub ).

We already provide a 30 mins backup TI( SaveData, \Data\*.* -> .ZIP ), but it still cannot satisfy them.
Does anyone have solutions?

In my thought, it seems to against the main tech. used in TM1 - Memery Base MOLAP.
TM1 cannot save data all the time( or performance will be ulgy ?? @@ ), am I right??

Yeah, performance will be ugly.

TM1 does have real time backup already. Unless you intentionally turn off cube logging, which you would normally only do for bulk clear and reload operations from external systems which could be replicated in the event of a server crash anyway, it's called tm1s.log. With that, there is no need, at all, for the .cub file to be constantly hammered into the hard drive.

You have explained this to your client, right?

Thanks for such a detail reply.


and Yes, they have ready known how tm1s.log works, but they still want to do "real time backup" such like tranditional DB does ( HA Cluster or something like that ).
Especially, tm1s.log is locked untill " SaveData" is executed.

I have 2 idea( maybe it's stupid...)

1.
I know log file could be listened via TM1API, but I listen all the time ( or active while tm1s.log status change ), how is the influence to the TM1 Performance ?
or Is there anyway to copy it or listen it from outside??

2.
If I set the path of LoggingDirectory in tm1s.cfg into a network shared folder which's phycial hard drive is use RAID with remote real-time backup techology.


Does anyone try it yet?? If it works, I could strat to survey what we need with this idea.


"IF the computer server with the VM witch installed with TM1 server just burned by a horriable fire, how could get my input data back 10 seconds before?", they said.
I know there are many technology for real time backup in DB or cloud storage, but they are all based one one thing: data are saved in HDD.
Maybe it hard to mirror data in memery only via pure software or it's too expensive , I am not sure.

[Alan Kirk]

"IF the computer server with the VM witch installed with TM1 server just burned by a horriable fire, how could get my input data back 10 seconds before?", they said.
I know there are many technology for real time backup in DB or cloud storage, but they are all based one one thing: data are saved in HDD.

Which, as we know, are impervious to fire. Especially the ones that write simultaneously to clusters on multiple cloud servers distributed across the globe without any performance impact on the application.

Maybe it hard to mirror data in memery only via pure software or it's too expensive , I am not sure.

No, the real problem is that the solutions are insufficient for the magnitude of the problem faced. They don't cover the very real possibility that each cloud server that you're backing up to could be simultaneously hit by meteors the size of Texas, which might then generate 70 metre high tidal waves that short out the memory chips of the client machine not to mention ruining the office carpet.

In such a case it would be impossible for them to recover their data as it was 10 seconds earlier.

I'd suggest implementing a further backup to a spacecraft in geostationary orbit, but that still won't overcome the possibility of a star going hypernova and generating a gamma ray burst that takes out the satellite and the Earth with it. And the soggy office carpet, come to that.

And man, do you know how much the data roaming charges are for geostationary satellites? Not to mention the property damage insurance premiums for hypernova GRBs?

I'm sorry, maybe someone else can take this ridiculous situation seriously, but I just can't. Your client needs to get their head out of their rectal passage and into the real world. Here's the fact, Jack. There is NO method, NO way, NO how that can give you an iron clad guarantee that you will not lose any data at all (certainly not down to a 10 second granularity) in the event of the physical destruction of your hardware. Even if this Heath Robinson construct of sucking out the tm1s.log file from the API actually worked, have you seen how slow querying the transaction log is? All you need to do is query it through the GUI, which would doubtless use the same API functions, or an equivalent of them. It would utterly slaughter the performance because the server would be too busy querying the transaction log to do any actual work.

Disaster recovery, just like any system design, is about balance. So you save regularly, backup regularly, use off site storage, if possible have multiple servers in different locations. But you cannot guarantee that you will not ever lose data, on TM1 or any other system, without making the system's whole raison d'être backing itself up constantly.

[mattgoff]

Trying to get the customer what they want is generally the best course, but in this case I really think you need to go over TM1's capabilities with him or her rather than continue trying to kludge a solution together. TM1 is not HA, pretty much end of story. It has solid failure recovery from system crashes, but your example of a loss-of-server is not a protected failure mode, and the customer shouldn't ever except TM1 to ever get there. That said, one option you could explore is 3rd party backup software that has the capability to do volume shadow copies to a remote server. This would eliminate the need for your current SaveDataAll->ZIP process.

BTW, as I'm sure you have noticed, SaveDataAll is a SLOW function, and as the model grows in size I think you're going to find that even a 30 minute period will cause unacceptable performance degradation for end-users.

Matt

[solverxyz]

Trying to get the customer what they want is generally the best course, but in this case I really think you need to go over TM1's capabilities with him or her rather than continue trying to kludge a solution together. TM1 is not HA, pretty much end of story. It has solid failure recovery from system crashes, but your example of a loss-of-server is not a protected failure mode, and the customer shouldn't ever except TM1 to ever get there. That said, one option you could explore is 3rd party backup software that has the capability to do volume shadow copies to a remote server. This would eliminate the need for your current SaveDataAll->ZIP process.

BTW, as I'm sure you have noticed, SaveDataAll is a SLOW function, and as the model grows in size I think you're going to find that even a 30 minute period will cause unacceptable performance degradation for end-users.

Matt

Thanks for the replies.

I cannot express how much thankful I am any more!

Updating the solution that out customer accepted eventually:

0. No need to shutdown the server anymore.
1. Every 6 hrs, backup the server ( .\Data folder ) with TI. => File_A in Network folder
2. Every 5 mins, backup Tm1s.log ( If its size or time changed ) with the software like "Cobian Backup" which support "shadow copy" that allow you can copy a file even while it's locked by another application. => File_B in Network folder
3. Any time, we can start the lastest server in another computer with File_A and File_B. This server is just like a mirror server.

Noticed:
In our case, there is no structure-changing in the production envirment, that means only the data in cube will change.( no new cube, dimension, element...,etc)
"1." was run by TM1 server, and "2." run by the other computer ( like a watch dog).
File_A and File_B have time stamp. All File_B will be deleted if a new File_A is created.

With this solution,
1. A 5-mins Backup is achieved.
2. Lower the infulence to the TM1 performance as possible.

Thanks again.

[paulsimon]

Hi

One other possibility is hardware level mirroring. Effectively the tm1s.log is written to two places at once over a SAN. As it works at the disk level, I think that this would probably get past the problem of the tm1s.log being locked until you do a SaveDataAll. However, it looks like your customer has agreed to a simpler solution.

Regards

Paul Simon