Local Backup

[Peter Picazio]

I would like to create and schedule a nightly backup of my production TM1 model. Currently there is a nightly backup which is saved on another server. What I want to do is to backup the model locally in a folder on the same server. The backups would be stored in a folder called "Backup" and have date/time stamps in the name. The name of my production model is PF_Data_Prod which is located in the Data folder. The backup would be called PF_Data_Prod_Backup_date/time and stored in a folder called Backup in the Data folder on the same server.

Looking to see how this can be done (script?) and scheduled via a Chore.

Thank you.

[Wim Gielis]

You could have a look at the ExecuteCommand function in a TI process, which then calls 7-zip.
If you want to copy all files as is, without zipping, then have a look at Bedrock: https://github.com/cubewise-code/bedroc ... aster/main
There is a process to backup the data directory.
That process can also be used as a starting point for a 7-zip routine if you don't want to reinvent wheels.

[Emixam]

You could have a look at the ExecuteCommand function in a TI process, which then calls 7-zip.

Here's what i'm using.. It works perfectly.

Feel free to change the code if you want a different name for the backup !

[kangkc]

Insetad of 7-zip, I used PowerShell System.IO.Compression.FileSystem to perform the zipping.
Alternatively you can use Powershell Compress-Archive cmdlet. All thee are available on Windows Server as long as PowerShell is not disabled.
No installation of 3rd party software.

[David Usherwood]

Alternatively you can use Powershell Compress-Archive cmdlet. All thee are available on Windows Server as long as PowerShell is not disabled.

I recently tested this out on the IBM SaaS. Powershell is enabled (bear in mind that the remote session you log into is not where TM1/PA itself runs) and it worked fine.
This arose as a result of an unsubstantiated rumour that an IBM client lost all their SaaS content because the partner didn't take action after receiving numerous renewal chases.
Stay Safe. (Doubly relevant at the moment )

[tomok]

Alternatively you can use Powershell Compress-Archive cmdlet. All thee are available on Windows Server as long as PowerShell is not disabled.

I recently tested this out on the IBM SaaS. Powershell is enabled (bear in mind that the remote session you log into is not where TM1/PA itself runs) and it worked fine.
This arose as a result of an unsubstantiated rumour that an IBM client lost all their SaaS content because the partner didn't take action after receiving numerous renewal chases.
Stay Safe. (Doubly relevant at the moment )

Yes, indeed Powershell works in the IBM Cloud. We have a PS email script that did not initially work when we moved to the Cloud but that's because we could not talk to our company mail server. I had to create an account at Sendgrid in order to send emails but it has worked perfectly since then.

[kangkc]

Side track on email. Cloud PA do provide SMTP for sending email.
See below:
https://www.ibm.com/support/pages/plann ... ell-script

[lotsaram]

Side track on email. Cloud PA do provide SMTP for sending email.
See below:
https://www.ibm.com/support/pages/plann ... ell-script

+1 Actually quite a good doc. Thumbs up!

[tomok]

Side track on email. Cloud PA do provide SMTP for sending email.
See below:
https://www.ibm.com/support/pages/plann ... ell-script

This option wasn't there when we migrated to the cloud last Fall. Maybe my grumbling about it to our IBM Cloud rep had some impact......Doubt it!!! Good news anyway.

[Alan Kirk]

Insetad of 7-zip, I used PowerShell System.IO.Compression.FileSystem to perform the zipping.
Alternatively you can use Powershell Compress-Archive cmdlet. All thee are available on Windows Server as long as PowerShell is not disabled.
No installation of 3rd party software.

True, but there is a downside. The Windows libraries for file compression can be charitably described as.... "working". Also the .zip format could be charitably described as "venerable" and less charitably as "antiquated". An example: 500 data feed files from my GL system over the course of a couple of months. Total size? 1.42 Gig.

Zipped using Windows libraries? Just under 271 meg.

Zipped into a .zip archive with the 7 Zip program? Just under 235 meg, so, a pretty substantial saving.

Zipped into a .7z archive with 7 zip? 553 kB.

WHAAAAT???

Yes, 553 kilobytes.

Now, this is potentially an exceptional example because the feeds concerned are a "zero out and reload" type which means that there is a lot of duplicated data within the files; most of the lines of file 2 will be the same as file 1, most of the lines of file 3 will be the same as file 2, etc, some files may be exactly the same as will be the case with many GL feeds. Clearly the 7z archive takes that into account and doesn't compress each file individually but rather looks at them as a batch. Unfortunately that means that it's pretty slow. Compacting to a .zip format may take 30 seconds, compacting to .7z could take quite a few minutes. But in an overnight process, who cares?

It's true that I don't see the same difference in, say compacting video recordings, but the difference between .zip and .7z is is still pretty significant. Example:
Original folder size, 74.2GB
.zip folder created by Windows, 69.0GB
.zip folder size created by 7Zip: 68.7GB, so, not such a great saving over Windows this time BUT
.7Z folder size: 49.7GB

That stacks up... rapidly.

The other negative is that courtesy of Windows now everyone can access a .zip format, while you would need the third party software to access .7zs... but we're talking about archives that aren't meant for the general public anyway.

[David Usherwood]

Indeed. But on IBM SaaS, you have to raise a ticket to get utilities like 7Zip installed. I don't know if anybody's tried, but it is good to know that you can do backups without anything extra.

[Emixam]

Indeed. But on IBM SaaS, you have to raise a ticket to get utilities like 7Zip installed. I don't know if anybody's tried, but it is good to know that you can do backups without anything extra.

Hello David. I've been able to download 7zip on the server that has Architect and Perspectives on it without raising a ticket. I'm using the TI attached on my previous post and it works perfectly on IBM Cloud.

[Alan Kirk]

Indeed. But on IBM SaaS, you have to raise a ticket to get utilities like 7Zip installed.

It's good to hear from Emixam in the following post that it's not necessary for 7zip at least. Because if you did, I think we know how that would turn out:

SAAS_SupportTicket.jpg (24.95 KiB) Viewed 12870 times

This is Reason Number 427 in my personal list of "why I consider enterprise level SAAS to be, in the immortal words of Ser Bron of the Blackwater, 'a s**t idea'." (That is, from the customers' perspective, in the long term, not the suppliers' perspective in either the long or short term. For the suppliers it's a never ending strip-mining session of the customers' bank accounts. The cash flow that never ends.)

I don't know if anybody's tried, but it is good to know that you can do backups without anything extra.

True, but I don't know how IBM charges for disk space or, more likely, how much they give you with a service contract. That, however, seems to be the topic where cloud providers in general, having lured you in with relatively cheap VM rates, make you pay through the nose for something that you obviously can't do without but may not think enough about at the outset. Certainly I know from my own experience that AWS servers aren't painfully expensive. But the storage volumes that you need to attach to them generate a reaction of "You want how much for how little???" And we're talking about standard storage, not platinum plated, diamond encrusted, optically connected with the signal passing through a wormhole to make it arrive faster SSDs. They're just charged as if they were those products. This has prompted the optimisation of disk space usage to come to the front of my mind over the last few months.

[ykud]

Indeed. But on IBM SaaS, you have to raise a ticket to get utilities like 7Zip installed.

AFAIK, you don't have to 'install' 7-zip, it's available as standalone executable, so you just need to upload it to the tm1 server (any of the shared folders you have access to) and it should work.
And 7z is a much better compression format than zip.

Y

[Alan Kirk]

Indeed. But on IBM SaaS, you have to raise a ticket to get utilities like 7Zip installed.

AFAIK, you don't have to 'install' 7-zip, it's available as standalone executable, so you just need to upload it to the tm1 server (any of the shared folders you have access to) and it should work.

Unless you want File Manager integration, which I find to be useful. However if the only thing you're doing with it is scripting, then agreed.

And 7z is a much better compression format than zip.

As I did indeed demonstrate a couple of posts back, with bells on.

[David Usherwood]

upload it to the tm1 server

and

I've been able to download 7zip on the server that has Architect and Perspectives

Without being familiar with the IBM SaaS Ts & Cs, I think it likely that dropping random EXEs into the instance would not comply with them. (I used to be a systems auditor before I found the True Way of the Cube )
No idea how IBM charges for storage, or even if. Anybody know?

[Alan Kirk]

No idea how IBM charges for storage, or even if. Anybody know?

At Enterprise level I think it'll be "how much can you afford"? It's "contact us for pricing", not unsurprisingly.

In Australia for PA On Demand, it's $AU55.66 / month for each user plus $AU6.96 / month for each 2GB (yes, GIGAbyte) container. There's no discount for annual rather than monthly payments and no bulk discount (though if you needed one I'd imagine that you'd go the Enterprise route), so for a year 2 Gig will set you back $83.52. That doesn't sound like much... but 2 Gig isn't much either. An average budgeting / forecasting / reporting model can fit within 2 Gig, more analytical stuff won't... and this is what I mean by it can get you because that 2 Gig would also have to accommodate data files which will increase in size - sometimes rapidly - over time.

Also you'd need to consider backups; if you're forced to use .zip rather than .7z (or even if you aren't) you ain't gonna be able to make many of those.

For those who do not use the AUD micro peso...
$AU41.76 per gigabyte per year = €25.50 = £22.71 = $USD28.38

Not bad headline rates... but they scale badly.

[declanr]

PA on demand is a different kettle of fish altogether; it is very much intended to handle individual smaller models - think of those large excel workbooks people might maintain on their own machines but having the ability to utilise PA functionality instead.
There is the "Digital Pack" which is sort of like the "Express" licencing option for the cloud (between 5 and 20 users give or take) and then PA on demand is the newer sibling designed to open Planning Analytics up to a much wider audience.

In terms of the Enterprise solutions; last time I checked the default comes with 64gb of RAM and 200gb of disk space included in the user licence pricing.
Then with regards to the disk space query you can buy specific part numbers of additional 250gb increments for circa £500 gbp per annum (last time I checked) which doesn't seem too bad at all.
And of course you can also add additional RAM.

Normally I would recommend keeping a limited number of backups in the cloud anyway for quick reversions and any older backups that you wish to maintain can be easily FTP'd off the cloud to a location of your choosing.

[Alan Kirk]

In terms of the Enterprise solutions; last time I checked the default comes with 64gb of RAM and 200gb of disk space included in the user licence pricing.

200 gig does seem more reasonable than 2 for anything serious. It's still squeezy with big data, but it's viable.

Then with regards to the disk space query you can buy specific part numbers of additional 250gb increments for circa £500 gbp per annum (last time I checked) which doesn't seem too bad at all.

{Executes choking on cornflakes manoeuvre...} For certain definitions of "not too bad". Maybe it's just the woeful exchange rate for the AUD, maybe it's just the general costi-ness of the UK that I sensed in almost every transaction last September, but...

A general purpose Amazon EBS volume in Sydney will cost $0.12/GB/month, charged by the second. For 250 gig, that's $AU30 Micro Aussie Pesos per lunar cycle, or $AU 360 per planetary orbit. That's about 196 quid per year in fish and chips down at the pub currency. Again it's less the absolute amount than how that scales when you start multiplying it.

And of course you can also add additional RAM.

{Glances at Performance Monitor, reads 58.1 gig and I'm not even doing Big Data stuff yet...}
I should bloody well hope so.

Normally I would recommend keeping a limited number of backups in the cloud anyway for quick reversions and any older backups that you wish to maintain can be easily FTP'd off the cloud to a location of your choosing.

Agreed, but you'd want to keep at least SOME on the server. Even with the Forum, we keep one backup on the server and one in another location. Redundancy is sometimes a goooood thing.

I have to admit, though... I think that On Demand is potentially a good thing. It will certainly make doing POCs a lot easier, and the price isn't excessive for that. It was in fact mentioned at one of the past Cubewise conferences but the prices cited then were way higher than what I'm seeing now, so maybe someone has clued in to an opportunity.

[Elessar]

There is an RFE to include 7zip, and it has "planned" status: https://ibm-data-and-ai.ideas.aha.io/ideas/PAOP-I-245
It's not clear that it's about cloud, but "as we use to do on-premise" implies