Server Data folder backup

Post Reply
Analytics123
Posts: 128
Joined: Tue May 23, 2017 12:45 pm
OLAP Product: Tm1
Version: 9.5
Excel Version: 2010

Server Data folder backup

Post by Analytics123 »

Hi All,

I am using the below code to create a data folder back up . My data folder size is 25 gb.
SaveDataAll();
sExec = 'C:\Program Files\7-Zip\7z.exe';
sBackupDir = 'D:\Backups\Data\Backups\';
sDataDir = 'D:\Data\';

sFileName = 'Data ' | TIMST(NOW(), '\Y\m\d \h.\i', 1) | '.zip';
sArgs = ' a -tzip ';
sCommand = sExec | sArgs | '"' | sBackupDir | sFileName | '" "' | sDataDir | '*"';
EXECUTECOMMAND(sCommand,0);

The issue I am having here is , the copy process takes a total of 45 mins , but if i do it manually it takes only 2 to 3 minutes.

Is the issue with my zip file .(7 zip is installed in server)
I also tried robo copy it completes the copy in 7 minutes.
Any suggestions is appreciated.

Thanks,
User avatar
gtonkin
MVP
Posts: 1192
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: Latest and greatest
Excel Version: Office 365 64-bit
Location: JHB, South Africa
Contact:

Re: Server Data folder backup

Post by gtonkin »

I use a similar process using 7zip to do backups and have not seen this huge variance between Process and Command Line.
I do however never back up the live folder. I always copy the live folder to another and then zip from there. Maybe worthwhile trying this approach.
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Server Data folder backup

Post by lotsaram »

That backup script looks "familiar".
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
Wim Gielis
MVP
Posts: 3105
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1, Jedox
Version: PAL 2.0.9.18
Excel Version: Microsoft 365
Location: Brussels, Belgium
Contact:

Re: Server Data folder backup

Post by Wim Gielis »

Hello,

Do you compare the timings with the same SaveDataAll ?
Or is the SaveDataAll excluded from the timings ?
Best regards,

Wim Gielis

IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
Analytics123
Posts: 128
Joined: Tue May 23, 2017 12:45 pm
OLAP Product: Tm1
Version: 9.5
Excel Version: 2010

Re: Server Data folder backup

Post by Analytics123 »

No the file copying starts immediately the process runs . I am able to see file size being increased . I dont think so savedata all is causing an issue .

However i can try doing the same bu directly not copying live folder , but taking a back up and then see if its the same .

Thanks,
Wim Gielis
MVP
Posts: 3105
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1, Jedox
Version: PAL 2.0.9.18
Excel Version: Microsoft 365
Location: Brussels, Belgium
Contact:

Re: Server Data folder backup

Post by Wim Gielis »

gtonkin wrote:I do however never back up the live folder. I always copy the live folder to another and then zip from there. Maybe worthwhile trying this approach.
Hello,

Do you mean that copying the data directory and then 7zip using executecommand on that folder takes less time than not copying and 7zip on the data directory ? Or what is the advantage of copying first ?
Best regards,

Wim Gielis

IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
User avatar
mattgoff
MVP
Posts: 516
Joined: Fri May 16, 2008 1:37 pm
OLAP Product: TM1
Version: 10.2.2.6
Excel Version: O365
Location: Florida, USA

Re: Server Data folder backup

Post by mattgoff »

Wim Gielis wrote:
gtonkin wrote:I do however never back up the live folder. I always copy the live folder to another and then zip from there. Maybe worthwhile trying this approach.
Do you mean that copying the data directory and then 7zip using executecommand on that folder takes less time than not copying and 7zip on the data directory ? Or what is the advantage of copying first ?
I also copy it first because I want to delete the feeder files and I haven't found a reliable way to exclude them from the zip without deleting them from the source folder first. It also reduces the window where the files in the dir could be mismatched due to a SaveDataAll mid-execution-- the copy process is much faster than the zip.

Matt
Please read and follow the Request for Assistance Guidelines. It helps us answer your question and saves everyone a lot of time.
kangkc
Community Contributor
Posts: 206
Joined: Fri Oct 17, 2008 2:40 am
OLAP Product: TM1, PA , TMVGate
Version: 2.x
Excel Version: 36x
Location: Singapore
Contact:

Re: Server Data folder backup

Post by kangkc »

In our case I separate the Save data and Zip in separate process with a window of 30 mins in between.
Possible that the Save Data actually takes a long time ? Quite common though.
User avatar
gtonkin
MVP
Posts: 1192
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: Latest and greatest
Excel Version: Office 365 64-bit
Location: JHB, South Africa
Contact:

Re: Server Data folder backup

Post by gtonkin »

Wim Gielis wrote:
gtonkin wrote:I do however never back up the live folder. I always copy the live folder to another and then zip from there. Maybe worthwhile trying this approach.
Hello,
Do you mean that copying the data directory and then 7zip using executecommand on that folder takes less time than not copying and 7zip on the data directory ? Or what is the advantage of copying first ?
I copy the files for a few reasons:
1) I can exclude files I do not want backed up e.g. feeders
2) I can never guarantee that users will not be accessing and making changes (updates, new, deletions) to items written to the folders e.g. vues, subs, blbs etc. I want to avoid any possible issues where a file that was on the filesystem has gone by the time the zip gets to it. I also want to avoid any possible sharing violations/file-open issues-maybe I am just a relic from pre-shadow volumes days but feels better to do it this way.
3) My backup scripts on newer, smaller, more volatile systems create and age a rolling 31 days set of folders e.g. T-00, T-01, T-30 - these are copies of the data. The process ages the folders, creates a T-00 for the current day, copies in the data set with aside from any exclusions then zips, copies zip to a folder on the DR server etc. etc.

In terms of speed, copying first *may* be faster due to exlcusion of feeders which can be big. Assuming there is no contention whilst zipping, the straight zip *should* be faster. The OP's issue could still however be down to not comparing apples with apples i.e. timing the whole TI including the SaveDataAll and zipping vs just zipping.
User avatar
mattgoff
MVP
Posts: 516
Joined: Fri May 16, 2008 1:37 pm
OLAP Product: TM1
Version: 10.2.2.6
Excel Version: O365
Location: Florida, USA

Re: Server Data folder backup

Post by mattgoff »

kangkc wrote:In our case I separate the Save data and Zip in separate process with a window of 30 mins in between.
Possible that the Save Data actually takes a long time ? Quite common though.
I assume this means you are using a scheduled task in the OS to do the zip. Have you considered using ExecuteCommand in TM1 instead? This would guarantee that the SaveDataAll is complete prior to zip.

Matt
Please read and follow the Request for Assistance Guidelines. It helps us answer your question and saves everyone a lot of time.
Post Reply