Server Data folder backup

[Analytics123]

Hi All,

I am using the below code to create a data folder back up . My data folder size is 25 gb.
SaveDataAll();
sExec = 'C:\Program Files\7-Zip\7z.exe';
sBackupDir = 'D:\Backups\Data\Backups\';
sDataDir = 'D:\Data\';

sFileName = 'Data ' | TIMST(NOW(), '\Y\m\d \h.\i', 1) | '.zip';
sArgs = ' a -tzip ';
sCommand = sExec | sArgs | '"' | sBackupDir | sFileName | '" "' | sDataDir | '*"';
EXECUTECOMMAND(sCommand,0);

The issue I am having here is , the copy process takes a total of 45 mins , but if i do it manually it takes only 2 to 3 minutes.

Is the issue with my zip file .(7 zip is installed in server)
I also tried robo copy it completes the copy in 7 minutes.
Any suggestions is appreciated.

Thanks,

[gtonkin]

I use a similar process using 7zip to do backups and have not seen this huge variance between Process and Command Line.
I do however never back up the live folder. I always copy the live folder to another and then zip from there. Maybe worthwhile trying this approach.

[lotsaram]

That backup script looks "familiar".

[Wim Gielis]

Hello,

Do you compare the timings with the same SaveDataAll ?
Or is the SaveDataAll excluded from the timings ?

[Analytics123]

No the file copying starts immediately the process runs . I am able to see file size being increased . I dont think so savedata all is causing an issue .

However i can try doing the same bu directly not copying live folder , but taking a back up and then see if its the same .

Thanks,

[Wim Gielis]

I do however never back up the live folder. I always copy the live folder to another and then zip from there. Maybe worthwhile trying this approach.

Hello,

Do you mean that copying the data directory and then 7zip using executecommand on that folder takes less time than not copying and 7zip on the data directory ? Or what is the advantage of copying first ?

[mattgoff]

I do however never back up the live folder. I always copy the live folder to another and then zip from there. Maybe worthwhile trying this approach.

Do you mean that copying the data directory and then 7zip using executecommand on that folder takes less time than not copying and 7zip on the data directory ? Or what is the advantage of copying first ?

I also copy it first because I want to delete the feeder files and I haven't found a reliable way to exclude them from the zip without deleting them from the source folder first. It also reduces the window where the files in the dir could be mismatched due to a SaveDataAll mid-execution-- the copy process is much faster than the zip.

Matt

[kangkc]

In our case I separate the Save data and Zip in separate process with a window of 30 mins in between.
Possible that the Save Data actually takes a long time ? Quite common though.

[gtonkin]

I do however never back up the live folder. I always copy the live folder to another and then zip from there. Maybe worthwhile trying this approach.

Hello,
Do you mean that copying the data directory and then 7zip using executecommand on that folder takes less time than not copying and 7zip on the data directory ? Or what is the advantage of copying first ?

I copy the files for a few reasons:
1) I can exclude files I do not want backed up e.g. feeders
2) I can never guarantee that users will not be accessing and making changes (updates, new, deletions) to items written to the folders e.g. vues, subs, blbs etc. I want to avoid any possible issues where a file that was on the filesystem has gone by the time the zip gets to it. I also want to avoid any possible sharing violations/file-open issues-maybe I am just a relic from pre-shadow volumes days but feels better to do it this way.
3) My backup scripts on newer, smaller, more volatile systems create and age a rolling 31 days set of folders e.g. T-00, T-01, T-30 - these are copies of the data. The process ages the folders, creates a T-00 for the current day, copies in the data set with aside from any exclusions then zips, copies zip to a folder on the DR server etc. etc.

In terms of speed, copying first *may* be faster due to exlcusion of feeders which can be big. Assuming there is no contention whilst zipping, the straight zip *should* be faster. The OP's issue could still however be down to not comparing apples with apples i.e. timing the whole TI including the SaveDataAll and zipping vs just zipping.

[mattgoff]

In our case I separate the Save data and Zip in separate process with a window of 30 mins in between.
Possible that the Save Data actually takes a long time ? Quite common though.

I assume this means you are using a scheduled task in the OS to do the zip. Have you considered using ExecuteCommand in TM1 instead? This would guarantee that the SaveDataAll is complete prior to zip.

Matt