How do YOU keep track of changes?

[Glass]

Hi all,

Excel 10, TM1 10.2.2.

I have dev and production environments (sadly no test). I refresh the dev environment with a copy of prod whenever a new requirement comes up. But I'm getting increasingly bitten by how to track changes made in Dev that need to be migrated over to prod.

I try to keep detailed notes of the dimensions/cubes/TIs etc that I've changed to make sure they get transferred when updating prod, but it only takes one distraction to miss noting down a critical item.

Adding to the complexity is that I can't stop and start the production server - my preferred approach (which I did at my previous employer) would be to prepare a migration package, shut down the server (say half and hour), copy the items into the DB directory and restart. Unfortunately IT are very protective of the prod environment, so I need to do my changes in the live system via Perspectives/Architect. Which is very susceptible to human error, and generally just a terrible way to do things. But it's the constraint I need to work with.

I know there are products that can assist with environment comparisons (Pulse springs to mind), however I'm not in a pay grade to buy it, and those that are aren't willing to.

My taking notes approach has bitten me on a few occasions - I've added a measure in Dev but didn't write it down, so Prod isn't happy. To achieve a result I needed to add a new attribute to a dimension, but didn't write it down, so Prod throws a hissy fit.

In the absence of software that will identify differences, does anyone have any tips on making sure migrations go smoothly?

[jim wood]

Moved to the correct forum.

[qml]

My first tip would be to move to a code-driven paradigm. Make changes only to code (TI and rules plus whatever else you're using - Java, shell scripting etc) and never directly to objects. Cubes, dimensions, views, subsets etc should only be created/manipulated by said code, never by hand, and should not be copied between environments. TI is versatile enough to allow you to do that almost without exceptions. When you've moved to that method of managing your models, you can easily keep track of changes done to your code, either by hand or using any (often free and open-source) code versioning software available (Git, Mercurial, Subversion...).

[Edward Stuart]

I mainly use a combination of Winmerge and Agent Ransack/ FileLocator to keep track of changes between environments. Winmerge is a simple folder/ file comparison tool which can tell you if files match or are different/ missing.

However, after some good discussions in forum posts in recent months I've moved more to the model QML describes where all changes being made through TI, this greatly improves the process when migrating from DEV > UAT > PREPROD > PROD and in many cases speeds up development as recreating a cube is quick and simple and it forces you to operate in a standard way and reduces the risk of human error creeping in.

I've not gone down the road of code versioning software as yet but this is an area I'm looking to explore

[tomok]

Unfortunately IT are very protective of the prod environment, so I need to do my changes in the live system via Perspectives/Architect. Which is very susceptible to human error, and generally just a terrible way to do things. But it's the constraint I need to work with.

Something tells me that your IT folks don't know what you are doing here because your assertion makes no sense. They are so protective that they won't allow the service to be taken down for code migrations, yet they'll let you make live code changes to a production environment with no way to back out said changes in the event of a failure, short of a restore from tape backup?

The tried and true method is still to create a migration package, with instructions, including back out procedures, doing it in off hours with a service restart.

[PavoGa]

I would suggest using Beyond Compare 4 vs Winmerge. If you are comparing a source code file against a TI process, Beyond Compare does a better job of ignoring irrelevant lines.

[Glass]

Tomok>
Welcome to the Australian Government!

[Glass]

"The tried and true method is still to create a migration package, with instructions, including back out procedures, doing it in off hours with a service restart."

Which was my suggested approach. I was also keen to clone the Prod env into Test, and test the upgrade from Dev to Test to make sure everything worked. But my overlords demanded I went to directly to Prod instead. Which, well, sucks, but it's the hand I've been dealt so I need to play it accordingly.

Thanks to all for the feedback - I'm a big fan of WinMerge, which is great for one on one comparisons but isn't as good for bulk comparisons (unless there's functionality in it that I'm not aware of, in which case feel free to enlighten me!).

Thankyou for your responses,

Glass.

[blackhawk]

Actually, there is a utility out there for visually comparing objects between TM1 servers (see www.TM1Compare.com). It does all the visual deltas (including things like attributes, cube data, application objects, etc.) and lets you examine what changed and then copy them to the destination server. It even creates migration packages that can be executed independently across machines (i.e. DEV, PROD, QA etc). All of this can be done without taking the server down. So no longer does this need to be the case:

"The tried and true method is still to create a migration package, with instructions, including back out procedures, doing it in off hours with a service restart."

It can be done a lot faster than having to go through this every time and possibly missing something along the way. How many times have we forgotten about an attribute that a process/rule depends on, only to find out it wreaked havoc 3 days in the planning cycle? OUCH!

Take a look at www.TM1Compare.com and let me know if that is more in lines of what you are looking for. We use it daily and it works very well.


P.S. I work for the company who created it.

[George Regateiro]

My first tip would be to move to a code-driven paradigm. Make changes only to code (TI and rules plus whatever else you're using - Java, shell scripting etc) and never directly to objects. Cubes, dimensions, views, subsets etc should only be created/manipulated by said code, never by hand, and should not be copied between environments. TI is versatile enough to allow you to do that almost without exceptions. When you've moved to that method of managing your models, you can easily keep track of changes done to your code, either by hand or using any (often free and open-source) code versioning software available (Git, Mercurial, Subversion...).

There are tools that have been mentioned in this thread for live migrations, but QML's approach is the one I have gone with to handle objects with our multiple developers. Unless it would overly difficult to code ALL changes should be made using TI. Every change comes with a setup script TI that is run to set up any needed dim, cube, attribute , ect changes. This way there is no guess work as to what needed to be migrated. I trust the live migration tools with my TIs but I just have an aversion to copying dims and cubes between models.

I will also second the versioning software of some kind. A lot of them are free. For complete tracking we use the atlassian toolset to make it more user friendly. If you have a small team then the cost is very minimal but it offers some decent functionality that is missing in alot of small teams.

[blackhawk]

George,

The TI coding may work fine for incremental changes to the system. Few dimensions here...few elements there. But what about dimensional re-orgs? What about attribute values and cube values? Now you have to either write a potentially gargantuan TI script or you have to start creating data files that your TI script loads. This just adds to the complexity of your migration script. So now you have end up performing a QA on your migration script, which usually involves backward promoting PROD to QA and re-test your migration script. If it fails, you have to restore and try again after the fixes are made. That is a lot of work!

Also, with this method, time is your enemy because generally you create, or at least finalize, the TI scripts once the TM1 Model QA has been completed. The longer the time gap that the move to production occurs, the higher the risk of something changing in PROD that could affect your migration script, especially if you have different teams implementing at different times.

This is precisely what TM1Compare addresses. When you create the migration script, you have the opportunity to re-compare against PROD and confirm that something didn't change during that DEV-QA-PROD time gap. Being able to document what SHOULD change and confirm that those changes are the same when you actually DO the migration is equally important.

With many financial systems, we see quite often that executives are demanding rapid turnaround in implementing the business logic changes to "their" Balance Sheet/P&L/Product Allocation etc. for a board meeting next week and there is no time to have them wait for the build/unit test/back promote to QA/QA/migration build/migration test/promote/backout recovery/document/yada, yada yada. If we are talking a government system here...ok all bets are off, and I retract this posting.

[lotsaram]

My first tip would be to move to a code-driven paradigm. Make changes only to code (TI and rules plus whatever else you're using - Java, shell scripting etc) and never directly to objects. Cubes, dimensions, views, subsets etc should only be created/manipulated by said code, never by hand, and should not be copied between environments. TI is versatile enough to allow you to do that almost without exceptions. When you've moved to that method of managing your models, you can easily keep track of changes done to your code, either by hand or using any (often free and open-source) code versioning software available (Git, Mercurial, Subversion...).

There are tools that have been mentioned in this thread for live migrations, but QML's approach is the one I have gone with to handle objects with our multiple developers. Unless it would overly difficult to code ALL changes should be made using TI. Every change comes with a setup script TI that is run to set up any needed dim, cube, attribute , ect changes. This way there is no guess work as to what needed to be migrated. I trust the live migration tools with my TIs but I just have an aversion to copying dims and cubes between models.

I also work for a company who have a tool for migration between servers which I won't name here but I'm sure it refered to enough elsewhere and I used to be a fan of the changes only via TI approach before I had a tool at my disposal to manage migration inclusive of diff tests & promoting incremental dimension structure changes etc. I get where Kamil & George are coming from really I do but I firmly believe that this approach stems from a lack of proper tool and TI being the default lowest risk fall back. If you have a proper tool to help with migrations it is a helluva lot easier (and a much nicer user experience) to use the tool rather than to code (and unit test) every single ity bity change. Been there, done both and I would rather have a tool for the job..

[paulsimon]

Hi

I am surprised that your IT Dept has a no restart policy on Services. How to they manage Windows Updates which often require a box restart? I believe that the advice from IBM is still to regularly restart the Service to avoid WIndows Memory leaks building up.

There are various tools that will allow code promotion into a live server. However, I would question the desirability of this, unless they also include a way to block user access during the live promotion. There is too much risk of eg a Websheet triggering a process with new parameters when the process promotion has not yet happened so it cannot accept those new parameters, etc.

We do all our non-emergency, ie proper release, promotions to Prod during an overnight service restart.

In part this is because TM1 is not an Island and some changes need to be coordinated with changes to other systems, changes to SQL databases, BI Reports, VB.Net, JavaScript, etc.

While comparison tools are a good way to identify changed items that need to be released, they are not the whole answer.

For example, they may identify a changed cube but unless the cube is brand new you should never promote a .cub file as you will overwrite production data with development data. The same applies to .dim files. Promoting a .dim file that does not include all the elements in Prod will delete data held against those elements.

Even if you adopt the purest approach of only promoting TI Processes and Rules, then there are still issues.

Your promotion should be subject to some form of Change Control procedure. In our environment, all changes must have a Change Request/Ticket. Each Change must be approved by users and the head of our department.

Quite often, coming up to a release cut off date, we have to pull some changes because users have other priorities and have not had the time anticipated to UAT the change.

Therefore although a comparison tool might identify that there is a difference in code between Dev and Prod that doesn't necessarily mean that this change should be promoted.

We have a release manager who has to ensure that he/she knows how the changes inter-relate, as pulling one change might also mean pulling another that relies on the former change.

I think the point that I am making is that while automated tools can help, they are only part of the picture, and you really do need to know about the items that you are promoting.

For major releases, we always copy Prod back to Pre-Prod a few days before the release, and then release to Pre-Prod to test that the release does not fail. We then perform a check out to ensure that everything that is being released works as planned. That gives a good degree of confidence that it will work when it does go to Prod.

We still have emergency release procedures where the copy back is not done but they are the exception rather than the norm.

I would certainly agree that promoting code by repeating the changes in Production is prone to error. There can be no guarantee that what a user signed off during UAT is what will be promoted. It also suffers from the timing issue for live promotion mentioned above, but this is likely to be exacerbated as manual live promotion will be slower than automated live promotion and there is more chance of timing issues with object A being updated before object B.

In most IT Depts the ideal would be that the developers no longer need access to edit code directly in Prod, and that everything goes up via a promotion mechanism. This is not always easy to achieve, it is what most would consider good practice.

Regards

Paul Simon

[pandinus]

Any fans of the Performance Modeler transfer in/transfer out functions? Do you have experiences to share?

From my experience they work well on very simple changes, but often crash or give non helpful errors when things get too complex, especially when TM1Applications is involved. This is why we refrain from using this in our production environments.

[George Regateiro]

I also work for a company who have a tool for migration between servers which I won't name here but I'm sure it refered to enough elsewhere and I used to be a fan of the changes only via TI approach before I had a tool at my disposal to manage migration inclusive of diff tests & promoting incremental dimension structure changes etc. I get where Kamil & George are coming from really I do but I firmly believe that this approach stems from a lack of proper tool and TI being the default lowest risk fall back. If you have a proper tool to help with migrations it is a helluva lot easier (and a much nicer user experience) to use the tool rather than to code (and unit test) every single ity bity change. Been there, done both and I would rather have a tool for the job..

The main thing that prevents me from fully utilizing automated tools is we have a certain set of dims that are generally involved in a lot of our changes, so 1 dimension will have multiple dev changes within it. I generally only want to move 1 specific change set to the next environment and leave the others in place. Same goes with rules, we have one reporting cube that almost every change will affect. Right now I have 10 separate dev changes to the same rule file pending in UAT and they will not go to prod in the order of commit. That said for new stand alone development or things that I know for certain will not impact other changes I utilize the automated tools to migrate cubes dims and rules.

Though I do not deny that there is a little bit of sticking with what has worked in the past, since I more then happily use newer toolsets to migrate our .net code around.

Any fans of the Performance Modeler transfer in/transfer out functions? Do you have experiences to share?

From my experience they work well on very simple changes, but often crash or give non helpful errors when things get too complex, especially when TM1Applications is involved. This is why we refrain from using this in our production environments.

I got bit by the we will insert feed strings everywhere in your model feature. They have given a work around to it but that was enough to make me never open it again.

[pundlesnorf]

@PaulSimon
I don't disagree with any of your points, but I'm an external resource with limited sway with the IT folk. I have suggested that they should reboot the server at least once a week (Sunday night being the preference, so they have a nice clean environment when they log on on Monday), but it's fallen on deaf ears.

[Steve Rowe]

Just to add this here

http://www.tm1forum.com/viewtopic.php?f=18&t=12675

as there is a plenty of chat in this thread about programmatic development.