Automated TM1 Regression Testing Framework like nUnit?

[bobgill]

Hi folks,

Is there something for TM1 following the nUnit style of automated test tools?

In theory, I would expect it to be something like the following
* A TI that runs all processes with a '.Test' suffix
* Each process with a .Test suffix runs unit tests on an underlying TI returning PASS or FAIL. i.e. The process p.Dim.Subset.Copy.Test runs unit tests of p.Dim.Subset.Copy.
* Notification is sent whenever any processes return FAIL

I can create this on my own, but I thought I would ask if there exists a common framework that everyone already collaborates on. Is there a sourceforge project already started? Is anyone interested in contributing to one if I were to start it?

Thanks for your help,

Bob

[Darkhorse]

Hi bob

Neverheard of this process before but would this not work

While I <= dim size (}processes)
cElename = dimix(}processes, I)
If( scan("test", celename)<>o)
Executeprocess(celename);
Endif
I =I +1
End

You could add the code to the process if fail then add to dimension }processes an attribute which catches flags like fail

Then in epilogue create a subset based on that attribute for fails etc then ASCII output each fail or create an email to auto email yourself the answers

[bobgill]

You're completely right. That would work well.

I was just wondering if I'm reinventing the wheel or not. An Agile project should have a continuous integration component that automatically does regression testing and provides immediate visibility to the team if an error comes up. Is there not something already out there to provide this kind of testing framework?

In the TM1 world, is there something like sourceforge where developers can work together on foundational pieces to be shared in the community?

[lotsaram]

The closest or maybe the only thing resembling an open source library of unit functionality in TM1 is the bedrock codebase.

I'm suspicious as to the worth of a fully automated process for unit testing TI processes. Many processes will have parameters that need to be passed, without passing appropriate parameters a simple pass/fail based on process completion status doesn't give you much useful information, you could well argue it doesn't give you ANY useful information. For a valid test you need and expected outcome and compare the result to the expected outcome. Code that runs without producing an error is not enough.

[Harvey]

I'm pretty sure there is no bona-fide unit testing framework for TM1 on the market. It's something we've thought about frequently at Flow, but have yet to find the time to prioritize the product. We'd be interested to hear your thoughts and requirements.

[Darkhorse]

I have constant chores on my server that run through all models giving me automatic emails from processes ran nightly to server statistics cube sizes dimensions garbage memory to users dormant users admins etc

These things are easily built but I believe a integrated system wouldn't work as most servers configs are different, in my workplace the servers are all slightly unique and all models although similar in nature work differently to the developers requirements and expertise,

[paulsimon]

Hi

I use something that I call a Master Process cube. One dimension is the Master Process dimension. Against each Master Process element you can set up a number of lines of processes to be run, and give parameters.

The master process execution process takes a parameter of an element in the Master Process dimension to be run. It then runs each process in turn, passing in the parameters. You can set each process to continue on error or to stop running further processes if there is an error. For an error testing suite you would probably want it to continue. At the end it will tell you whether each process succeeded or failed and how long it took to run.

However, the hard part is writing the processes to do the checks.

In practice I tend to incorporate the tests into the system rather than having a separate test suite.

For example, at the end of a dim build process there will be checks to ensure that all elements below a given top level element only have one parent.

For general day to day testing, I usually build an Excel framework where I perform the same calculations in Excel and compare the results to TM1. I also use it to report on Orphans (elements that have no parents in the hierarchy and which are linked to an Orphans consolidation that then consolidates into All elements.

The other part is setting up a reliable set of test data. In the past it took quite some time to set up a relational database with a set of test data for TM1 ( and also relational Cognos BI) so that we have a reliable set of data for test loads.

However, with a TI process based test suite you will only really be testing loads and dim builds or simulations of data entry, but you still won't be testing the data entry sheets, security rights, etc.

Regards

Paul Simon

[CathyBoots]

hi all

I'd be very interested if there's a way to automate testing in TM1. We have a large model including around 500 excel/perspectives reports using VBA and 500 processes. If we could get at least some of that automated it would make upgrades and improvements easier and less risky!

Thanks
Cathy

[paulsimon]

Hi Cathy

It looks like you are more interested in comparing Excel workbook reports? If so, then we use a utility to do this. If you are looking at doing regression testing after a version upgrade, then we use the following approach:

Start the old and new versions on two different servers with identical data.

Run the same set of reports from each server
We also have a utility that slices and paste special values the default view on each cube in the server to Excel so that can be compared.

We then have a tool that compares the Excel reports from the two servers and reports any differences.

We then run key TI processes on the two servers, eg the month end rollover

We then run the reports and the comparison again.

Its not totally automated but for the frequency with which we upgrade it has been good enough for us.

Regards

Paul Simon