Approaching project complexity

Post Reply
User avatar
20 Ton Squirrel
Posts: 71
Joined: Tue Jul 14, 2020 9:53 pm
OLAP Product: TM1
Version: Planning Analytics with Watson
Excel Version: Office 365
Location: Houston, TX

Approaching project complexity

Post by 20 Ton Squirrel »

I want to discuss project complexity and learn about the limits of TM1. I dream big but a recent project is proving to me that just because I can do something in TM1 doesn't mean I should. It's too late for me to turn back now but I'd like to learn how other developers approach project inception to avoid this in the future.

Here are questions that I'm curious about:
  • How do you take on TM1 projects initially?
  • What steps are taken to assess if TM1 is the right fit?
  • Do you set a limit on complexity by the number of lines in rules? For example, does a single model with 20+ cubes and over 15,000 combined lines of rules raise red flags?
  • At what point do you say HOLD UP let's use another tool?
  • Do you have a set of guidelines that dictate the appropriate tool for the job or just go with your gut?
I would truly appreciate any input you fellow developers could provide. Thanks :)
War teaches us geography, getting old teaches us biology.
MarenC
Regular Participant
Posts: 350
Joined: Sat Jun 08, 2019 9:55 am
OLAP Product: Planning Analytics
Version: Planning Analytics 2.0
Excel Version: Excel 2016

Re: Approaching project complexity

Post by MarenC »

Hi,

I think what is missing from your reasonable list is this:

Have we got the requisite skills and experience in the technologies to make such assessments?
Are the people we are getting advice from giving us objective advice or selling us something?
Are the people developing the solution appropriately qualified, and getting the best out of the technology, to deliver the objectives in the most efficient way? Bad workmen blame their tools :D

Maren
lotsaram
MVP
Posts: 3654
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: Approaching project complexity

Post by lotsaram »

Great topic. Just replying in order to follow the thread. When I get some time I will come back and write a proper response.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
User avatar
20 Ton Squirrel
Posts: 71
Joined: Tue Jul 14, 2020 9:53 pm
OLAP Product: TM1
Version: Planning Analytics with Watson
Excel Version: Office 365
Location: Houston, TX

Re: Approaching project complexity

Post by 20 Ton Squirrel »

My project has taken on so much complexity and scope creep that it raised the concern of upper management and shook their confidence in TM1 a bit. I'm still confident on my course but I can't blame them for doubts.

I want to learn more from the experience and having YOUR feedback can help me approach it better in the future, so thank you all very much for taking the time to answer. I appreciate it. :)
War teaches us geography, getting old teaches us biology.
burnstripe
Regular Participant
Posts: 197
Joined: Wed May 06, 2020 2:58 pm
OLAP Product: Planning Analytics
Version: 2.0.9
Excel Version: 2016

Re: Approaching project complexity

Post by burnstripe »

I wouldn't say complexity is determined by how many cubes or rules you have but rather how manageable will it be to support and develop on into the future.

20+ cubes isn't bad or unmanageable, personally I tend to favour smaller more manageable cubes over all in one cubes for efficiency and performance. With a good naming convention they can even be easier to update.

1000 lines of rules per cube I would say is on the high side but could still be manageable. Saying that though when I typically see rules this long it's normally with a head in hands moment because there's hundreds of lines doing the exact same thing which could have been done instead with just a few lines. I've seen some hideous ones :)

I think the most critical part of any project is the start (requirement gathering). Before beginning any big project I would do a mind maps, data flow diagrams and report drawings.

First step is I would hold workshops with the client/users and produce Mind maps I would jot down anything relevant:
Key contacts/Software the client has/licence counts (if applicable)/User requirements etc...

Then I'd take away those ideas/notes and try and produce some kind of data flow. This may include a tm1 diagram containing cubes/dimensions showing rule and process links. I would also include flows to other tools like sql/oracle or reporting tools like powerbi/cognos analytics.

If you're not sure what tool to use take it a step higher, and determine
What data repositories are required to store/grab information
What interfaces may the user require, aka reports/entry forms

Then you can thrash out this diagram with the client/users and adjust accordingly. It's during this stage you'll be able to determine what tools might be most appropriate for the job and understand how long it may take, at which point you may swap and change things.

Once this is all in place the development is a lot easier and because design issues where thrashed at early on, the project is more likely to succeed, be on schedule and require minimal redevelopment.

In short plan, plan and more planning before you start developing.

I hope that is somewhat useful, and keep your chin up we all learn through the difficult moments.
User avatar
20 Ton Squirrel
Posts: 71
Joined: Tue Jul 14, 2020 9:53 pm
OLAP Product: TM1
Version: Planning Analytics with Watson
Excel Version: Office 365
Location: Houston, TX

Re: Approaching project complexity

Post by 20 Ton Squirrel »

Burnstripe, my dude, that was awesome. Thank you.

The initial planning stages are really where this all went awry. The client came to us with a massive shambolic model that was broken because he couldn't extend the time periods any further. Worse, he INHERITED the model from someone who left the company and there was no documentation, so he didn't know all the ins/outs. This introduced all manner of scope creep into development since we were constantly finding surprises. >__<

Hindsight tells me I should have made the client reproduce the model on a much smaller scale, essentially make his own proof of concept within Excel. This would force him to understand his model and have practical examples for each stage. If he can't explain it to himself, he certainly can't explain it to me!
War teaches us geography, getting old teaches us biology.
aboyzhou
Posts: 15
Joined: Sun Jul 17, 2016 4:21 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2010

Re: Approaching project complexity

Post by aboyzhou »

Maybe, I think that you should read and undstand inner moduler logic with technical language, and then talk about business logic with business user and make them confirmed.

If you would like to business user can undstand TM1 technical logic, this is crazy idea.

I think that there are so many work must be done by yourself, not business user. because they don't undstand TM1 logic langulage and they can't draw whole picture with business logic.

Best Regards,
burnstripe
Regular Participant
Posts: 197
Joined: Wed May 06, 2020 2:58 pm
OLAP Product: Planning Analytics
Version: 2.0.9
Excel Version: 2016

Re: Approaching project complexity

Post by burnstripe »

It's always difficult picking up someone elses mess. Last year I encountered a model that took 5 hours to start and had over 3000 lines in some cubes, it was easy to spot the fundamental flaws with the cube structures but I made the mistake of not allocating a day or two properly looking through the model with a fine tooth comb as after that length of time it became apparent that it need a belt and braces approach and practically rebuilt the whole model. Took 6 weeks instead of 4 and an awful lot of testing/bug fixing but the model now starts in 2 minutes and the client couldn't be happier.

But now if I were coming into an unfamiliar model, I would allocate a day or two sifting (depending on size) through the environment before conjuring a solution so that I can set more realistic expectations.
User avatar
20 Ton Squirrel
Posts: 71
Joined: Tue Jul 14, 2020 9:53 pm
OLAP Product: TM1
Version: Planning Analytics with Watson
Excel Version: Office 365
Location: Houston, TX

Re: Approaching project complexity

Post by 20 Ton Squirrel »

aboyzhou wrote: Tue Mar 08, 2022 1:41 am Maybe, I think that you should read and undstand inner moduler logic with technical language, and then talk about business logic with business user and make them confirmed.

If you would like to business user can undstand TM1 technical logic, this is crazy idea.

I think that there are so many work must be done by yourself, not business user. because they don't undstand TM1 logic langulage and they can't draw whole picture with business logic.
Agreed, Aboyzhou. Most of my clients have never heard of TM1 before so expecting them to understand the technical aspects of the system would be asking far too much. It is on me, the developer, to fully grasp the tools I intend to implement.

The client should have a conceptual grasp of what they want developed. There should be clearly defined goals. I might have tricks and tools up my sleeve to automate or improve their ideas, however, if they come to my doorstep with another broken, undocumented model… I might just flip a table. ;D
War teaches us geography, getting old teaches us biology.
Wim Gielis
MVP
Posts: 3117
Joined: Mon Dec 29, 2008 6:26 pm
OLAP Product: TM1, Jedox
Version: PAL 2.0.9.18
Excel Version: Microsoft 365
Location: Brussels, Belgium
Contact:

Re: Approaching project complexity

Post by Wim Gielis »

The question also arises: do we train the key users to be in TM1 before requirements gathering and workshops. Or afterwards ? With knowledge of the tool they might benefit and improve the efficiency and understand the consequences of what they ask for.
Best regards,

Wim Gielis

IBM Champion 2024
Excel Most Valuable Professional, 2011-2014
https://www.wimgielis.com ==> 121 TM1 articles and a lot of custom code
Newest blog article: Deleting elements quickly
User avatar
20 Ton Squirrel
Posts: 71
Joined: Tue Jul 14, 2020 9:53 pm
OLAP Product: TM1
Version: Planning Analytics with Watson
Excel Version: Office 365
Location: Houston, TX

Re: Approaching project complexity

Post by 20 Ton Squirrel »

Wim Gielis wrote: Tue Mar 08, 2022 10:26 pm The question also arises: do we train the key users to be in TM1 before requirements gathering and workshops. Or afterwards ? With knowledge of the tool they might benefit and improve the efficiency and understand the consequences of what they ask for.
In my experience user acceptance is often the biggest hurdle, regardless of the tool. Clients want things fixed but, paradoxically, they don't want to change anything. I've had mixed reactions offering TM1 but none of it was immediately positive. People feared complexity, costs, lack of support… in short, they feared losing control of their tools, even if they were completely broken.

We have a few examples on TM1 that are used for demonstration, it always helps to SEE something working. If the project is simple enough we try to build a proof of concept to demonstrate the possibilities. Even with all that, there are stubborn, diehard types who loathe the idea of change.

My company is working on establishing a "TM1 Ambassador" that can tackle this specific hurdle of user acceptance. This will ease the burden on me as a developer since it will free me from making presentations, explanations, or negotiations. Seems like a good concept, at least.

Thorough documentation and training can't be neglected. I find most of the documentation/training IBM offers lacking or too technical for my clientele. We're forced to write up our own stuff, which is always a drag. ;D
War teaches us geography, getting old teaches us biology.
User avatar
20 Ton Squirrel
Posts: 71
Joined: Tue Jul 14, 2020 9:53 pm
OLAP Product: TM1
Version: Planning Analytics with Watson
Excel Version: Office 365
Location: Houston, TX

Re: Approaching project complexity

Post by 20 Ton Squirrel »

Ima just nudge this here post a little in hopes of getting lotsaram to respond. No rush, don't mind me. ;)
War teaches us geography, getting old teaches us biology.
Mark RMBC
Community Contributor
Posts: 292
Joined: Tue Sep 06, 2016 7:55 am
OLAP Product: TM1
Version: 10.1.1
Excel Version: Excel 2010

Re: Approaching project complexity

Post by Mark RMBC »

Hi,

Just to keep you entertained until lotsaram shows up, here are my thoughts:

I assume your questions relate to internal developers within an organisation (external consultancy firms would have differences)
How do you take on TM1 projects initially?
One of 2 ways:

Boss informs you of a new requirement.
Developer team actively seek out opportunities within organisation, as part of efficiency drives etc. Increasing data maturity and all that spiel.
What steps are taken to assess if TM1 is the right fit?
Assess requirements - from security, calculations, to how users will interact with the solution. How long development is estimated to take.
Understand source data
Understand data flows

I should note that in some TM1 models I have developed, the solutions sometimes come from outside TM1. For example, I had to report and plan agency staff, and in order to facilitate this, procurement and payroll had to change their procedures to give the agency an employee number and include certain data on the invoice order.
Another example, an organisation changed its entire vacancy rules to facilitate a TM1 project, so vacancies older than n number of months got auto deleted.
Do you set a limit on complexity by the number of lines in rules? For example, does a single model with 20+ cubes and over 15,000 combined lines of rules raise red flags?
For me it is more a question of userability vs performance. TI's vs Rules etc.
I notice in a few of your posts that you don't mention TI's much but you do mention rules, I personally like to see a mix. But depends on the model.

One bit of advice I would have is to always include a Manual Adjustment measure where appropriate, so the end users can adjust the figures by simply entering a value.
This gives the end users some level of control and avoids you having to continually tinker with the logic to cater for immaterial and trivial requests. It can also tick off a few problematical but trivial areas during development!

Also, ensure the model is future proof and includes appropriate error trapping. For example, if a data load fails it should be easy just to reload, rather than say having to restore a back up.

At what point do you say HOLD UP let's use another tool?
At the point there is a complete show stopper! Otherwise you muddle on.

I remember starting work on a schools insurance fund model. At the same time the web team were developing a new front end for schools to submit insurance details.
The way the web form was designed meant a veritable forest of trees/paths, which resulted in the TM1 source file having 400+ columns.
At that stage I was wishing TM1 had not been the go to solution!
Do you have a set of guidelines that dictate the appropriate tool for the job or just go with your gut?
Generally it is a gut feeling, though one based on past experience and knowledge of the tool.
I usually reject TM1 as a solution if the requirement is really simple, as I often think why are they suggesting using a sledgehammer to crack a walnut.
Generally if it is really complex, and it makes material efficiencies I am more inclined to suggest TM1.


Lastly, I fully agree that managing user expectations is the key, and emphasising materiality. Getting excel loving users on board is the biggest challenge in my experience.

regards,

Mark
User avatar
20 Ton Squirrel
Posts: 71
Joined: Tue Jul 14, 2020 9:53 pm
OLAP Product: TM1
Version: Planning Analytics with Watson
Excel Version: Office 365
Location: Houston, TX

Re: Approaching project complexity

Post by 20 Ton Squirrel »

All input is welcome so thanks for taking the time, Mark!

You are correct, my clients are all internal. Everything I do in TM1 is based around data input, modeling, and forecasting… I make things efficient and take data management out of the hands of clumsy analysts. Dancing to the tune of external clients is an entirely different game, I'm content to be nestled in the dark inner workings of this corporation.

It is interesting that you align TM1 with complexity. Given my recent project, management is of the mind that TM1 is not a good candidate for projects requiring a high level of agility and Excel should be considered. I disagree but my only refute to the claim is to ask, "If Excel failed in the original version, how would you build this monstrosity in Excel all over again?"

As for rules versus process, I have a better grasp of scripting than I do rules/feeders so I don't tend to post questions for scripts. I confess that I have trust issues with analysts skipping steps, so I get caught up making everything as rule-based as possible to avoid manual steps in the workflow. Looking at this recent project, it seems TM1Py would have been a better option to automate some of the iterative balancing stuff going on. :T

I'm still learning ;)
War teaches us geography, getting old teaches us biology.
User avatar
Steve Rowe
Site Admin
Posts: 2416
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: Approaching project complexity

Post by Steve Rowe »

I've been following this thread with interest.

20TS, my reading of the issues you are raising is that they were process and project management issues rather than a technology problem?

Any project against any technology is going to look a mess if the initial scope is not correct and then there are significant levels of creep throughout the duration. You've touched on this a few times, so I think it's clear to you where it went wrong too? It obviously depends on the organisation but I think an internal person is a lot more at risk of problems arising from informal project managment.

Your comment about getting the SME to do a version of the Excel model that would of formed the basis of the requirements would have been a good step in getting the foundations right. When in your situaton I would get the SME to do a model that has all the "maths" in it but has all the volume related pieces removed (things like versioning, lots of different time periods, every single SKU etc), we know TM1 can do volume (within limits of course!) What we need to see in the Excel model is the full workings of the model.

Your comments about agility / flexability are also something we come across occasionally. If the users of the model have an expectation that they should be able to change literally any part of the model at any given time, then I would say there are really two options.

1. Push back on the requirement, complete freedom for the user to change the model whenever they feel like with no logic is not a sensible basis for a business model.
2. More analysis is required to understand the logic the user is applying and build this into the model. Quite often what seems like a special case isn't if you can generalise your logic sufficently.

If your requirements and users are all in camp 1 and want complete freedom to change everything at anytime then they aren't good candidates for any centralised application and IMO its still not a technology problem. Putting some rigour and controls in place is usually why management want appplication in the first place....

So in terms of convincing your management team about why TM1, I'd emphasise the positives of TM1. Control, auditability, confidence in the results, one version of the truth, etc, etc.
Technical Director
www.infocat.co.uk
User avatar
20 Ton Squirrel
Posts: 71
Joined: Tue Jul 14, 2020 9:53 pm
OLAP Product: TM1
Version: Planning Analytics with Watson
Excel Version: Office 365
Location: Houston, TX

Re: Approaching project complexity

Post by 20 Ton Squirrel »

Steve Rowe wrote:So in terms of convincing your management team about why TM1, I'd emphasise the positives of TM1. Control, auditability, confidence in the results, one version of the truth, etc, etc.
The phrase "one version of truth" is my team mantra, that is the ultimate goal.

I like your commentary about agility, THAT is a good and proper response to the doubts on TM1… especially #2! I'll be quoting you in my next project meeting. ;D
War teaches us geography, getting old teaches us biology.
Post Reply