Visualizing and Analyzing TM1 Data Flow

[Benno]

UPDATE 2016-11-09
First public release:
https://github.com/kaylon/tm1doc

Hey guys,
this is my first forum post. So I'll just introduce myself real quick:
I am a BI dev, focussed on TM1 at ma mid size pharmaceutical company in Germany. I have been working with TM1 for roughly three years now and I love and hate it at the same time I'm sure most of you understand what I mean. My work focusses mostly on the companies financial side as you will see later in this post.

When I discovered the REST API I did kind of a rework of this little tool (http://www.bihints.com/graphing_tm1_data_flow) in python.
It has minimal dependencies (python, networkx (pip install networkx) and graphviz) and will give you a model of your TM1 Server and its data sources. The internal data model is a little more complex. It would be possible to render all sorts of graphs and run all sorts of dependency queries between tm1 objects.
I plan to extend it, integrating a logparser to augment the model with statistics about loading times, cube sizes etc. In a perfect world I would continue to write a simple analysis algorithm that cuts the graph into independent subgraphs and schedules a multi-thread loader. But that is a discussion for another day.

For now I am interested to know if you guys would be interested in that piece of software. I am thinking of putting it up on github for everyone to use, reuse and expand.

As an example this the Architect-View:

tm1_architect.png (42.6 KiB) Viewed 6829 times

My tool can create (among others) this image:

tm1.png (186.4 KiB) Viewed 6829 times

(blue = ODBC tables, pink = cubes)
Modifying the style of the ouput graph is very simple if you don't like the colors.

So let me know what you think!




Cheers
Benno

[qml]

There are commercial products out there that do similar things e.g Cubewise's Pulse for TM1 (I am not affiliated). But of course it's a great idea to have an open-source solution for relationship charting, especially for those that don't have the need for something as feature-rich as a commercial product or those that might want to play with the code themselves. I'm sure there will be people interested in using your tool or even wanting to participate in its development. So I say: go for it!

[Steve Rowe]

Hi Benno,
Fair play on getting this far, as qml says I'm sure there will be people interested in partaking.

A query on your system diagram, does your tool get to the layout shown automatically or have you rearranged the boxes post generation? I have seen many tools that could graphically document the links but very few (if any) that get to a sensible arrangement.

Cheers and good luck!

[Benno]

Hey guys, that was quick.

@ qml: Yeah the Cubewise software sounds very appealing. But after I got a price quote on their Pulse (I think) I started doing it on my own.

@ Steve: the layout is created by graphviz. You can use all the generators, I prefer dot for this, because it captures the hierarchy nicely. My tool will simply output a string that is piped into graphviz. There are a lot of ways to configure the rendering process.

I'll try to write up some readme, add one more feature and push it to github by weeks end.

[Wim Gielis]

Sounds very good Benno, I would be happy to use it !

[Edward Stuart]

I'd be interested in contributing to the project, I was looking at a similar project utilising the collapsible tree style seen here http://bl.ocks.org/mbostock/4339083

[Benno]

I published the software on github. (link -> see op)

A few short notes:
- It is released under MIT License, so you can do nearly whatever you want with it. However I would like it if you contribute cool changes back to the project.
- It will always be free.
- it will always be open source.
- it will never send any of your data anywhere.
- I created the project for me, so it supprts my coding style. I tried to make some necessary assumptions clear in the readme. If it won't work for you give me some feedback and we can improve on it together.
- I like to integrate a lot of automatic tests. Especially I want to add some demo instance or run it on one of IBMs demo instances that ship with TM1. Do you have any suggestions on that?


@Edward: adding that should be easy.