STARGATE VIEWS CREATION

Post Reply
zodiacviv
Posts: 16
Joined: Tue Nov 05, 2013 9:07 am
OLAP Product: cognos tm1
Version: 10.2.1
Excel Version: 2010

STARGATE VIEWS CREATION

Post by zodiacviv »

How to use ViewConstruct function for stargate views and do i need to set VTM and VMM values for it and how can i check if my stargate view is created or not. Should i create another TI process and use VIEWCONSTRUCT there?? I am using a cube of size 10 GB and mainly i am using one view of this cube frequently for different process. Do stargate views has any impact on TI Process execution time? Any information regarding stargate view is really appreciated.
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: STARGATE VIEWS CREATION

Post by TableManagerOne »

You will almost certainly need to adjust VMT and VMM as the defaults are not effective. VMT essentially says "How long, in seconds, must an individual stargate calculation take before its deemed worthy of saving in the cache." VMM says, "How much memory, in KB (the doc says bytes?), can we spare to cache *all* stargates of a given cube".

So, with VMT set to 5, a stargate that takes 4 seconds will not be considered for cache, while a stargate that takes 77 seconds will. If the 77 second stargate is 2 MB, and VMM is set to 1024 (1MB,) it cannot be cached. If that stargate was 512 KB, it can be, however, it may need to "push out" old stargates if 512 KB isn't available.

As I often say, debug logging is your friend here:

Code: Select all

log4j.logger.TM1.Cube.Stargate=DEBUG
log4j.logger.TM1.Cube.Stargate.Reference=INFO
Stargate will tell you the time it took for calculation, in memory size, etc. Stargate.ViewStorage will tell you when stargates are cached, when they are pushed out for lack of space, etc. (I disable the Reference sub-logger to filter noise.)

The built in Performance Monitor (stats cubes) and Performance Counters may also reflect some of this information.

I've heard that a sub-process to do the ViewConstruct is necessary for the master process to leverage - though I don't know any details. Ignoring this though, yes, stargates can absolutely (positively) the execution time of a TI.
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: STARGATE VIEWS CREATION

Post by tomok »

TableManagerOne wrote:You will almost certainly need to adjust VMT and VMM as the defaults are not effective. VMT essentially says "How long, in seconds, must an individual stargate calculation take before its deemed worthy of saving in the cache." VMM says, "How much memory, in KB (the doc says bytes?), can we spare to cache *all* stargates of a given cube".

So, with VMT set to 5, a stargate that takes 4 seconds will not be considered for cache, while a stargate that takes 77 seconds will. If the 77 second stargate is 2 MB, and VMM is set to 1024 (1MB,) it cannot be cached. If that stargate was 512 KB, it can be, however, it may need to "push out" old stargates if 512 KB isn't available.

As I often say, debug logging is your friend here:

Code: Select all

log4j.logger.TM1.Cube.Stargate=DEBUG
log4j.logger.TM1.Cube.Stargate.Reference=INFO
Stargate will tell you the time it took for calculation, in memory size, etc. Stargate.ViewStorage will tell you when stargates are cached, when they are pushed out for lack of space, etc. (I disable the Reference sub-logger to filter noise.)

The built in Performance Monitor (stats cubes) and Performance Counters may also reflect some of this information.

I've heard that a sub-process to do the ViewConstruct is necessary for the master process to leverage - though I don't know any details. Ignoring this though, yes, stargates can absolutely (positively) the execution time of a TI.
Mods, can we incorporate this post into the FAQ somehow? This is great information on a relatively misunderstood topic. Thanks for posting.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: STARGATE VIEWS CREATION

Post by TableManagerOne »

You're welcome. Wrt loggers - I'm a fan, though they aren't all that easy to work with. I was considering writing up a blog post somewhere (or more ideally, starting a wiki page) that would cover some of the logging tricks I've picked up over the years: How to find out which loggers are available. How to track down lock contention. How to diagnose cache issues. etc. Maybe this weekend...
User avatar
Alan Kirk
Site Admin
Posts: 6606
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: PA2.0.9.18 Classic NO PAW!
Excel Version: 2013 and Office 365
Location: Sydney, Australia
Contact:

Re: STARGATE VIEWS CREATION

Post by Alan Kirk »

tomok wrote: Mods, can we incorporate this post into the FAQ somehow? This is great information on a relatively misunderstood topic.
That's been done.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.
lotsaram
MVP
Posts: 3648
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: STARGATE VIEWS CREATION

Post by lotsaram »

TableManagerOne wrote:You will almost certainly need to adjust VMT and VMM as the defaults are not effective. VMT essentially says "How long, in seconds, must an individual stargate calculation take before its deemed worthy of saving in the cache." VMM says, "How much memory, in KB (the doc says bytes?), can we spare to cache *all* stargates of a given cube".
Can you or anyone else please confirm that the doco is incorrect and VMM is in fact specified in KB and not Bytes?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
hyunjia
Posts: 64
Joined: Fri Jul 27, 2012 4:13 pm
OLAP Product: TM1
Version: 2010
Excel Version: Excel 2010

Re: STARGATE VIEWS CREATION

Post by hyunjia »

lotsaram wrote:
TableManagerOne wrote:You will almost certainly need to adjust VMT and VMM as the defaults are not effective. VMT essentially says "How long, in seconds, must an individual stargate calculation take before its deemed worthy of saving in the cache." VMM says, "How much memory, in KB (the doc says bytes?), can we spare to cache *all* stargates of a given cube".
Can you or anyone else please confirm that the doco is incorrect and VMM is in fact specified in KB and not Bytes?
This is really what I've been looking for . Has anyone tested it ? I can't risk the possibility that it may blow up the memory in server , but I might just create a local instance and play with the setting .
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: STARGATE VIEWS CREATION

Post by TableManagerOne »

Can you or anyone else please confirm that the doco is incorrect and VMM is in fact specified in KB and not Bytes?
I'll test it out later tonight when I have some time.

If someone would like to beat me to it though, my plan is just to enable Stargate Debug logging and note the memory used when constructing some view. I'd then set the threshold below and above this level, in terms of KB, and see if that stargate gets cached as it should.
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: STARGATE VIEWS CREATION

Post by TableManagerOne »

I have some results to share.

To test, I build a dead simple cube, 'StargateCacheTest', comprised of two dimensions, 'd1' and 'd2'.
'd1' had elements 'x', 'y', and 'total' (consolidation of x&y).
'd2' had elements 'a','b','c', and 'd'.

I attached the following rule:

Code: Select all

['x']=sleep( ['y'] * 1000 );
feeders;
['y']=>['x'];
Which allowed me to force a long but small view.

I enabled the following loggers:

Code: Select all

log4j.logger.TM1.Cube.Stargate=DEBUG
log4j.logger.TM1.Cube.Stargate.Reference=INFO
log4j.logger.TM1.Cube.Invalidation=DEBUG
Version tested: 10.2 FP1

I've attached the abridge version of my log bellow, but I'll summarize first.
  • Using the default setting (empty) for VMM, I opened the default view on my cube (after entering some data at 'y'.) The logs indicate that the stargate consumes 82kb, and that it is added to view storage. This indicates that the default is not 64kb (I believe it is 128kb - at least in the version I am using.)
  • I then changed the VMM setting to 80.
  • I invalidated my test view with some writeback, and the logs indicate that the old stargate is discarded.
  • I recalculated my view and the logs did not indicate that a stargate was cached.
  • Just to be sure, I switched the VMM back above the stargate's size to 85, and after invalidating and recalcing, the stargate was cached. This indicates that the VMM setting is, in fact, in terms of kb.

Code: Select all

15792   [2]   DEBUG   2014-01-29 00:35:19.870   TM1.Cube.Invalidation   TM1CubeImpl: Invalidating cube and dependents for cube 'StargateCacheTest'.
15792   [2]   DEBUG   2014-01-29 00:35:19.870   TM1.Cube.Invalidation   TM1CubeImpl::InvalidateCube: Invalidating base cube 'StargateCacheTest'.

15792   [2]   DEBUG   2014-01-29 00:35:23.548   TM1.Cube.Stargate.Definition   
Cube: StargateCacheTest
Cube to View dimension map: 
   [0,1]
   [1,2]
Stargate Expanse: 
   Dimension: d1
      Axis Selections:
         [ (total,U)
         , (x,U)
         , (y,U)
         ]
   Dimension: d2
      Axis Selections:
         [ (b,U)
         , (a,U)
         , (c,U)
         , (d,U)
         ]

15792   [2]   DEBUG   2014-01-29 00:35:41.704   TM1.Cube.Stargate.ViewStorage   ViewStorage::AddStargate: Stargate 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) added to ViewStorage (0x0000000007A19058).
15792   [2]   DEBUG   2014-01-29 00:35:41.704   TM1.Cube.Stargate   New Stargate Created 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) in 18156ms SGSize=82Kb workPoolSize=0Kb hPoolSize=67Kb.

   "13","20140129003612","20140129003612","Admin","S","","80","}CubeProperties","StargateCacheTest","VMM",""
   #"13","20140129003612","Change set 13 complete : 1"

   "14","20140129003627","20140129003627","Admin","N","1","0.7","StargateCacheTest","y","d",""
   #"14","20140129003627","Change set 14 complete : 1"
   
15792   [2]   DEBUG   2014-01-29 00:36:27.040   TM1.Cube.Invalidation   TM1CubeImpl: Invalidating cube and dependents for cube 'StargateCacheTest'.
15792   [2]   DEBUG   2014-01-29 00:36:27.040   TM1.Cube.Invalidation   TM1CubeImpl::InvalidateCube: Invalidating base cube 'StargateCacheTest'.

15792   [2]   DEBUG   2014-01-29 00:36:27.081   TM1.Cube.Stargate.ViewStorage   ViewStorage::DeleteStargate: Stargate 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) removed from ViewStorage (0x0000000007A19058).

15792   [2]   DEBUG   2014-01-29 00:36:37.039   TM1.Cube.Stargate.Definition   
Cube: StargateCacheTest
Cube to View dimension map: 
   [0,1]
   [1,2]
Stargate Expanse: 
   Dimension: d1
      Axis Selections:
         [ (total,U)
         , (x,U)
         , (y,U)
         ]
   Dimension: d2
      Axis Selections:
         [ (b,U)
         , (a,U)
         , (c,U)
         , (d,U)
         ]

15792   [2]   DEBUG   2014-01-29 00:36:44.040   TM1.Cube.Stargate   New Stargate Created 'StargateCacheTest:( [d1].*, [d2].* )' (0x00000000047C0010) in 7001ms SGSize=82Kb workPoolSize=0Kb hPoolSize=67Kb.  
   
   "15","20140129003702","20140129003702","Admin","S","80","85","}CubeProperties","StargateCacheTest","VMM",""
   #"15","20140129003702","Change set 15 complete : 1"
   
   "16","20140129003709","20140129003709","Admin","N","1","0.6","StargateCacheTest","y","c",""
   #"16","20140129003709","Change set 16 complete : 1"
   
15792   [2]   DEBUG   2014-01-29 00:37:09.591   TM1.Cube.Invalidation   TM1CubeImpl: Invalidating cube and dependents for cube 'StargateCacheTest'.
15792   [2]   DEBUG   2014-01-29 00:37:09.591   TM1.Cube.Invalidation   TM1CubeImpl::InvalidateCube: Invalidating base cube 'StargateCacheTest'.

15792   [2]   DEBUG   2014-01-29 00:37:12.248   TM1.Cube.Stargate.Definition   
Cube: StargateCacheTest
Cube to View dimension map: 
   [0,1]
   [1,2]
Stargate Expanse: 
   Dimension: d1
      Axis Selections:
         [ (total,U)
         , (x,U)
         , (y,U)
         ]
   Dimension: d2
      Axis Selections:
         [ (b,U)
         , (a,U)
         , (c,U)
         , (d,U)
         ]

15792   [2]   DEBUG   2014-01-29 00:37:18.450   TM1.Cube.Stargate.ViewStorage   ViewStorage::AddStargate: Stargate 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) added to ViewStorage (0x000000000BF1BD58).
15792   [2]   DEBUG   2014-01-29 00:37:18.450   TM1.Cube.Stargate   New Stargate Created 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) in 6201ms SGSize=82Kb workPoolSize=0Kb hPoolSize=67Kb.
This also illustrates how woefully inadequate the default setting is. My toy view with 12 cells used more than half the allotted space...
hyunjia
Posts: 64
Joined: Fri Jul 27, 2012 4:13 pm
OLAP Product: TM1
Version: 2010
Excel Version: Excel 2010

Re: STARGATE VIEWS CREATION

Post by hyunjia »

This is absolutely very useful info, thanks so much TableManagerOne . Out of my curiosity, what’s the use of Sleep function here? Is it a hidden function from TM1?
zodiacviv
Posts: 16
Joined: Tue Nov 05, 2013 9:07 am
OLAP Product: cognos tm1
Version: 10.2.1
Excel Version: 2010

Re: STARGATE VIEWS CREATION

Post by zodiacviv »

thanks every1 for your valuable reply. Really i got my most of queries solved. Now i created a TI Process and inside it i called ViewConstruct function for all 10 cube whose views are taking good time to open. Can any one help me how to check if my stargate views is created or not?? I have set vmt as 3 and vmm as 30 for a cube for which i am trying to create stargate. Still i am not able to find that stargate view is created or not. for reference i am attaching screen shots
100.PNG
100.PNG (17.34 KiB) Viewed 19611 times

101.PNG
101.PNG (12.61 KiB) Viewed 19611 times
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: STARGATE VIEWS CREATION

Post by tomok »

zodiacviv wrote:Can any one help me how to check if my stargate views is created or not??
Did even take the time to read all the posts in this thread? If you set the logging properties as mentioned above, the creation of stargate views will be logged in the tm1server.log file.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
User avatar
qml
MVP
Posts: 1094
Joined: Mon Feb 01, 2010 1:01 pm
OLAP Product: TM1 / Planning Analytics
Version: 2.0.9 and all previous
Excel Version: 2007 - 2016
Location: London, UK, Europe

Re: STARGATE VIEWS CREATION

Post by qml »

hyunjia wrote:This is absolutely very useful info, thanks so much TableManagerOne . Out of my curiosity, what’s the use of Sleep function here? Is it a hidden function from TM1?
Seems to be an undocumented function that works in both TI and rules that waits (as the name suggests) for a given number of milliseconds. I must say that is quite a discovery for me. The fact that it works in rules blows my mind. TableManagerOne is using it to make sure that the small sample view takes long enough to refresh so that the TM1 engine decides to cache it. Simply brilliant.
Kamil Arendt
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: STARGATE VIEWS CREATION

Post by TableManagerOne »

and vmm as 30
Also, from further up the thread, 30 (30kb) is too low for any useful views.
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: STARGATE VIEWS CREATION

Post by TableManagerOne »

I had heard of Sleep as a TI function, though, it isn't documented (any more?) I've used it in TI for synchronization and lock testing (e.g. perform some operation and sleep for 30 seconds to see if it blocks some other operation I run in another Architect window.)

However, I did come across it in the rules documentation "Rule Function Names":

Code: Select all

...
SIN
SLEEP
SQRT
...
Color me curious, I tried it out, and it worked. I can't imagine any practical uses beyond the hack I presented above though...
hyunjia
Posts: 64
Joined: Fri Jul 27, 2012 4:13 pm
OLAP Product: TM1
Version: 2010
Excel Version: Excel 2010

Re: STARGATE VIEWS CREATION

Post by hyunjia »

qml wrote:
hyunjia wrote:This is absolutely very useful info, thanks so much TableManagerOne . Out of my curiosity, what’s the use of Sleep function here? Is it a hidden function from TM1?
Seems to be an undocumented function that works in both TI and rules that waits (as the name suggests) for a given number of milliseconds. I must say that is quite a discovery for me. The fact that it works in rules blows my mind. TableManagerOne is using it to make sure that the small sample view takes long enough to refresh so that the TM1 engine decides to cache it. Simply brilliant.
Now , that was Ah-Ha moment for me :) tahnks qml . I agree with TableManagerOne , there are more than you would expected from this chocolate box called TM1
PlanningDev
Community Contributor
Posts: 349
Joined: Tue Aug 17, 2010 6:31 am
OLAP Product: Planning Analytics
Version: 2.0.5
Excel Version: 2016

Re: STARGATE VIEWS CREATION

Post by PlanningDev »

One last thing, can someone confirm if a sever restart is required for VMM changes to take effect?
TableManagerOne
Posts: 42
Joined: Fri Apr 19, 2013 7:07 pm
OLAP Product: TM1
Version: 10.1 RP1 FP1
Excel Version: 2003 SP3

Re: STARGATE VIEWS CREATION

Post by TableManagerOne »

Sorry, I didn't make it clear above, but my test case, reflected in the log extract attached above, was in a single server session. I changed VMM on the fly, and it took effect immediately. No restart required.
Post Reply