STARGATE VIEWS CREATION

[zodiacviv]

How to use ViewConstruct function for stargate views and do i need to set VTM and VMM values for it and how can i check if my stargate view is created or not. Should i create another TI process and use VIEWCONSTRUCT there?? I am using a cube of size 10 GB and mainly i am using one view of this cube frequently for different process. Do stargate views has any impact on TI Process execution time? Any information regarding stargate view is really appreciated.

[TableManagerOne]

You will almost certainly need to adjust VMT and VMM as the defaults are not effective. VMT essentially says "How long, in seconds, must an individual stargate calculation take before its deemed worthy of saving in the cache." VMM says, "How much memory, in KB (the doc says bytes?), can we spare to cache *all* stargates of a given cube".

So, with VMT set to 5, a stargate that takes 4 seconds will not be considered for cache, while a stargate that takes 77 seconds will. If the 77 second stargate is 2 MB, and VMM is set to 1024 (1MB,) it cannot be cached. If that stargate was 512 KB, it can be, however, it may need to "push out" old stargates if 512 KB isn't available.

As I often say, debug logging is your friend here:

Code: Select all

log4j.logger.TM1.Cube.Stargate=DEBUG
log4j.logger.TM1.Cube.Stargate.Reference=INFO

Stargate will tell you the time it took for calculation, in memory size, etc. Stargate.ViewStorage will tell you when stargates are cached, when they are pushed out for lack of space, etc. (I disable the Reference sub-logger to filter noise.)

The built in Performance Monitor (stats cubes) and Performance Counters may also reflect some of this information.

I've heard that a sub-process to do the ViewConstruct is necessary for the master process to leverage - though I don't know any details. Ignoring this though, yes, stargates can absolutely (positively) the execution time of a TI.

[tomok]

You will almost certainly need to adjust VMT and VMM as the defaults are not effective. VMT essentially says "How long, in seconds, must an individual stargate calculation take before its deemed worthy of saving in the cache." VMM says, "How much memory, in KB (the doc says bytes?), can we spare to cache *all* stargates of a given cube".

So, with VMT set to 5, a stargate that takes 4 seconds will not be considered for cache, while a stargate that takes 77 seconds will. If the 77 second stargate is 2 MB, and VMM is set to 1024 (1MB,) it cannot be cached. If that stargate was 512 KB, it can be, however, it may need to "push out" old stargates if 512 KB isn't available.

As I often say, debug logging is your friend here:

Code: Select all

log4j.logger.TM1.Cube.Stargate=DEBUG
log4j.logger.TM1.Cube.Stargate.Reference=INFO

Stargate will tell you the time it took for calculation, in memory size, etc. Stargate.ViewStorage will tell you when stargates are cached, when they are pushed out for lack of space, etc. (I disable the Reference sub-logger to filter noise.)

The built in Performance Monitor (stats cubes) and Performance Counters may also reflect some of this information.

I've heard that a sub-process to do the ViewConstruct is necessary for the master process to leverage - though I don't know any details. Ignoring this though, yes, stargates can absolutely (positively) the execution time of a TI.

Mods, can we incorporate this post into the FAQ somehow? This is great information on a relatively misunderstood topic. Thanks for posting.

[TableManagerOne]

You're welcome. Wrt loggers - I'm a fan, though they aren't all that easy to work with. I was considering writing up a blog post somewhere (or more ideally, starting a wiki page) that would cover some of the logging tricks I've picked up over the years: How to find out which loggers are available. How to track down lock contention. How to diagnose cache issues. etc. Maybe this weekend...

[Alan Kirk]

Mods, can we incorporate this post into the FAQ somehow? This is great information on a relatively misunderstood topic.

That's been done.

[lotsaram]

You will almost certainly need to adjust VMT and VMM as the defaults are not effective. VMT essentially says "How long, in seconds, must an individual stargate calculation take before its deemed worthy of saving in the cache." VMM says, "How much memory, in KB (the doc says bytes?), can we spare to cache *all* stargates of a given cube".

Can you or anyone else please confirm that the doco is incorrect and VMM is in fact specified in KB and not Bytes?

[hyunjia]

You will almost certainly need to adjust VMT and VMM as the defaults are not effective. VMT essentially says "How long, in seconds, must an individual stargate calculation take before its deemed worthy of saving in the cache." VMM says, "How much memory, in KB (the doc says bytes?), can we spare to cache *all* stargates of a given cube".

Can you or anyone else please confirm that the doco is incorrect and VMM is in fact specified in KB and not Bytes?

This is really what I've been looking for . Has anyone tested it ? I can't risk the possibility that it may blow up the memory in server , but I might just create a local instance and play with the setting .

[TableManagerOne]

Can you or anyone else please confirm that the doco is incorrect and VMM is in fact specified in KB and not Bytes?

I'll test it out later tonight when I have some time.

If someone would like to beat me to it though, my plan is just to enable Stargate Debug logging and note the memory used when constructing some view. I'd then set the threshold below and above this level, in terms of KB, and see if that stargate gets cached as it should.

[TableManagerOne]

I have some results to share.

To test, I build a dead simple cube, 'StargateCacheTest', comprised of two dimensions, 'd1' and 'd2'.
'd1' had elements 'x', 'y', and 'total' (consolidation of x&y).
'd2' had elements 'a','b','c', and 'd'.

I attached the following rule:

Code: Select all

['x']=sleep( ['y'] * 1000 );
feeders;
['y']=>['x'];

Which allowed me to force a long but small view.

I enabled the following loggers:

Code: Select all

log4j.logger.TM1.Cube.Stargate=DEBUG
log4j.logger.TM1.Cube.Stargate.Reference=INFO
log4j.logger.TM1.Cube.Invalidation=DEBUG

Version tested: 10.2 FP1

I've attached the abridge version of my log bellow, but I'll summarize first.

• Using the default setting (empty) for VMM, I opened the default view on my cube (after entering some data at 'y'.) The logs indicate that the stargate consumes 82kb, and that it is added to view storage. This indicates that the default is not 64kb (I believe it is 128kb - at least in the version I am using.)
• I then changed the VMM setting to 80.
• I invalidated my test view with some writeback, and the logs indicate that the old stargate is discarded.
• I recalculated my view and the logs did not indicate that a stargate was cached.
• Just to be sure, I switched the VMM back above the stargate's size to 85, and after invalidating and recalcing, the stargate was cached. This indicates that the VMM setting is, in fact, in terms of kb.

Code: Select all

15792   [2]   DEBUG   2014-01-29 00:35:19.870   TM1.Cube.Invalidation   TM1CubeImpl: Invalidating cube and dependents for cube 'StargateCacheTest'.
15792   [2]   DEBUG   2014-01-29 00:35:19.870   TM1.Cube.Invalidation   TM1CubeImpl::InvalidateCube: Invalidating base cube 'StargateCacheTest'.

15792   [2]   DEBUG   2014-01-29 00:35:23.548   TM1.Cube.Stargate.Definition   
Cube: StargateCacheTest
Cube to View dimension map: 
   [0,1]
   [1,2]
Stargate Expanse: 
   Dimension: d1
      Axis Selections:
         [ (total,U)
         , (x,U)
         , (y,U)
         ]
   Dimension: d2
      Axis Selections:
         [ (b,U)
         , (a,U)
         , (c,U)
         , (d,U)
         ]

15792   [2]   DEBUG   2014-01-29 00:35:41.704   TM1.Cube.Stargate.ViewStorage   ViewStorage::AddStargate: Stargate 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) added to ViewStorage (0x0000000007A19058).
15792   [2]   DEBUG   2014-01-29 00:35:41.704   TM1.Cube.Stargate   New Stargate Created 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) in 18156ms SGSize=82Kb workPoolSize=0Kb hPoolSize=67Kb.

   "13","20140129003612","20140129003612","Admin","S","","80","}CubeProperties","StargateCacheTest","VMM",""
   #"13","20140129003612","Change set 13 complete : 1"

   "14","20140129003627","20140129003627","Admin","N","1","0.7","StargateCacheTest","y","d",""
   #"14","20140129003627","Change set 14 complete : 1"
   
15792   [2]   DEBUG   2014-01-29 00:36:27.040   TM1.Cube.Invalidation   TM1CubeImpl: Invalidating cube and dependents for cube 'StargateCacheTest'.
15792   [2]   DEBUG   2014-01-29 00:36:27.040   TM1.Cube.Invalidation   TM1CubeImpl::InvalidateCube: Invalidating base cube 'StargateCacheTest'.

15792   [2]   DEBUG   2014-01-29 00:36:27.081   TM1.Cube.Stargate.ViewStorage   ViewStorage::DeleteStargate: Stargate 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) removed from ViewStorage (0x0000000007A19058).

15792   [2]   DEBUG   2014-01-29 00:36:37.039   TM1.Cube.Stargate.Definition   
Cube: StargateCacheTest
Cube to View dimension map: 
   [0,1]
   [1,2]
Stargate Expanse: 
   Dimension: d1
      Axis Selections:
         [ (total,U)
         , (x,U)
         , (y,U)
         ]
   Dimension: d2
      Axis Selections:
         [ (b,U)
         , (a,U)
         , (c,U)
         , (d,U)
         ]

15792   [2]   DEBUG   2014-01-29 00:36:44.040   TM1.Cube.Stargate   New Stargate Created 'StargateCacheTest:( [d1].*, [d2].* )' (0x00000000047C0010) in 7001ms SGSize=82Kb workPoolSize=0Kb hPoolSize=67Kb.  
   
   "15","20140129003702","20140129003702","Admin","S","80","85","}CubeProperties","StargateCacheTest","VMM",""
   #"15","20140129003702","Change set 15 complete : 1"
   
   "16","20140129003709","20140129003709","Admin","N","1","0.6","StargateCacheTest","y","c",""
   #"16","20140129003709","Change set 16 complete : 1"
   
15792   [2]   DEBUG   2014-01-29 00:37:09.591   TM1.Cube.Invalidation   TM1CubeImpl: Invalidating cube and dependents for cube 'StargateCacheTest'.
15792   [2]   DEBUG   2014-01-29 00:37:09.591   TM1.Cube.Invalidation   TM1CubeImpl::InvalidateCube: Invalidating base cube 'StargateCacheTest'.

15792   [2]   DEBUG   2014-01-29 00:37:12.248   TM1.Cube.Stargate.Definition   
Cube: StargateCacheTest
Cube to View dimension map: 
   [0,1]
   [1,2]
Stargate Expanse: 
   Dimension: d1
      Axis Selections:
         [ (total,U)
         , (x,U)
         , (y,U)
         ]
   Dimension: d2
      Axis Selections:
         [ (b,U)
         , (a,U)
         , (c,U)
         , (d,U)
         ]

15792   [2]   DEBUG   2014-01-29 00:37:18.450   TM1.Cube.Stargate.ViewStorage   ViewStorage::AddStargate: Stargate 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) added to ViewStorage (0x000000000BF1BD58).
15792   [2]   DEBUG   2014-01-29 00:37:18.450   TM1.Cube.Stargate   New Stargate Created 'StargateCacheTest:( [d1].*, [d2].* )' (0x0000000004500010) in 6201ms SGSize=82Kb workPoolSize=0Kb hPoolSize=67Kb.

This also illustrates how woefully inadequate the default setting is. My toy view with 12 cells used more than half the allotted space...

[hyunjia]

This is absolutely very useful info, thanks so much TableManagerOne . Out of my curiosity, what’s the use of Sleep function here? Is it a hidden function from TM1?

[zodiacviv]

thanks every1 for your valuable reply. Really i got my most of queries solved. Now i created a TI Process and inside it i called ViewConstruct function for all 10 cube whose views are taking good time to open. Can any one help me how to check if my stargate views is created or not?? I have set vmt as 3 and vmm as 30 for a cube for which i am trying to create stargate. Still i am not able to find that stargate view is created or not. for reference i am attaching screen shots

100.PNG (17.34 KiB) Viewed 22484 times

101.PNG (12.61 KiB) Viewed 22484 times

[tomok]

Can any one help me how to check if my stargate views is created or not??

Did even take the time to read all the posts in this thread? If you set the logging properties as mentioned above, the creation of stargate views will be logged in the tm1server.log file.

[qml]

This is absolutely very useful info, thanks so much TableManagerOne . Out of my curiosity, what’s the use of Sleep function here? Is it a hidden function from TM1?

Seems to be an undocumented function that works in both TI and rules that waits (as the name suggests) for a given number of milliseconds. I must say that is quite a discovery for me. The fact that it works in rules blows my mind. TableManagerOne is using it to make sure that the small sample view takes long enough to refresh so that the TM1 engine decides to cache it. Simply brilliant.

[TableManagerOne]

and vmm as 30

Also, from further up the thread, 30 (30kb) is too low for any useful views.

[TableManagerOne]

I had heard of Sleep as a TI function, though, it isn't documented (any more?) I've used it in TI for synchronization and lock testing (e.g. perform some operation and sleep for 30 seconds to see if it blocks some other operation I run in another Architect window.)

However, I did come across it in the rules documentation "Rule Function Names":

Code: Select all

...
SIN
SLEEP
SQRT
...

Color me curious, I tried it out, and it worked. I can't imagine any practical uses beyond the hack I presented above though...

[hyunjia]

This is absolutely very useful info, thanks so much TableManagerOne . Out of my curiosity, what’s the use of Sleep function here? Is it a hidden function from TM1?

Seems to be an undocumented function that works in both TI and rules that waits (as the name suggests) for a given number of milliseconds. I must say that is quite a discovery for me. The fact that it works in rules blows my mind. TableManagerOne is using it to make sure that the small sample view takes long enough to refresh so that the TM1 engine decides to cache it. Simply brilliant.

Now , that was Ah-Ha moment for me tahnks qml . I agree with TableManagerOne , there are more than you would expected from this chocolate box called TM1

[PlanningDev]

One last thing, can someone confirm if a sever restart is required for VMM changes to take effect?

[TableManagerOne]

Sorry, I didn't make it clear above, but my test case, reflected in the log extract attached above, was in a single server session. I changed VMM on the fly, and it took effect immediately. No restart required.