Too many subsets

[BariAbdul]

I have recently came across a model which has 7 dimensions but 50,000 odd subsets tied to these 7 dimensions,I know the best practice is to create and destroy subsets run time within a TI process.Does it impact cube performance?What should things I should look in to before getting rid off them, Do someone else come across like this scenerio.Thanks

[Alan Kirk]

I have recently came across a model which has 7 dimensions but 50,000 odd subsets tied to these 7 dimensions,I know the best practice is to create and destroy subsets run time within a TI process.Does it impact cube performance?What should things I should look in to before getting rid off them, Do someone else come across like this scenerio.Thanks

Unfortunately yes. I inherited some Bedrock-based models that hadn't been clearing the temporary subsets after use, and there was a LOT of them. I'm assuming that you're referring to public subsets.

I'm not sure that performance would be hugely impacted if you have a 64 bit server with plenty of RAM overhead. But it will increase RAM usage and probably load time (perhaps not materially, but some). But it will clutter the living crud out of the GUI, making subset selection a pain in the nethers to the users.

Much will depend on what the naming pattern of the subsets is (if there is one; there usually is) and how confident you are that they won't be used in views.

If they are likely to be used in views you should delete the views first. And bear in mind that there is the danger that some users have used some of the subsets in private views too. If the subset naming pattern is reasonably consistent you could probably do a quick check of that by having Notepad ++ do a search for the common part of the subset name in all .vue files in the data directory.

The best case scenario is when you're reasonably comfortable that (a) they aren't going to be used in views (or at least used so infrequently that you can wear the occasional "Error creating view array" message) and (b) the subsets have a reasonably consistent naming pattern. If that's the case then all you'd need to do is shut down the server, back up the database directory, do a search for all files which have the naming pattern and an extension of .sub, then delete them. Restart the server, and they're gone. If anything goes too wrong you can always recover the deleted .subs from the backup.

Alternatively if the subset names are very, very regular (something like zTemp_0001, zTemp_0002, etc), you can just have TI do a loop and attempt to delete the subsets. If they're not used in any views the deletion will be successful, if they are it'll generate a minor error.

[BariAbdul]

Thanks a lot for elaborate response Alan.

[hyunjia]

I have the same issue with the Bedrock process that convert dynamic subset to static one , need to add a statement to destory the subset once the txt file is genereated

I have recently came across a model which has 7 dimensions but 50,000 odd subsets tied to these 7 dimensions,I know the best practice is to create and destroy subsets run time within a TI process.Does it impact cube performance?What should things I should look in to before getting rid off them, Do someone else come across like this scenerio.Thanks

Unfortunately yes. I inherited some Bedrock-based models that hadn't been clearing the temporary subsets after use, and there was a LOT of them. I'm assuming that you're referring to public subsets.

I'm not sure that performance would be hugely impacted if you have a 64 bit server with plenty of RAM overhead. But it will increase RAM usage and probably load time (perhaps not materially, but some). But it will clutter the living crud out of the GUI, making subset selection a pain in the nethers to the users.

Much will depend on what the naming pattern of the subsets is (if there is one; there usually is) and how confident you are that they won't be used in views.

If they are likely to be used in views you should delete the views first. And bear in mind that there is the danger that some users have used some of the subsets in private views too. If the subset naming pattern is reasonably consistent you could probably do a quick check of that by having Notepad ++ do a search for the common part of the subset name in all .vue files in the data directory.

The best case scenario is when you're reasonably comfortable that (a) they aren't going to be used in views (or at least used so infrequently that you can wear the occasional "Error creating view array" message) and (b) the subsets have a reasonably consistent naming pattern. If that's the case then all you'd need to do is shut down the server, back up the database directory, do a search for all files which have the naming pattern and an extension of .sub, then delete them. Restart the server, and they're gone. If anything goes too wrong you can always recover the deleted .subs from the backup.

Alternatively if the subset names are very, very regular (something like zTemp_0001, zTemp_0002, etc), you can just have TI do a loop and attempt to delete the subsets. If they're not used in any views the deletion will be successful, if they are it'll generate a minor error.

[nick_leeson]

I believe Bedrock.Fork.Dim.Sub.Delete does that for you.

[lotsaram]

I believe Bedrock.Fork.Dim.Sub.Delete does that for you.

Actually I think it is just Bedrock.Dim.Sub.Delete which allows for a delimited list of dimension names and subset names (with string match and wildcard). If there is a naming convention for temp objects (which you would seriously hope that there is!) then it is just a matter of scheduling this process in a chore and matching the subset name parameter to the temp object prefix. There is also a Bedrock.Cube.View.Delete and Bedrock.Cube.ViewAndSubsets.Delete. Obviously if temp subsets are being used by temp views then the views need to be deleted first.

Bedrock v1 came out when 9.4 was the latest and greatest TM1 version. At this time system wide locks due to metadata changes, particularly object creation and destruction were still quite an issue. The decision was made that performance during run-time execution was the most important thing and so object creation and destruction was avoided where possible to minimize the possibility of lock generation. So in v1 Bedrock processes only create views and subsets after checking first if they exist and don't destroy afterwards (relying instead on a daily or whatever other frequency of the "tidy up" processes to do this.)

As the TM1 server became better at avoiding locks due to view & subset metadata changes and the Bedrock library was improved this changed with subsequent releases. In Bedrock v3 processes have a parameter to retain or destroy temp objects with the default value being to destroy and tidy up automatically. So it makes sense to be using the latest Bedrock release. The Bedrock releases are backwards compatible as old parameter names are preserved with just new params introduced. Executing processes internally with the TI function has the nice feature that you only need to reference parameters when passing a non-default value. HOWEVER if bedrock processes are integrated with API or 3rd party (using API) which requires each parameter to be explicitly listed in the correct order and a value passed then obviously you do need to be very careful when upgrading.

[Alan Kirk]

As the TM1 server became better at avoiding locks due to view & subset metadata changes and the Bedrock library was improved this changed with subsequent releases. In Bedrock v3

Who in the what now? I'm looking at the page right now and "DOWNLOAD THE LATEST RELEASE" is still "BEDROCK TI VERSION 2.0 ZIP". Was that just a typo and you meant v2 (which does also have the arguments of which you speak), or is there actually an un-generally-released v3 out there somewhere?

[lotsaram]

There certainly is a v3 which has been around for some months and I had assumed was public. Oops my bad. I can only assume that it will be in the near future, could be the main developers have been a bit distracted with conference presentations, doing a javerized version of the library, and the occasional bit of paid work and neglected to put v3 up for download. Anyways you know who to contact if you want a copy.