AsciiOutput Taking Too Long
Posted: Fri Apr 01, 2011 1:31 pm
I have a chore that runs 4 processes that do the following:
1) Export data for 1 year from a forecasting cube (calculated from 10+ other cubes) to a csv file.
2) Export data for 1 year from a forecasting cube containing headcount (calculated from same sources as above) to a csv file.
3) Import data from csv file created by #1 to a cube that essentially holds our Income Statement and Balance Sheet.
4) Import data from csv file created by #2 to a cube that holds headcount (just FTE totals).
#2 - #4 are relatively quick (42s, 4s, and 1s respectively), but #1 takes a whopping 10 mins! I know the issue is that the rules/feeders that make up this data need to be recalculated, but it just should not take this long. I left this company a year ago and am now back, so it seems like this length of time has occurred in the year since I've been gone because I don't recall it taking this long. I also know the underlying issue behind these recalcs taking so long is the rules/feeders themselves, because since I left the time the server takes to come up has gone from ~20 mins to 48 mins (!!!).
These copies are done once an hour and then are synced the star server. The one thing that puzzles me is that the copies can vary in length from hour to hour. One hour it will perform as described above, while the next hour the entire chore will run in 15s. But it's all still the same data - export all non-zero leaf-level data, zero-out target, copy in source. I could understand if it was the sync chore and it varied because of the number of transactions needed to process. But a straight data copy? My thought on this is that possibly a lack of updates means the stargate view doesn't have to be recalculated? My knowledge of stargate views is less than expert.
So I see two possible solutions to this:
1) The first is obvious to me: improve the rules and feeders. I need to do this anyway. However, I have little time right now and so I need solution #2 to come through for me;
2) There HAS to be a way to export the data in #1 without recalculating the affected data, reprocessing the rules/feeders, reassigning cube dependencies, etc.
Am I overlooking a config setting or some TI function that will get me what I want?
Thanks in advance for the help!
1) Export data for 1 year from a forecasting cube (calculated from 10+ other cubes) to a csv file.
2) Export data for 1 year from a forecasting cube containing headcount (calculated from same sources as above) to a csv file.
3) Import data from csv file created by #1 to a cube that essentially holds our Income Statement and Balance Sheet.
4) Import data from csv file created by #2 to a cube that holds headcount (just FTE totals).
#2 - #4 are relatively quick (42s, 4s, and 1s respectively), but #1 takes a whopping 10 mins! I know the issue is that the rules/feeders that make up this data need to be recalculated, but it just should not take this long. I left this company a year ago and am now back, so it seems like this length of time has occurred in the year since I've been gone because I don't recall it taking this long. I also know the underlying issue behind these recalcs taking so long is the rules/feeders themselves, because since I left the time the server takes to come up has gone from ~20 mins to 48 mins (!!!).
These copies are done once an hour and then are synced the star server. The one thing that puzzles me is that the copies can vary in length from hour to hour. One hour it will perform as described above, while the next hour the entire chore will run in 15s. But it's all still the same data - export all non-zero leaf-level data, zero-out target, copy in source. I could understand if it was the sync chore and it varied because of the number of transactions needed to process. But a straight data copy? My thought on this is that possibly a lack of updates means the stargate view doesn't have to be recalculated? My knowledge of stargate views is less than expert.
So I see two possible solutions to this:
1) The first is obvious to me: improve the rules and feeders. I need to do this anyway. However, I have little time right now and so I need solution #2 to come through for me;
2) There HAS to be a way to export the data in #1 without recalculating the affected data, reprocessing the rules/feeders, reassigning cube dependencies, etc.
Am I overlooking a config setting or some TI function that will get me what I want?
Thanks in advance for the help!