Multi-threaded data save

[lotsaram]

We have had multi-threaded server loading for some time now and can now also multi-thread data loads. Together with 64 bit servers this has facilitated a huge expansion with the volume of data TM1 can handle.

However this all comes unstuck when a data save is executed following batch updates. As the data save is single threaded it can take a considerable amount of time to save data in a very large model.

My suggestion is to allow saving of .cub files to be multi-threaded in much the same way that loading of cubes is multi-threaded. Possibly with an additional config file parameter or possibly using the same settings as MaximumCubeLoadThreads.

I don't see why this shouldn't be possible ...

[jstrygner]

We have had multi-threaded server loading for some time now and can now also multi-thread data loads.

Multi-threaded server loading - yes, we have it, but AFAIK only for cubes (mentioned MaximumCubeLoadThreads) parameter. But what do you mean by multi-thread data loads? TI loading data to a cube (which I think always uses only one core), or you are still talking about data in cubes being load to RAM on server startup?

I don't see why this shouldn't be possible ...

Agree with your enhancement and that it should be possible.

But maybe somebody could answer, why multi-threading (using more than one core) is not available by:
1. loading dimensions to RAM on server startup,
2. running TI processes,
3. building views,
4. other situations where you could make use of all cores you have, not just one being fully engaged?

Is it historical cause and a change is fundamental (not easy to implement) or maybe there is some kind of best practice in such solutions with advantages I am just not aware of?

[lotsaram]

Yes multi-threaded server loading is for data only. Metadata is single threaded.

But no I was also talking about data loading no just server loading. This is getting off topic of the enhancement to enable multi-threaded data saves but in 9.4 and 9.5 at least it is very easy to do multi-threaded data loads. Multiple TI processes can run at once each loading to a separate cube. It is also possible to load multiple (non-overlapping) data sets to the same cube at the same time via multiple TI processes provided the server is in batch update mode.

[mattgoff]

We have had multi-threaded server loading for some time now and can now also multi-thread data loads. Together with 64 bit servers this has facilitated a huge expansion with the volume of data TM1 can handle.

However this all comes unstuck when a data save is executed following batch updates. As the data save is single threaded it can take a considerable amount of time to save data in a very large model.

My suggestion is to allow saving of .cub files to be multi-threaded in much the same way that loading of cubes is multi-threaded. Possibly with an additional config file parameter or possibly using the same settings as MaximumCubeLoadThreads.

I don't see why this shouldn't be possible ...

The bottleneck in loading is computing rules and firing feeders, so enabling CPU multithreading increases capacity. The bottleneck in saving is the disk subsystem, at least on my system, so adding compute capacity will not improve performance.

Matt

[lotsaram]

The bottleneck in saving is the disk subsystem, at least on my system, so adding compute capacity will not improve performance.

Duh - Oh, Houston we have a problem. Poor old hard disk head can only do so many things, should have thought of that. Hmm maybe another take on things would be to enable a background save data mode that would not impose a server lock! I don't see why that should not be outside the realms of possibility ...

[ParisHilton]

Or buy one of these funky new solid state disk drives !

The disk system is normally going to be the bottleneck, but it must be possible to write to disks in parallel.