Help: sizing data base on RAM on server

[harrytm1]

Hi all,

A customer asked me this question today and I was stumped. Now, this customer seems to be quite tight on budget and is trying hard to reduce hardware cost for a TM1 implementation.

So he asked if the server is to be equipped with 4GB RAM, how much data can the TM1 server load into memory? He's willing to sacrifice in terms of performance.

Correct me if I'm wrong here, but this is not an issue with performance. Rather, if there is insufficient RAM to even build the dimensions, cubes, rules and feeders during startup, then the TM1 application will be out of memory. Also, it greatly depends on the overall model. Hence, it is a rather difficult question to answer.

Can i assume that, for a 4GB RAM, 50% will go into loading data while the rest can be used for rules, feeders, structures etc?

I once came across a post in this forum that talks about the number of bytes of data per GB of RAM. Can someone advice how I should size up the data size for a hypothetical environment with 4GB RAM and NO rules and feeders? I will probably add in a safety factor to be on the safe side.

Another question is, say if a database on Oracle that is currently 100GB is to be migrated into TM1 cubes, what is the ratio to apply to estimate the total file size of all files in the TM1 Data folder? 10:1 to yield 10GB of Tm1 files?

Many thanks!

[tomok]

Download and read this http://www.ibm.com/developerworks/data/ ... ge205.html

[mattgoff]

I hope I don't sound flippant, but if the client is so price-sensitive as to want to avoid spending a few $k extra to get a 64-bit server and sufficient RAM, why is he considering TM1 in the first place? Seems like you might want to try to steer him towards Palo instead. The license/support cost differences there FAR outweigh any hardware cost considerations. A 32-bit TM1 server is plenty powerful, but if TM1 runs out of memory at any time, things will go badly. If there's a question that he'll exceed it, it's risky to proceed.

In terms of how much data TM1 can hold given a quantity of RAM, that's 100% model dependent. If it's a pure load model with no calculations and feeders, RAM size vs source data size is very modest. If there are lots of calcs and feeders, it can be an unlimited multiplier of the source data size.

For reference, I have a middle-of-the-road model in terms of loaded vs. calculated data. It's around 1GB uncompressed on disk, including data (cub), model files (dim, pro, rux, cho, etc), and user data (views, subsets). Loaded, the model is around 5GB in RAM at launch but can climb to 8-9 GB if it's been running for a month or so (I think there's a problem with ODBC that causing a lot of this creep).

Matt

[stephen waters]

Hi all,

A customer asked me this question today and I was stumped. Now, this customer seems to be quite tight on budget and is trying hard to reduce hardware cost for a TM1 implementation.
So he asked if the server is to be equipped with 4GB RAM, how much data can the TM1 server load into memory? He's willing to sacrifice in terms of performance.

The laptop I bought my son for university had 4 Gb Ram (and cost about £400!)

I assume customer will be buying Cognos Express, which has to run on 64 bit anyway? You could get a 2x4 core 2.5 GHz, 16 Gb server for about £2,000 from Dell online ( before any corporate discount). If he wants to economise, do so on the processor.

But really, I agree with Matt, if he is economising that much is he the right customer to be able to use TM1? Have you actually educated him about how TM1 works? If he skimps on RAm and you then have to work within that limitation it will probably mean lots more work, OK for you if you are charging the customer but more expensive for him\her.

[lotsaram]

Correct me if I'm wrong here, but this is not an issue with performance. Rather, if there is insufficient RAM to even build the dimensions, cubes, rules and feeders during startup, then the TM1 application will be out of memory. Also, it greatly depends on the overall model. Hence, it is a rather difficult question to answer.

Can i assume that, for a 4GB RAM, 50% will go into loading data while the rest can be used for rules, feeders, structures etc?

I once came across a post in this forum that talks about the number of bytes of data per GB of RAM. Can someone advice how I should size up the data size for a hypothetical environment with 4GB RAM and NO rules and feeders? I will probably add in a safety factor to be on the safe side.

Another question is, say if a database on Oracle that is currently 100GB is to be migrated into TM1 cubes, what is the ratio to apply to estimate the total file size of all files in the TM1 Data folder? 10:1 to yield 10GB of Tm1 files?

Harry you're both right and in some instances very wrong.

As you know (and as you should tell your customer) TM1 is an in-memory database. therefore the amount of RAM required had nothing to do with performance and everything to do with database size. Performance will depend on number of processors, processor clock speed, number of concurrent users and model design. If there is insufficient RAM then performance is a moot question because the application will simply fail to load if there is not enough RAM.

However, Harry you simply cannot make any assumptions like 50% of RAM is for dimensions and cube data and 50% is for calculations or memory in TM1 compared to relational database. The way that data is structured in a TM1 cube versus a relational table is very different. The only way to even moderately accurately do this is to go back to first principles and calculate the number of numeric data points (leaf level cells in a TM1 cube) that will hold data. Note the distinction of cells actually holding data as opposed to number of cells theoretically in a cube. IBM do have a server sizing spreadsheet available, I can't share it but if you work for a partner you should be able to get hold of it. However many assumptions are going to be quite arbitrary anyway. The core calculation is quite simple being data points x bytes per point. For x64 the number of bytes per leaf cell is of the order of 16 - 40 bytes and for a fed cell 4 - 20 bytes. The reason this is not precise and there is such a large range is that TM1 cube size is very sensitive to optimal dimension ordering in the cube. (That is a whole other topic which you can find many threads on.)

Doing this calculation will give you an approximation of memory in RAM when the database first loads. However as Matt has already said memory will grow as TM1 caches rule calculations, consolidations and views. The order of magnitude of this additional memory could be small compared to the data model or could be many times larger, it all depends on the application and its usage and the frequency of data updates.

In addition to all that as everyone else has hinted, the cost of hardware is as a rule much less than the cost of the actual software and the effort to design and build a system and train users. Skimping on memory is quite simply looking in the wrong place to save money.

[Alan Kirk]

Harry you're both right and in some instances very wrong.

As you know (and as you should tell your customer) TM1 is an in-memory database. therefore the amount of RAM required had nothing to do with performance and everything to do with database size. Performance will depend on number of processors, processor clock speed, number of concurrent users and model design. If there is insufficient RAM then performance is a moot question because the application will simply fail to load if there is not enough RAM.

I'm not sure it's quite that straightforward. On a 32 bit system it will fail to load if it can't fit within the memory space available to a single application (2 or 3 gig, as the case may be depending on the Windows configuration). Helooo, 9.1, in my case. However even in 32 bit if you had 1 gig of physical RAM and loaded a 1.5 gig model on it, I don't think that TM1 would be aware of the shortage of physical RAM since the difference would be made up via the Virtual Memory swap file. As far as I know most applications don't know whether they're getting physical or virtual memory. And there you would get performance degradation since you'd have constant swapping out to the pagefile, which is much slower than physical memory.

With Windows 32 bit the VM is limited to 4 gigabytes but on 64 bit it's 16 terabytes. Unless I'm wrong and TM1 really is somehow aware of the difference between physical and virtual memory and refuses to load unless it has enough of the former, I can see models getting out of bed on a system which lacks adequate physical memory but then running like a 3 legged dog because the machine is constantly going to its hard disks to make up the shortfall. And it's for that reason that...

In addition to all that as everyone else has hinted, the cost of hardware is as a rule much less than the cost of the actual software and the effort to design and build a system and train users. Skimping on memory is quite simply looking in the wrong place to save money.

... I couldn't agree more. Hardware is relatively cheap these days, and effectively cheaper for businesses who can claim the costs to reduce tax. Shove as much memory as you can in the direction of the application and you won't be limited in the future.

[harrytm1]

hi all,

thank you so much for your prompt replies. I cannot agree with you more on whether this is the right client. however, since these questions were posted to me, i'm obligated professionally to revert. I absolutely agree that they are trying to save in the wrong area i.e. hardware given the prices of server and RAM now.

I'm also hesitant to give a definite answer and end up developing a TM1 application on a low configuration server. This will certainly handicap us and any future enhancement will be limited.

Anyway, I will reply as politically correct as possible. Thank you all once again!

[stephen waters]

hi all,

however, since these questions were posted to me, i'm obligated professionally to revert. ... I will reply as politically correct as possible. Thank you all once again!

Harry, maybe your professional obligation should be to explain that you are out of your depth technically, do not have enough experience of TM1 and are unable to explain the issues. Then recommend they consult one of the long standing partners (quiet a few of whom give advice on this forum) who can advise them properly!