9.1 SP3 Instability Issue

[jim wood]

Guys,

We have recently upgraded both our server operating system and TM1 version. The server was upgraded from Solaris 8 to 10 and TM1 was upgraded from 8.4.5 to 9.1 SP3.

Before the upgrade the server was solid as a rosk. We didn't have a crash in 18 months. Since the upgrade we have been getting them every week. Sometimes we have had multiple crashes in a week.

As far as I am aware I have taken steps to avoid problems with concurrent schedules / processes and a known issue hasn't caused a crash (As far as I am aware).

I was wondering if anybody else has experienced a problem?

It mostly seems to happen when somebody is browsing our larger cubes,

Jim.

[Steve Vincent]

It mostly seems to happen when somebody is browsing our larger cubes,

How much RAM is it using? Think i saw somewhere that 9.1 used more RAM than previous versions, if you were tight on RAM before maybe it's now tipping over the edge? Not used TM1 on *nix but are there any machine error messages that might help to explain the failures?

[Steve Rowe]

Any progress on this Jim?
If you having a 'mare with stability then your best bet is to downgrade to the latest 9.0 before they "improved" the object locking....
Cheers,

[jim wood]

Hi Guys,

I've been unwell so sorry for the delay. To answer your questions:

1) I have removed everything that I can and memory usage has gone down from 34GB to 28GB. Also I have reduced the startup time from 6 hours to 2.5. We have 64GB available so lack of memory aint the issue. To get teh startup time right I have removed some variable feeders so that aint the issue.

2) Downgrading might be an option. It will be a mare for me as changing desktop version in my company can be a little difficult to say the least. (IT and their froms etc...) I have taken the stpe of removing the laregst cube from public view. This essm to have helped as I don't think their has been a crash since.

A Core dump of memory has been taken and I working Cognos to find out what the issue is. I am hoping this will lead to us upgrading rather than downgrading. (The anti-locking feature has lead to some real positive feedback from the users.)

Jim.

[jim wood]

Cognos have got back to me. They think it is related to users passing the allowed memory usage limit for data views. They have asked me to put the level up while they look in to why this is causing a crash,

Jim.

[Steve Vincent]

Jim, what exactly do they mean? Are they talking about MaximumViewSize=100?

The help file says;

MaximumViewSize Optional Sets the maximum amount of memory (in MB) to be allocated when accessing a view. If no parameter is set, the default maximum view size is 100MB on a 32-bit system, and 500 MB on a 64-bit system.To specify a maximum amount of memory allocation for views, add the following line to Tm1s.cfg:MaximumViewSize=n where n represents the amount of memory in MB to be allocated.You must manually add the MaximumViewSize parameter to Tm1s.cfg. The parameter is not part of a standard file created when you install a TM1 server.

That's a little vague, but the way i read it was it's there to protect the server, so if anyone picks a silly view (ie. all zero levels) it will try until it reaches that limit then tell the user it can't show the view. If you increase that, surely all it's going to do is put the server more at risk, or maybe it's not working in your version and that's causing it?

[jim wood]

Hi Steve,

Apologies if I was a little unclear. I was talking about MaximumViewSize=100. The difference is ours was set 50. This was inherited from when we were on a windows machine.

We have now set it to 512 and hope to see more stability. This is there to protect the server and Cognos have admitted that reaching this limit should not crash the server. You have to bare in mind that this is their attempt at an answer. I'm sure we will find other problems as well.

Another interesting thought. I have posted on here complaining about the increase in my weekend load shcedule from 6 hours to over 13. Within the schedule there are large data copies from sandbox cubes to reporing cubes. Maybe our low view size limit casued this delay as well? Kind of makes sense. I guess I'll soon find out on Sunday,

Jim.

[ScottW]

If you are experiencing a blowout in processing times try setting ProgressMessage=F in the config file.
For a fresh 9.1.3.x installation this is set by default but is easily overlooked if you have upgraded. I'm not sure of the technical reasons why but Cognos (or Applix as it was then) engineering found during stress testing somewhere on the transition between 9.1.2.3 and 9.1.3 while the new locking model was being fine tuned that this setting vastly decreased processing times.

From a user perspective this setting disables the cube view processing dialog so the user doesn't have the option to cancel if a large view is opened and they don't want to wait. From an admin perspective the record counter when running a TI manually is also disabled (this isn't too bad to live without). Takeout, make sure the MaximumViewSize parameter is also set to something sensible.

The section from the operations guide is copied below.
ProgressMessage

This parameter determines whether users will have the option to cancel lengthy view calculations.
When a user opens a view that takes a significant amount of time to calculate (usually a view with high levels of consolidation or complex rules), the Cube View Processing dialog box opens.
If a user clicks Stop Building View, the view is discarded on the client end, but view calculation continues on the server. In some instances, this can lead to a server hang.
You can prevent the Cube View Processing dialog box from appearing (and prevent users from cancelling view calculation) by adding the line
ProgressMessage=F
to the Tm1s.cfg file for your server.
When the ProgressMessage parameter is set to T or is not present in the Tm1s.cfg file, the Cube View Processing dialog box opens during lengthy view calculations.
NOTE: In TM1 9.1 SP3 and later, ProgressMessage=F is inserted into the Tm1s.cfg file during server installation.

[jim wood]

Cheers Scott,

I will probably just change the maxiumu view setting for now and see what happens. If we still experience problems I will add in what you have suggested. We have some large cubes and sometimes th users can be a bit daft. They need the cancel button. Also we didn't upgrade, we went for a clean install of 9.1 SP3,

Jim.

[jim wood]

Guys,

We are stil experiencing the crashes even with the higher maximum view size. We have sent yet another stack trace to Cognos. Deep joy,

Jim.

[Ken Vuong]

Hi Jim,

Can you update us when you get something\anything back from Cognos, as we are due to move to 9.1 SP3 64bit (as the parent company has it) and this sounds quite worrying indeed.

I was recent warned of 9.1 SP3 instabilities by the great man himself, Mr Usherwood, but is does appear that we have to take the plunge whether we like it or not.
Also we have specified 16GB of ram for the new 64 Bit server, but after reading your posts, we may have to double that

Bring back 8.4.5 I say!

Cheers,
Ken Vuong
Equity Group

[rossi]

I will second the that. 8.4.5 was far more stable. One has to question whether to move to 9.1 really. It has become a very serious question as to whether they can get it stable at all quite frankly.
Paolo

[jim wood]

Hi Jim,

Can you update us when you get something\anything back from Cognos, as we are due to move to 9.1 SP3 64bit (as the parent company has it) and this sounds quite worrying indeed.

I was recent warned of 9.1 SP3 instabilities by the great man himself, Mr Usherwood, but is does appear that we have to take the plunge whether we like it or not.
Also we have specified 16GB of ram for the new 64 Bit server, but after reading your posts, we may have to double that

Bring back 8.4.5 I say!

Cheers,
Ken Vuong
Equity Group

Hi Ken,

I feel your pain. On the memory side, we have very large models, hence the need for lots of RAM. The general rule is (As I understand it) that you double your 32-bit memory usage. I would probably double it and add a bit just to be sure. Make sure you leave enough room for future development.

As for 8.4.5. We went over 8 months without a single crash. The issue we have is that as a business we have moved to Solaris 10. 8.4.5 isn't supported on Solaris 10. I am stuck with 9.1!

I will post on here any developments as they happen. (I am keeping the faith. Just.)

Jim.

[David Usherwood]

Flattery will get you anywhere, Ken...

Jim, why don't you go for 9.0SP3U4? I have no direct experience with this on Unix, but on Wintel, it's been fine for us and our clients. I have yet to see a 'must have' in the 9.1 codebase that means you have to 'up'-grade.
The worry I have is that Cognos consultants and support staff will only know 9.1+ and assume the users are all on that too, so there's a communication gap opening up between them and the real world.

[Eric]

When were Action Buttons introduced? I guess more importantly re they in 9.0SP3U4?

[Herman Moller]

Eric,

Action Buttons was introduced in 9.1 and with some other interesting functionality.....

[jim wood]

When was the anti locking model introduced?

[John Hobson]

AFAIK the locking model changed to an object wide scope (as opposed to a server wide scope) in 9.1 (although it might have been 9.0) and it sounds like they may still be tweaking it.

Someone will be along to correct me shortly

J

[David Usherwood]

Eric - action buttons came in in 9.1, so if you _must_ have them then you're stuffed.
Probably worth considering what you need them for. If you don't need to use them in TM1Web, then you can work round them with VBA etc.

[jim wood]

The anti-lockiong model is seen by the business as a real improvement. During the budgeting cycle data is being loaded in all the time. Having the server lock while this is happening has in the past caused a large amount of frustration for all users,

Jim.