Server Upgrade Performance Degradation

[shorsted]

Hi

We are in the process of upgrading our Dev and Test servers from Server 2008 R2 Standard to the Enterprise edition, plus allocating more RAM (from 32gb to 128gb) Both are virtual servers. We are running on Tm1 9.5.2 (upgrading next year finally!)

The upgrade on dev has been completed, however there is noticeable performance degradation on the server - the startup times have increased from 197 seconds to 11000-17000 seconds, although I did get one restart of 397 seconds, following a restart. We do use Persistent Feeders so if they do not load correctly then I would expect the start up times to be much longer however even with PF loading, the last startup was still in excess of 17000 seconds. We are due to start the upgrade on Test tomorrow, however I am loathe to do this due to this detrimental impact

I'm not aware of any reconfiguration I needed to do for this upgrade. We did have to change the cfg file as it was just using Localhost in the adminhost but we were having problems connecting until I changed it to the IP address. We do run 2 instances of Tm1 on this server, I am only looking at the largest and have disabled the other whilst I'm testing this

Can anyone suggest anything that might be causing this or that I might need to change? There is nothing else running on the server aside from standard server applications

Thanks
Sarah

[shorsted]

It looks like there is a windows updates service running which is utilising most of the CPU and slowing things down. Hopefully this will eventually complete and we will then be running ok!

[declanr]

There must be one hell of an update required if its using all of your CPU.

Are the servers like for like in terms of processors?
The newer versions are much better at utilising cores so I would look at adding more if possible.

[shorsted]

Yes the dev and test servers are like for like on processors. We've disabled the windows update service but are still getting really extreme start up times on the dev server. I'm investigating further but it looks like an automatic startup with a server restart is fairly quick (440 seconds) but if I try and manually restart the service without a full server restart then we are in excess of 10,000 seconds (and that's with persistent feeders loading the feeders from file!)

[lotsaram]

Yes the dev and test servers are like for like on processors. We've disabled the windows update service but are still getting really extreme start up times on the dev server. I'm investigating further but it looks like an automatic startup with a server restart is fairly quick (440 seconds) but if I try and manually restart the service without a full server restart then we are in excess of 10,000 seconds (and that's with persistent feeders loading the feeders from file!)

I have often seen the opposite, i.e. when a TM1 service is set to automatically restart it can be very very slow if it comes up before the server is fully up with all background services & OS prerequisites running; probably due to competition for resources. If this happens then configuring the TM1 service for a delayed start fixes the problem.

What you are experiencing sounds like TM1 is being crippled by something like a virus scanning process. Make sure that antivirus is completely disabled and try. If it is back to normal speed then you have your answer and know where you need to look, then you can see what needs to be done to configure the antivirus scanner exceptions.

[shorsted]

It does look like the antivirus software is kicking in and using lots of resource.

Will check that out, thanks

[moby91]

Yes the dev and test servers are like for like on processors. We've disabled the windows update service but are still getting really extreme start up times on the dev server. I'm investigating further but it looks like an automatic startup with a server restart is fairly quick (440 seconds) but if I try and manually restart the service without a full server restart then we are in excess of 10,000 seconds (and that's with persistent feeders loading the feeders from file!)

What you are experiencing sounds like TM1 is being crippled by something like a virus scanning process. Make sure that antivirus is completely disabled and try. If it is back to normal speed then you have your answer and know where you need to look, then you can see what needs to be done to configure the antivirus scanner exceptions.

What happens when you add the directories and files used by TM1 to the whitelist of your virus scanner as outlined by the IBM Technotes 1454290 and 1458762 ?


(1)
http://www-01.ibm.com/support/docview.w ... wg21454290
Slow TM1 Performance

Technote (troubleshooting)

Problem(Abstract)
This document provides a number of suggestions on how to address some TM1 performance issues that may arise, and offers possible explanations of root cause.


Resolving the problem

Suggestions to improve performance

3. Make sure that the virus scan software is skipping the TM1 files. Specifically, skip the TM1 data directory/logging directory, TM1 install directory, TM1Web directory and the directory where sheets are stored that are uploaded to TM1 as a reference. See Virus Scan technote link below for more detail.


(2)
http://www-01.ibm.com/support/docview.w ... wg21458762
TM1 and virus scan software

Technote (FAQ)

Question
What TM1 files should virus scan software be set to avoid scanning?

[shorsted]

Hi

Thanks for the help, unfortunately that anti virus exceptions have not resolved the issue. I've asked IT to produce some logs of the activity during startup times now so that we can analyse it.

We are wondering whether we need to reinstall TM1 as the OS was upgraded without a complete rebuild. I'm also going to try logging with IBM and see if they can offer anything further