The AWS Cloud no good, very bad performance

Post Reply
User avatar
PavoGa
MVP
Posts: 616
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.9.1
Excel Version: 2013 PAW
Location: Charleston, Tennessee

The AWS Cloud no good, very bad performance

Post by PavoGa »

We have just moved our development from on-prem servers to the AWS Cloud. The performance is, to be kind, lacking. The processes that move data from one slice of a cube to another are taking roughly double or more the amount of time to perform the same tasks. Running TIs side-by-side for the two environments and the Cloud is like a three legged mule racing a thoroughbred.

Architect (I use ARC, but we had some holdouts) is virtually unusable, as I think has been noted in this Forum. ARC seems to be fine. PAW seems to be fine and the installation went well. The AWS server has the same memory and cores as the on-prem. We use AWS as our development workstations.

If anyone else is using AWS cloud, were performance issues observed? Any particular setting(s) to check?
Ty
Cleveland, TN
tomok
MVP
Posts: 2831
Joined: Tue Feb 16, 2010 2:39 pm
OLAP Product: TM1, Palo
Version: Beginning of time thru 10.2
Excel Version: 2003-2007-2010-2013
Location: Atlanta, GA
Contact:

Re: The AWS Cloud no good, very bad performance

Post by tomok »

If the type and number of cores is the same I would suspect hard drive issues, i.e., SAN. Are you sure your drive space is in the same region as your server? Even in cube to cube copies TM1 is going to utilize drive space in the memory temp file.

As to Architect, at least from an Admin standpoint, can't you just use RDP to the TM1 server and run it from there? I have used AWS for other systems and RDP is how you get access to the server to do stuff (assuming you are on Windows). If this is Linux don't they have something similar.
Tom O'Kelley - Manager Finance Systems
American Tower
http://www.onlinecourtreservations.com/
User avatar
PavoGa
MVP
Posts: 616
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.9.1
Excel Version: 2013 PAW
Location: Charleston, Tennessee

Re: The AWS Cloud no good, very bad performance

Post by PavoGa »

tomok wrote: Mon May 09, 2022 4:07 pm If the type and number of cores is the same I would suspect hard drive issues, i.e., SAN. Are you sure your drive space is in the same region as your server? Even in cube to cube copies TM1 is going to utilize drive space in the memory temp file.

As to Architect, at least from an Admin standpoint, can't you just use RDP to the TM1 server and run it from there? I have used AWS for other systems and RDP is how you get access to the server to do stuff (assuming you are on Windows). If this is Linux don't they have something similar.
Thanks, Tom. I'll follow up on this with the server team. I'm new to a cloud installation, literally every install I've ever worked on was on-prem.


With regards to Architect, logging into the server with RDP is being taken away. We had it with the on-prem. :x
Ty
Cleveland, TN
User avatar
gtonkin
MVP
Posts: 1192
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: Latest and greatest
Excel Version: Office 365 64-bit
Location: JHB, South Africa
Contact:

Re: The AWS Cloud no good, very bad performance

Post by gtonkin »

Got a client looking to go this way too, so jumping in to keep an eye.

The one thing I always look to is how your connection is made. From your corporate network vs home (Fibre/4/5G)
Have found that corporate IT often throttle traffic they do not know and limp it into lead class, quality of service wise.

May be an idea to do some more tests from different networks. Another thing to keep in mind is different technologies utilise different MTU settings. Users on 4G often report better Perspectives performance than fibre peers.
User avatar
vovanenok
Posts: 88
Joined: Mon Jun 23, 2014 4:54 pm
OLAP Product: TM1
Version: 2.0.9
Excel Version: Office 365
Location: Toronto, Canada
Contact:

Re: The AWS Cloud no good, very bad performance

Post by vovanenok »

I use AWS and even with small instances (like t3.medium) had no issues. Though I've never compared vs on prem. I use RDP to access Architect and other stuff on the server.
Assuming you have the same or similar enough server specs, I would look into running some stress test software to compare your AWS vs on prem, which would tell you about the disk performance as well
I remember I used Prime95 for Windows a while ago when we switched from physical servers to VMs and I wanted to compare both.
----------
TeamOne Google Sheets add-on for IBM Planning Analytics
Let's connect on LinkedIn
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: The AWS Cloud no good, very bad performance

Post by lotsaram »

What type of AWS instance are you using?
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
User avatar
PavoGa
MVP
Posts: 616
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.9.1
Excel Version: 2013 PAW
Location: Charleston, Tennessee

Re: The AWS Cloud no good, very bad performance

Post by PavoGa »

lotsaram wrote: Tue May 10, 2022 7:47 am What type of AWS instance are you using?
I am not conversant on AWS instance types, but does x1e.2xlarge answer your question?
Ty
Cleveland, TN
User avatar
PavoGa
MVP
Posts: 616
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.9.1
Excel Version: 2013 PAW
Location: Charleston, Tennessee

Re: The AWS Cloud no good, very bad performance

Post by PavoGa »

gtonkin wrote: Mon May 09, 2022 5:46 pm Got a client looking to go this way too, so jumping in to keep an eye.

The one thing I always look to is how your connection is made. From your corporate network vs home (Fibre/4/5G)
Have found that corporate IT often throttle traffic they do not know and limp it into lead class, quality of service wise.

May be an idea to do some more tests from different networks. Another thing to keep in mind is different technologies utilise different MTU settings. Users on 4G often report better Perspectives performance than fibre peers.
The DEV connections are mostly made via corporate network in DEV. Prod is still on-prem and a lot of those users will be via home. I use an AWS instance from home, so I'm inside the corporate network.

I tested copying a 100MB file from an on-prem server using psh. To another on-prem server it took just over one minute (1:00.685) and copying the same file to the AWS server took over three (3:35.851). Once this file was loaded onto the servers, I ran a TI that did nothing more than read and count the lines. There was no significant difference On-Prem vs. AWS.

I then ran a TI reading approximately 23,750,000 numeric cells from a cube. Did nothing more than count the cells read. Cube was not populated identically in the two instances. Ran the process three times in each instance. Here there was a difference.

The AWS read 23,744,470 cells averaging 101.88 seconds and 235,567 records per second. Run times: 94.08, 117.28, 94.28.
The on-Prem read 23,796,470 cells averaging 70.74 seconds and 336,447 records per second. Run times: 71.67, 70.58, 69.96. Significantly faster.

Was actually surprised that reading a text file showed no significant difference.
Ty
Cleveland, TN
lotsaram
MVP
Posts: 3651
Joined: Fri Mar 13, 2009 11:14 am
OLAP Product: TableManager1
Version: PA 2.0.x
Excel Version: Office 365
Location: Switzerland

Re: The AWS Cloud no good, very bad performance

Post by lotsaram »

PavoGa wrote: Tue May 10, 2022 3:54 pm I am not conversant on AWS instance types, but does x1e.2xlarge answer your question?
Yes that's what I meant.
Please place all requests for help in a public thread. I will not answer PMs requesting assistance.
User avatar
PavoGa
MVP
Posts: 616
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.9.1
Excel Version: 2013 PAW
Location: Charleston, Tennessee

Re: The AWS Cloud no good, very bad performance

Post by PavoGa »

Further update:

Running a TI with a simple loop (1-15,000,000) takes about 45% longer, on average, on the AWS than the on-prem server.
Ty
Cleveland, TN
User avatar
Steve Rowe
Site Admin
Posts: 2410
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: TM1 v6,v7,v8,v9,v10,v11+PAW
Excel Version: Nearly all of them

Re: The AWS Cloud no good, very bad performance

Post by Steve Rowe »

The AWS server must have slower cores in it then or being throttled in some way?
Technical Director
www.infocat.co.uk
User avatar
PavoGa
MVP
Posts: 616
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.9.1
Excel Version: 2013 PAW
Location: Charleston, Tennessee

Re: The AWS Cloud no good, very bad performance

Post by PavoGa »

Steve Rowe wrote: Thu May 12, 2022 8:21 am The AWS server must have slower cores in it then or being throttled in some way?
I'm not on the team working on the infrastructure, but I have asked that question as I think George mentioned it earlier. I'm told they do not believe there is any throttling occurring.

One other thing: is the "not running virus protection on the DATA folder" still a thing? That is being checked on, but one of the team asked if it was and frankly, I have not seen anything on that recently that was yay or nay.
Ty
Cleveland, TN
User avatar
PavoGa
MVP
Posts: 616
Joined: Thu Apr 18, 2013 6:59 pm
OLAP Product: TM1
Version: 10.2.2 FP7, PA2.0.9.1
Excel Version: 2013 PAW
Location: Charleston, Tennessee

Re: The AWS Cloud no good, very bad performance

Post by PavoGa »

An update: pay attention to the processor speed. While the RAM and cores were equivalent, switching to another AWS with faster processors resolved the issues. Faster now than the on-prem server was. I was not directly involved with the setup, but did devise several tests measuring performance.

Thanks, George & Steve, for providing the ultimate hint. Thanks to everyone for their comments. Some did not pan out, but helped eliminate other potential issues. Your suggestions were passed along to the server team.
Ty
Cleveland, TN
User avatar
gtonkin
MVP
Posts: 1192
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: Latest and greatest
Excel Version: Office 365 64-bit
Location: JHB, South Africa
Contact:

Re: The AWS Cloud no good, very bad performance

Post by gtonkin »

Glad to hear you got it sorted and we all have learnt something through your school fees.
Post Reply