Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post Reply
redflame2012
Posts: 5
Joined: Wed Sep 25, 2019 7:31 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2016

Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by redflame2012 » Tue Jul 14, 2020 11:03 am

I encountered some weird problems when running TM1 10.2.2 on Vmware virtual machine running Windows Server 2012R2 Stanrdard.
I describe the following error messages in chronological order:

First some Windows system events log reported 299 error:
Unable to start a DCOM server: {73E709EA-5D93-4B2E-BBB0-99B7938DA9E4}. The error: "299" Happened while starting this command C:\Windows\system32\wbem\wmiprvse.exe -Embedding

I started to notice that I cannot connect to TM1 server via remote desktop. But I can still connect to TM1 service via Architect from another machine.

Then in the tm1server.log:
TM1.System.File TM1 Server Abort: sf_Write() - Write error on file "d:\servers\excegroup\data\F110202_xxxxx.feeders$". Error = error code:8 reason:"Not enough memory resources are available to process this command" buffer = 0x0000000A95E998E8, len= 1022

Then another Windows system event log error happened,this time error code 1450:
Unable to start a DCOM server: {73E709EA-5D93-4B2E-BBB0-99B7938DA9E4}. The error: "1450" Happened while starting this command C:\Windows\system32\wbem\wmiprvse.exe -Embedding

Then in the tm1server.log:
TM1.Process Process "xxxx": : Execution was aborted. Error file: <TM1ProcessError_20200706015551_64805444_23_00 xxxx.log> : Only part of a ReadProcessMemory or WriteProcessMemory request was completed

At this time, the Architect client lost connection with TM1 service.

Finally, another Windows system event log error happened,this time error code 1453:
svchost (2348) 系统错误 1453 (0x000005ad): "Insufficient quota to complete the requested service" on "C:\Windows\system32\LogFiles\Sum\SystemIdentity.mdb"

I looked up these Windoes system error codes:
ERROR_NOT_ENOUGH_MEMORY
8 (0x8)
Not enough memory resources are available to process this command.

ERROR_PARTIAL_COPY
299 (0x12B)
Only part of a ReadProcessMemory or WriteProcessMemory request was completed.

ERROR_NO_SYSTEM_RESOURCES
1450 (0x5AA)
Insufficient system resources exist to complete the requested service.

ERROR_WORKING_SET_QUOTA
1453 (0x5AD)
Insufficient quota to complete the requested service.

They basically say system resources are exhausted. But when I checked the memory, the available memory is 200+ GB.
What was happening that caused the these errors?

Many thanks to any suggestions and expertise.

HighKeys
Posts: 97
Joined: Fri Aug 09, 2019 10:11 am
OLAP Product: TM1 / TM1 Web / Perspectives
Version: Planning Analytics V2.0.9
Excel Version: Plus 2013

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by HighKeys » Wed Jul 15, 2020 1:09 pm

Hi,

200gb space on Harddrive or in RAM? :D

I think the error is from the RAM not from the discspace.


BR

redflame2012
Posts: 5
Joined: Wed Sep 25, 2019 7:31 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2016

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by redflame2012 » Fri Jul 17, 2020 8:20 am

It is 200GB of physical memory, when these error happened, the working set of tm1s.exe is about 50GB, so 200GB of physical is still availale.
This is why these errors confused me.

burnstripe
Posts: 7
Joined: Wed May 06, 2020 2:58 pm
OLAP Product: Planning Analytics
Version: 2.0.9
Excel Version: 2016

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by burnstripe » Fri Jul 17, 2020 7:56 pm

Did the host of the VM run out of memory? Does the VM have dedicated Ram or is it set to only take what it needs. Is the 200GB Ram for the VM made purely from physical ram or is part of it using disk space via paging? The system could have been using paging and then ran out of disk space?

redflame2012
Posts: 5
Joined: Wed Sep 25, 2019 7:31 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2016

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by redflame2012 » Sat Jul 18, 2020 12:10 pm

burnstripe. Thanks for asking.

The ESXi 6.5 host still has 33G of physical memory available when the problems happend. And the host only has one single VM which is the VM that runs TM1 server. The host has 256G of physical memory, and it allocates all physical memory to this single VM.

About the disk angle, the VM has two disks, C: still has 44GB free, D: still has 84GB free when the problem happened.

User avatar
Steve Rowe
Site Admin
Posts: 2083
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: 10.2.2., PAW
Excel Version: Nearly all of them

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by Steve Rowe » Mon Jul 20, 2020 8:00 am

Just in case, consider that the TM1 DB will probably have shut down after generating these messages so if you looked at RAM consumption after the messages you won't have seen the impact of the TM1 DB.

Are you sure the TM1 DB was running when you checked the resource levels?

Most likely explanation is that TM1 DB consumed all RAM and shut itself down, so you need to understand why that happened rather than a problem with your infrastructure.

User avatar
ykud
Posts: 112
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by ykud » Tue Jul 21, 2020 1:08 am

I'd check whether you have LockPagesInMemory set up in tm1s.cfg file, it can cause the 'out of memory' error even on a system with plenty RAM available as TM1 will 'mark' pages without really 'filling' them up.

redflame2012
Posts: 5
Joined: Wed Sep 25, 2019 7:31 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2016

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by redflame2012 » Thu Jul 23, 2020 2:12 am

Thanks, Steve Rowe.
Yes, I'm sure the tm1s.exe process was still up when the 145x error codes happened, and it did not restart when the errorrs happened.

redflame2012
Posts: 5
Joined: Wed Sep 25, 2019 7:31 am
OLAP Product: TM1
Version: 10.2.2
Excel Version: 2016

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by redflame2012 » Thu Jul 23, 2020 2:18 am

Thanks, ykud.

I will check if setting LockPagesInMemory=F could solve the problem.

When the error happened, the working set of tm1s.exe process is about 49G, and almost of these 49G is locked and garanteed non-paging like the IBM doc says, but the free physical memory on that VM is still 100G which could be used by the system and other processes, and this is what baffled me.

User avatar
ykud
Posts: 112
Joined: Sat Jan 10, 2009 10:52 am
Contact:

Re: Windows Server errors ERROR_NOT_ENOUGH_MEMORY ERROR_NO_SYSTEM_RESOURCES

Post by ykud » Thu Jul 23, 2020 6:55 am

redflame2012 wrote:
Thu Jul 23, 2020 2:18 am
Thanks, ykud.

I will check if setting LockPagesInMemory=F could solve the problem.

When the error happened, the working set of tm1s.exe process is about 49G, and almost of these 49G is locked and garanteed non-paging like the IBM doc says, but the free physical memory on that VM is still 100G which could be used by the system and other processes, and this is what baffled me.
this parameter is set to F by default, you should check if it's present in the config file and set to T.

Working set is how much TM1 is really using, it can mark a page as 'locked' and not use it, so it's a bit misleading. I saw this when a 12-16 gb Tm1 server was bringing down a 64 box with 30+ GB 'free' at the time.

Cheers

Post Reply