Loss of Whole Data Directory

Post Reply
User avatar
jim wood
Site Admin
Posts: 3532
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Loss of Whole Data Directory

Post by jim wood » Thu Aug 13, 2015 1:43 pm

We came in this morning to our UAT service down and the whole data directory empty. Luckily our logs are in a different directory and are still there. This is what we found:

Code: Select all

2836   []   INFO   2015-08-12 23:00:05.702   TM1.Chore   Chore "Execute_MCPR_Stored_Processes _Every_1_Hr" finished executing
6396   []   INFO   2015-08-12 23:09:55.796   TM1.Server   Closing...
6396   []   INFO   2015-08-12 23:09:55.796   TM1.Server   Saving...
6396   []   INFO   2015-08-12 23:09:55.796   TM1.Server   The server is coming down...
6396   []   INFO   2015-08-12 23:09:55.796   TM1.Server   TM1ServerImpl::Destroy: disconnect clients
6396   []   INFO   2015-08-12 23:09:55.796   TM1.Server   TM1ServerImpl::Destroy: Deactive all chores
6396   []   INFO   2015-08-12 23:09:55.796   TM1.Chore   Deactivating chore: Execute_MCPR_Stored_Processes _Every_1_Hr
6396   []   INFO   2015-08-12 23:09:57.621   TM1.Chore   Deactivating chore: Meta Build Project Hierarchy - CMWP Subsets
6396   []   INFO   2015-08-12 23:09:59.992   TM1.Chore   Deactivating chore: Save_Data_All_Every_Hour
6396   []   INFO   2015-08-12 23:10:02.566   TM1.Server   TM1ServerImpl::Destroy: Save server data
6396   []   INFO   2015-08-12 23:10:02.582   TM1.Server   TM1ServerImpl::Destroy: destroy chores
6396   []   INFO   2015-08-12 23:10:02.582   TM1.Server   TM1ServerImpl::Destroy: destroy processes
6396   []   INFO   2015-08-12 23:10:02.582   TM1.Server   TM1ServerImpl::Destroy: destroy blobs
6396   []   INFO   2015-08-12 23:10:02.582   TM1.Server   TM1ServerImpl::Destroy: destroy sets
6396   []   INFO   2015-08-12 23:10:02.582   TM1.Server   TM1ServerImpl::Destroy: destroy groups
6396   []   INFO   2015-08-12 23:10:02.582   TM1.Server   TM1ServerImpl::Destroy: destroy clients
6396   []   INFO   2015-08-12 23:10:02.597   TM1.Server   TM1ServerImpl::Destroy: destroy cubes
6396   []   INFO   2015-08-12 23:10:02.956   TM1.Server   TM1ServerImpl::Destroy: destroy dimensions
6396   []   INFO   2015-08-12 23:10:03.939   TM1.Server   TM1ServerImpl::Destroy: destroy connections
6396   []   INFO   2015-08-12 23:10:03.939   TM1.Server   TM1ServerImpl::Destroy: Destroy unregistered objects
6396   []   INFO   2015-08-12 23:10:03.939   TM1.Server   TM1ServerImpl::Destroy: Commit changes
6280   []   INFO   2015-08-12 23:10:04.095   TM1.Server   Terminating Admin Server poller thread.
6396   []   ERROR   2015-08-12 23:10:04.111   TM1.Server   net_SetReadBufferAt: Attempted to set position = 6 past received network data size = 0.
6396   []   INFO   2015-08-12 23:10:04.111   TM1.Server   Server shutdown
I have seen one post on here with something similar but no real help. Have any of you guys seen anything like it? We can't seen any reason why it would happen,

Jim.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 7 64-bit. TM1 Version: 10.1.1

TrevorGoss
Community Contributor
Posts: 217
Joined: Thu Aug 15, 2013 9:05 am
OLAP Product: TM1
Version: 10.2.1.1
Excel Version: 14.0.6129.5000

Re: Loss of Whole Data Directory

Post by TrevorGoss » Thu Aug 13, 2015 2:20 pm

If this makes any difference,

in our logs the line "Terminating Admin Server poller thread." comes before the destruction of objects, cubes, processes etc...

Code: Select all

7336   []   INFO   2015-07-08 02:15:02.700   TM1.Server   Closing...
7336   []   INFO   2015-07-08 02:15:02.700   TM1.Server   Saving...
7336   []   INFO   2015-07-08 02:15:02.700   TM1.Server   The server is coming down...
7336   []   INFO   2015-07-08 02:15:02.700   TM1.Server   TM1ServerImpl::Destroy: disconnect clients
7336   []   INFO   2015-07-08 02:15:02.700   TM1.Server   TM1ServerImpl::Destroy: Deactive all chores
7336   []   INFO   2015-07-08 02:15:02.700   TM1.Chore   Deactivating chore: _HotBackupZip
7336   []   INFO   2015-07-08 02:15:06.834   TM1.Chore   Deactivating chore: Chore_Reload_CoA_and_Prj_Mappings
7336   []   INFO   2015-07-08 02:15:09.408   TM1.Chore   Deactivating chore: FindServiceDetails
7336   []   INFO   2015-07-08 02:15:13.589   TM1.Chore   Deactivating chore: KickContractAnalysis_Reporting
7336   []   INFO   2015-07-08 02:15:18.050   TM1.Chore   Deactivating chore: ManualTrigger_Reset_Calendar_To_Default
7336   []   INFO   2015-07-08 02:15:22.356   TM1.Chore   Deactivating chore: ManualTriggerCalenderCheck
7336   []   INFO   2015-07-08 02:15:23.729   TM1.Chore   Deactivating chore: NightlyRestartService
7336   []   INFO   2015-07-08 02:15:28.690   TM1.Chore   Deactivating chore: PBF_CurrentForecast_Move
7336   []   INFO   2015-07-08 02:15:33.619   TM1.Chore   Deactivating chore: Reprocess Costbase Feeders
6336   []   INFO   2015-07-08 02:15:33.978   TM1.Server   Terminating Admin Server poller thread.
7336   []   INFO   2015-07-08 02:15:33.978   TM1.Server   TM1ServerImpl::Destroy: Save server data
7336   []   INFO   2015-07-08 02:15:34.025   TM1.Server   TM1ServerImpl::Destroy: destroy chores
7336   []   INFO   2015-07-08 02:15:34.025   TM1.Server   TM1ServerImpl::Destroy: destroy processes
7336   []   INFO   2015-07-08 02:15:34.040   TM1.Server   TM1ServerImpl::Destroy: destroy blobs
7336   []   INFO   2015-07-08 02:15:34.040   TM1.Server   TM1ServerImpl::Destroy: destroy sets
7336   []   INFO   2015-07-08 02:15:34.040   TM1.Server   TM1ServerImpl::Destroy: destroy groups
7336   []   INFO   2015-07-08 02:15:34.040   TM1.Server   TM1ServerImpl::Destroy: destroy clients
7336   []   INFO   2015-07-08 02:15:34.040   TM1.Server   TM1ServerImpl::Destroy: destroy cubes
7336   []   INFO   2015-07-08 02:15:34.165   TM1.Server   TM1ServerImpl::Destroy: destroy dimensions
7336   []   INFO   2015-07-08 02:15:34.321   TM1.Server   TM1ServerImpl::Destroy: destroy connections
7336   []   INFO   2015-07-08 02:15:34.321   TM1.Server   TM1ServerImpl::Destroy: Destroy unregistered objects
7336   []   INFO   2015-07-08 02:15:34.321   TM1.Server   TM1ServerImpl::Destroy: Commit changes
7336   []   INFO   2015-07-08 02:15:34.992   TM1.Server   Server shutdown
Maybe this is somthing significant?

BrianL
MVP
Posts: 262
Joined: Mon Nov 03, 2014 8:23 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2 PA2
Excel Version: 2016

Re: Loss of Whole Data Directory

Post by BrianL » Thu Aug 13, 2015 2:31 pm

I've never heard of the data directory becoming empty. Was the server shutdown expected? If not and you're running TM1 as a service you could check the MS event viewer for Windows logs on why the service shutdown.

declanr
MVP
Posts: 1547
Joined: Mon Dec 05, 2011 11:51 am
OLAP Product: Cognos TM1
Version: PA2.0 and most of the old ones
Excel Version: All of em
Location: Manchester, United Kingdom
Contact:

Re: Loss of Whole Data Directory

Post by declanr » Thu Aug 13, 2015 2:34 pm

Blame all your colleagues for pressing delete and see which one cracks first.

User avatar
jim wood
Site Admin
Posts: 3532
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: Loss of Whole Data Directory

Post by jim wood » Thu Aug 13, 2015 2:41 pm

Brian,

In the event viewer we have no critical errors. We have the following error at 7pm:

Code: Select all

 tm1sdx64 error: 0 
   E16) Cannot connect to ODBC data source "TELEDB14_MCPR" IM002[Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified. 
Then at 7.10pm we get this:

Code: Select all

tm1sdx64 error: 2 
   Data directory not specified. Aborting server start up. 
Nothing in between that mentions the service coming down,

Jim.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 7 64-bit. TM1 Version: 10.1.1

BrianL
MVP
Posts: 262
Joined: Mon Nov 03, 2014 8:23 pm
OLAP Product: TM1
Version: 9.5.2 10.1 10.2 PA2
Excel Version: 2016

Re: Loss of Whole Data Directory

Post by BrianL » Thu Aug 13, 2015 2:57 pm

How about the "System" logs in the "Windows Logs" folder? Anything from "Service Control Manager"? I'd expect to see (at least) a message indicating the service entered the stopped state.

User avatar
gtonkin
MVP
Posts: 563
Joined: Thu May 06, 2010 3:03 pm
OLAP Product: TM1
Version: PAL 2.0.1
Excel Version: 2016 64-bit
Location: JHB, South Africa
Contact:

Re: Loss of Whole Data Directory

Post by gtonkin » Thu Aug 13, 2015 3:20 pm

Hi Jim, is the data directory on the same device as the logs, not network attached / a symbolic link / junction etc?
I have seen something similar where we used a SAN and the SAN connection disappeared.
The only other thing that comes to mind is a script Task Scheduler or similar that may have truncated.

User avatar
jim wood
Site Admin
Posts: 3532
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: Loss of Whole Data Directory

Post by jim wood » Thu Aug 13, 2015 3:26 pm

BrianL wrote:How about the "System" logs in the "Windows Logs" folder? Anything from "Service Control Manager"? I'd expect to see (at least) a message indicating the service entered the stopped state.
Brian,

I couldn't find anything in the log directory.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 7 64-bit. TM1 Version: 10.1.1

User avatar
jim wood
Site Admin
Posts: 3532
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: Loss of Whole Data Directory

Post by jim wood » Thu Aug 13, 2015 3:28 pm

gtonkin wrote:Hi Jim, is the data directory on the same device as the logs, not network attached / a symbolic link / junction etc?
I have seen something similar where we used a SAN and the SAN connection disappeared.
The only other thing that comes to mind is a script Task Scheduler or similar that may have truncated.
We are indeed on SAN storage. I've asked our server guy to see if there were any connection issues last night,

Jim.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 7 64-bit. TM1 Version: 10.1.1

User avatar
Alan Kirk
Site Admin
Posts: 5760
Joined: Sun May 11, 2008 2:30 am
OLAP Product: TM1
Version: 9.5.2 64 bit moving to 10.2.2
Excel Version: 2010
Location: Sydney, Australia
Contact:

Re: Loss of Whole Data Directory

Post by Alan Kirk » Thu Aug 13, 2015 7:44 pm

jim wood wrote:We came in this morning to our UAT service down and the whole data directory empty.
...
I have seen one post on here with something similar but no real help. Have any of you guys seen anything like it? We can't seen any reason why it would happen,
The only time I've seen that happen is here, but you, unlike IBM, probably know better than to store data files in the Program Files path. That being the case the SAN issue suggested by gtonkin would seem to be the more productive line of enquiry.
"To them, equipment failure is terrifying. To me, it’s 'Tuesday.' "
-----------
Before posting, please check the documentation, the FAQ, the Search function and FOR THE LOVE OF GLUB the Request Guidelines.

User avatar
jim wood
Site Admin
Posts: 3532
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: Loss of Whole Data Directory

Post by jim wood » Fri Aug 14, 2015 12:02 pm

Alan Kirk wrote:The only time I've seen that happen is here, but you, unlike IBM, probably know better than to store data files in the Program Files path.
This was a setup I inherited. Thankfully this was one thing the previous owners got right. I mean one thing.
Alan Kirk wrote:That being the case the SAN issue suggested by gtonkin would seem to be the more productive line of enquiry.
We've asked our server team to check this out. They haven't spotted anything so far but it could have been caused by only a brief connection issue. This is not a production server so less attention is paid to it. It does (btw) sound like the most likely cause. I don't know how much backside covering is happening within the server team when they say they haven't found anything. Only time will tell I guess.

Thanks for all your input guys. When I get anything back from IBM I'll post it here for future reference,

Jim.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 7 64-bit. TM1 Version: 10.1.1

User avatar
Steve Rowe
Site Admin
Posts: 1719
Joined: Wed May 14, 2008 4:25 pm
OLAP Product: TM1
Version: 10.2.2., PAW
Excel Version: Nearly all of them

Re: Loss of Whole Data Directory

Post by Steve Rowe » Mon Aug 17, 2015 10:01 pm

Hi Jim,
Just to say in my last role we were running prod / uat / dev for 30 plus instances on SANs and not once did I hear of the DD going totally AWOL in the few years I was there.

If the SAN had dropped off for some time I can't see how the whole DD would get deleted, at worst the instance would be unable to write back to the DD and then fall over or you would end up with a bunch of dot $ files or similar? You could test the behaviour of TM1 when the DD goes missing by just running up an instance and deleting / moving the DD and see what messages TM1 produces.

IMO the most likely explanation is that someone deleted the DD in error and is keeping quiet.....In the absence of any evidence that there was a technical issue I'd be shrugging my shoulders and moving on and maybe look at the security on the infrastructure.

Cheers,

User avatar
jim wood
Site Admin
Posts: 3532
Joined: Wed May 14, 2008 1:51 pm
OLAP Product: TM1
Version: TM1 10.2.2
Excel Version: 2007
Location: 1639 Route 10, Suite 107, Parsippany, NJ, USA
Contact:

Re: Loss of Whole Data Directory

Post by jim wood » Tue Aug 18, 2015 12:17 pm

Steve Rowe wrote:IMO the most likely explanation is that someone deleted the DD in error and is keeping quiet.....In the absence of any evidence that there was a technical issue I'd be shrugging my shoulders and moving on and maybe look at the security on the infrastructure.
Normally I'd think the same but only the files stated in the log are missing. The view folders etc are still there. The only file deleted that wasn't in the log was the CFG file.
Struggling through the quagmire of life to reach the other side of who knows where.
Application Consulting Group (ACG) TM1 Consulting
OS: Windows 7 64-bit. TM1 Version: 10.1.1

Post Reply