BulkLoadMode - Hanging 10.1

[failurehappening]

Background
The overnight process is scheduled daily and has the structure:

Start Maintenance
Load MetaData
Update Subsets
Stop Maintenance
Load Landing Data
Load Cube Data
Finish

The Start Maintenance part of the process only has one function, it enables BulkLoadMode with the command
EnableBulkLoadMode() ;

Problem Definition
For the second time in a week, when the overnight process was scheduled and the chore was run, the start maintenance process would start, but would not finish processing the TI process that enabled the bulk load:

6364 [] INFO 2014-04-10 07:00:14.200 TM1.Chore Chore "OVERNIGHT" executed by scheduler
6364 [] INFO 2014-04-10 07:00:14.200 TM1.Process Process sys - Start Maintenance executed by chore OVERNIGHT
Server Restarted
6572 [] INFO 2014-04-10 19:01:09.242 TM1.Server Closing...
6572 [] INFO 2014-04-10 19:01:09.242 TM1.Server Saving...
6572 [] INFO 2014-04-10 19:01:09.242 TM1.Server The server is coming down...

The system has entered bulk load mode as can be seen through Ops Console and if you try to connect to the instance through Architect, you get the error ‘ServerInBulkMode’.

The only way to get the system back is to kill the process and restart.

When the process runs successfully it can be seen in the tm1server.log that the next phase of the overnight chore has executed:

2360 [] INFO 2014-04-08 07:00:06.323 TM1.Chore Chore "OVERNIGHT" executed by scheduler
2360 [] INFO 2014-04-08 07:00:06.323 TM1.Process Process sys - Start Maintenance executed by chore OVERNIGHT
2360 [] INFO 2014-04-08 07:00:11.081 TM1.Process Process "sys - Start Maintenance": finished executing normally, elapsed time 4.76 seconds
2360 [] INFO 2014-04-08 07:00:11.081 TM1.Process Process Dimension Load executed by chore OVERNIGHT
2360 [] INFO 2014-04-08 07:00:11.081 TM1.Process Process "Project 1 Dimension Load" run from process "Dimension Load" by user "R*OVERNIGHT"

The TM1 model code had not changed between these two runs and as can be seen, they were scheduled to run at the same time.

There are no additional details in the tm1server.log.

EnableBulkLoadMode() ; Does not work reliably, causing the system to hang and require a service restart

TM1 Build Number: 10.1.10000.26473 (No FixPacks......)

Windows Server 2008 R2

I saw that the same problem was high lighted here http://www.tm1forum.com/viewtopic.php?p=14838 but this link relates to 9.4.1 (FP3) David said that he thought bulk load mode was being depreciated from 9.5.2, but it's still here in 10.1 has anyone else found similar issues using BulkLoadMode in 10.1 or beyond?

[qml]

I guess the main question is: why do you even need the Bulk Load Mode? In TM1 10.1.1 most of the typical reasons for using it do not exist anymore because of PI.

[failurehappening]

Essentially we have 2 instances of TM1 running on the one admin server, both using contributor, but performing different business functions. The overnight load of data happens in instance1, performing metadata updates for at least an hour. While this chore is in the metadata phase, when users of Instance2 attempt to connect to contributor, they enter a wait state. This is because contributor attempts to connect to instance1 to establish which applications are set to active. However, because instance1 is doing metadata updates, the contributor user sits waiting to logon.

Enabling BulkLoadMode in instance1 means that it actively refuses connections so the instance2 contributor user tries to connect and instantly gets rejected from instance1 moving on to instance2 and suffers no wait time.

Ideally we would have a separate contributor server, but standing another server up in production is not a quick activity here...

[Gabor]

The only problem I can remember is, that bulk load became unstable, if users or lost user sessions are still connected and maybe running queries/calculations.
I haven't really tried to find the root cause. BTW, I had used it to limit RAM usage during TI processing and it boosts the refeeding triggered by TI.

Another question, what drives the one hour metadata processing? I guess, it's the time to retrieve data from your external source.
So my preferred solution would be, to load the data into a dummy cube in TM1, where the dims with missing elements are a copy of the original ones.

This should give you the option to load without any impact to users. Once the data reside in TM1, you can simply perform a quick update of the original dims and load data using PI.

Regards
Gabor