Consolidation Issues

[jduplessis]

So I have a rather bizarre issue.

I have an archival process for a model that creates a STET version of a budget or forecast. The process first creates a new element for the archive using DimensionElementInsertDirect in the Prolog. Then cycles through a control cube that lists all the cubes that should be archived.

The metadata component of the triggers three separate processes for each cube.

The first process sets up the rules file and inserts a rule to STET out the archive version. ( We split our rules files so that consolidation rules are first then the stet rule inserted by the process and finally n level rules at the end. )

The second process is a Bedrock cube data export process. To output the active version to a flat file, rules calculated values and all.

The third process is a Bedrock cube data import process. To read the flat file and store the values in the STET version.

Feeders in the model are fed from an "Active Versions" hierarchy to prevent feeding from an archive since the whole point is to store everything in the archive as a value.

Now for the weird bit. The process does everything we'd expect: creates the version, sets up rules file and copies the data without issue. But none of the rules based consolidations work for the new version unless the server is restarted. We've tried unloading the cubes, re-firing feeders, reattaching the rules file, everything we could think of short of a rain dance, in all combinations imaginable but the only thing that seems to work is a server restart.

I understand a rules calculation not consolidating since that would indicate a feeder issue, but its almost as if TM1 is not detecting a STET value. And having to restart a production server after an archive is not exactly ideal.

The issue is occurring in TM1 version 10.2.2 FP4 and on a server with FP3.

Any ideas?

[qml]

I think I saw a problem where DimensionElementInsertDirect() is causing rules not to be applied to new elements.
Check what happens if you run DimensionUpdateDirect() on that dimension - does it fix it without restarting the server?

[jduplessis]

Hi Kamil,

I have not tried using that function. I'll give it a try and let you know if it works.

[jduplessis]

I think I saw a problem where DimensionElementInsertDirect() is causing rules not to be applied to new elements.
Check what happens if you run DimensionUpdateDirect() on that dimension - does it fix it without restarting the server?

Success, this fixed the issue. Archive now has rules calculations working on consolidations without a restart.

Does anyone know why rules aren't applied to elements added using the DimensionElementInsertDriect() function? Is it best practice to use DimensionUpdateDirect any time you use DimensionElementInsertDirect?

[qml]

Does anyone know why rules aren't applied to elements added using the DimensionElementInsertDriect() function? Is it best practice to use DimensionUpdateDirect any time you use DimensionElementInsertDirect?

Considering that DimensionElementInsertDriect does not cause a recompilation of dimensions then it's not exactly a surprise that there might be side effects like that. This behaviour is not documented, but I doubt IBM would accept it as a defect.

I would consider best practice to avoid using DimensionElementInsertDirect unless there is a really convincing use case. It should definitely not be the default way of updating dimensions. And if you do use it then you should do a DimensionUpdateDirect at the end to avoid all the problems associated with direct inserts. IBM documentation mentions something about the memory footprint of the dimension being affected, but you have just found another problem and there could be others too.

In your case I would replace the direct insert with a subprocess that adds the element in the normal way (DimensionElementInsert). In your solution you cannot use that function in the Prolog of the main process as the changes will not be committed until Metadata finishes and will not be visible to other subprocesses, so you would have to redesign everything otherwise.

[jduplessis]

Considering that DimensionElementInsertDriect does not cause a recompilation of dimensions then it's not exactly a surprise that there might be side effects like that. This behaviour is not documented, but I doubt IBM would accept it as a defect.

I would consider best practice to avoid using DimensionElementInsertDirect unless there is a really convincing use case. It should definitely not be the default way of updating dimensions. And if you do use it then you should do a DimensionUpdateDirect at the end to avoid all the problems associated with direct inserts. IBM documentation mentions something about the memory footprint of the dimension being affected, but you have just found another problem and there could be others too.

In your case I would replace the direct insert with a subprocess that adds the element in the normal way (DimensionElementInsert). In your solution you cannot use that function in the Prolog of the main process as the changes will not be committed until Metadata finishes and will not be visible to other subprocesses, so you would have to redesign everything otherwise.

This makes sense. I had to use the insert direct function for the reason you mentioned. Had avoided using a sub process since I did not know there would be an issue with the DimensionInsertDirect. At least now the TI is working but at least in future cases I'll know to be aware of the issue.

Thanks for your help and for the quick responses!