TM1 Forum

For newly added element, I would like to change the status from 0 to 1 which indicate that this element is newly added. Very simple process but somehow i coudlnt get it working.

Metadata

IF (DIMIX(vDim, MainNOrderJobNo) = 0);
AttrPutN(1, vDim, MainNOrderJobNo, 'NewlyAddedStatus');
ENDIF;
DimensionElementInsert(vDim, ' ', MainNOrderJobNo,'N');

You have to insert all your elements in the Metadata tab of the process before you try and update attribute data. The elements don't actually exist until the Metadata tab has been completed for every row in the data source. Then, and only then, can you write attributes against the new elements.

If that is the case how can i assign '1' attribute to newly added element ? I tried the code below it doesnt work either.

Metadata

status = 0;
IF (DIMIX(vDim, MainNOrderJobNo) = 0);
status = 1;
ENDIF;
DimensionElementInsert(vDim, ' ', MainNOrderJobNo,'N');

Data

IF(status = 1);
AttrPutN(1, vDim, MainNOrderJobNo, 'NewlyAddedStatus');
ENDIF;

TI runs the metadata script for each source record, then it commits any dimension changes, then it runs the data script for each source record. This means that you need to keep a track of the status of all records. You could do it like this.

Prolog Metadata Data BTW - I had to paste the tab characters into the forum edit field. It made me think I was in the TI editor.

Doesnt seems like there is any data in the text file when i try asciioutput status in "Data". when a variable is change in metadata, does the changes get pass onto data ?

Duncan P wrote:

Data

Code: Select all

datarecord = datarecord + 1;
ASCIIOUTPUT('c:\test.txt', status);
IF ( CODE(status, datarecord) = code1 );
	AttrPutN(1, vDim, MainNOrderJobNo, 'NewlyAddedStatus');
ENDIF;

Variables, once assigned are available in all sections of the process and the values persist across sections.

1. Is the Metadata section being executed? Try ASCIIOUTPUT of status in the Metadata section.

2. Have you put the status = ... lines in the Metadata section?

3. Have you initialised status in the prolog?

It works fine for me. If you can't find out why it doesn't for you then post the .PRO file. There may be something you've missed.

The following are my prolog, metadata, data and epilog code. did i miss out anything ?

Seems like I have included everything you have mention and when i try asciioutput there is data on the status in the metadata section.It is just that the variable content is not pass to data.

Duncan P wrote:Variables, once assigned are available in all sections of the process and the values persist across sections.

1. Is the Metadata section being executed? Try ASCIIOUTPUT of status in the Metadata section.

2. Have you put the status = ... lines in the Metadata section?

3. Have you initialised status in the prolog?

It works fine for me. If you can't find out why it doesn't for you then post the .PRO file. There may be something you've missed.

winsonlee wrote:Doesnt seems like there is any data in the text file when i try asciioutput status in "Data". when a variable is change in metadata, does the changes get pass onto data ?

DuncanP wrote:Variables, once assigned are available in all sections of the process and the values persist across sections.

Yes, but the Data tab will have the last value assigned to 'status' in the Metadata tab. If you have 10 records and, in the Metadata tab, the 'status' is 1 for records 1-9 and 0 for record 10 then when the code in the Data tab executes the 'status' variable will have a value of 0.

Perhaps an alternative is if you create a 'Newly Created' consolidation point and add the new elements in there as well as adding them to other consolidations. Then you can look at the children of that consolidation later in your process(es) and take the relevant action.

Using solution propose by DuncanP, after going through 10 record in metadata, the status will be = '1001100010' as each line of record will concatenate the previous status variable with 1 indicate element does not exist and 0 indicate element exist. and when it goes through the "Data", status should remain to have '1001100010' through out the execution of all records.

After looking through the ASCIIOUTPUT probably i might find out the reason why it is not working. Just to confirm is there any limit how many character a variable able to hold ? after 7987 record which means 'status' is holding 7987 character, the subsequent record, status becomes empty again.

will that be a reason why when i asciioutput 'status' in 'Data', there isnt any data in the variable ?

rmackenzie wrote:

winsonlee wrote:Doesnt seems like there is any data in the text file when i try asciioutput status in "Data". when a variable is change in metadata, does the changes get pass onto data ?

DuncanP wrote:Variables, once assigned are available in all sections of the process and the values persist across sections.

Yes, but the Data tab will have the last value assigned to 'status' in the Metadata tab. If you have 10 records and, in the Metadata tab, the 'status' is 1 for records 1-9 and 0 for record 10 then when the code in the Data tab executes the 'status' variable will have a value of 0.

Perhaps an alternative is if you create a 'Newly Created' consolidation point and add the new elements in there as well as adding them to other consolidations. Then you can look at the children of that consolidation later in your process(es) and take the relevant action.

winsonlee wrote:Using solution propose by DuncanP, after going through 10 record in metadata, the status will be = '1001100010' as each line of record will concatenate the previous status variable with 1 indicate element does not exist and 0 indicate element exist. and when it goes through the "Data", status should remain to have '1001100010' through out the execution of all records.

After looking through the ASCIIOUTPUT probably i might find out the reason why it is not working. Just to confirm is there any limit how many character a variable able to hold ? after 7987 record which means 'status' is holding 7987 character, the subsequent record, status becomes empty again.

If I may quote the TI guide (and, at one time, I did):

TurboIntegrator is capable of handling string data in sizes of up to 8000 single-byte characters at a time. This limit applies when your TI process is performing actions such as assigning a value to a variable or importing individual records of data. Any value or record longer than 8000 singlebyte characters is truncated.

Of course, there's a difference between "truncated" and "obliterated", but that was from the 9.5.0 manual and I've noticed that the one thing that's been consistent about oversized string handling from version to version is... it's not consistent.

I haven't tested it in later incarnations of 9.5.

Since the introduction of extended strings a few releases ago, the maximum length was set at 8000, up from 255. DuncanP's idea is nice but won't work for big input data sets.

Here's the link to the IBM website

It is a good to know info. Probably that is the reason why it is not working in my situation as my data set is more then 8000. Initially i thought it is just a simple solution of if element not found, assign 1 to the attribute.

But when comes to putting it in place, it is not as simple as i thought it is. Probably i will move towards the direction of adding element to 'Newly Created' consolidation and do the necessary checking from there.

Thanks for the help guys.

rmackenzie wrote:Since the introduction of extended strings a few releases ago, the maximum length was set at 8000, up from 255. DuncanP's idea is nice but won't work for big input data sets.

Here's the link to the IBM website

OP doesn't say which flavour of 9.5 he is using but in 9.5.2 the 8000 character limit no longer applies to manipulation of string variables. The lines produce a file with the following line so the variable is there and it has 400000 characters in it. If however I try to ASCIIOUTPUT the string itself I get nothing. There appears to be a limit of 64999 on the length of the overall line output in a single call to ACSIIOUTPUT.

So you would think that the concatenation would be OK still. However that is not the case. A simple test looping while concatenating a character to a variable and then outputting its length shows nothing after about 44399. However if I look in the server log it has the following error message :- At the time of the failure the process (32bit) had just over a gigabyte allocated. What is probably happening is that of the available 2 Gb of address space there is not a single continuous block of the 44K required. This will be because each loop is allocating a block one byte larger than the largest one used so far and so TM1 is having to go to the operating system to give it another one. Furthermore TM1 is not giving back the previous blocks but is hanging on to them while the process is executing which means that the OS is allocating from an ever depleting pool. Using Address Space Monitor (http://hashpling.org/asm/) I got the following map of the address space just before it died.
So concatenation is not a goer, but in 9.5.2 not for the reasons stated in previous posts.

Besides rmackenzie's suggestion is much the better one in this instance.

Duncan P wrote:in 9.5.2 the 8000 character limit no longer applies to manipulation of string variables.

Thanks for the update. Your idea is a good work-around to the absence of an array construct in TI.

Duncan P wrote:What is probably happening is that of the available 2 Gb of address space there is not a single continuous block of the 44K required. This will be because each loop is allocating a block one byte larger than the largest one used so far and so TM1 is having to go to the operating system to give it another one.

I guess the issue is string immutability. TM1 is often frustratingly awkward when it comes to string processing.