Returning values from Turbo Integrator processes

[Harvey]

Bedrock has shown us the potential for Turbo Integrator to be a pretty decent development platform. But it also has some limitations, and misses the opportunity for certain types of reusability.

The two cornerstone features of any generic programming platform are encapsulation and reuse. These related ideas are simply the ability to write a piece of code once and then use it many times.

Being able to call processes within other processes is a useful technique for performing a set repeated task, but what if you would like the two processes to interact and communicate?

"Hey", I hear you say, "TI has that covered, doesn't it?" Well, sort of!

Yes, the ability to call a process from within another can allow processes to act like functions or procedures, and yes, they do accept parameters, so the calling process can modify the behavior of child process.

But what about return values? What if you want to build a utility function that performs some processing, and returns a value to the calling process?

I recently decided to tackle this problem, and here's a discussion of various techniques, for those interested.

Read the article here.

[Duncan P]

The limit that you quote of 65536 characters on the length of a string is not borne out by my experience, as has been discussed here. Did you get this from the documentation? If so it needs fixing.

[Harvey]

Hey Duncan, thanks for the feedback. I'm pretty sure I got the number from the forum, but you could well be right, perhaps it would be best to do some practical testing.

Although the article does mention storing global variables in string cells, it is really recommending using StringGlobalVariable commands instead. I know the limit on internal strings is larger than 8k, as I can concatenate the lines of a >12k ASCII file into a global string variable and it works.

I will test the various scenarios, report back here, and update the article with what I find.

[Harvey]

Update: I tested a Turbo integrator process that simply reads all the lines of a text file, concatenates them into a single global string variable, then outputs the variable to an output file. With files > 64k, this process fails, but it works fine with smaller strings.

Note that when it fails, it does not produce an error; it simply creates an empty file. This is curious, as I expected it to result in a truncated string.

I then tried calling ASCIIOUTPUT two times, using SUBST to split the string variable into 2 chunks < 64k each. This was successful, so the 64K limit applies to the ASCIIOUTPUT call itself, NOT the string variable. I checked TEXTOUTPUT and it does the same thing.

To find out the upper limit of the variable itself, I created a loop that continually splits the string into 64k chunks and calls multiple ASCIIOUTPUT functions to output the result. I then tested many times, increasing the string size each time.

Interestingly, the failure was not an error, but a complete freeze up and crash of my entire system!

The result is that TI can (barely) store a global string value of around half a megabyte, on my laptop with 4GB RAM installed. I would imagine since that slows down the entire machine linearly as it completes, it approaches some internal memory limit.

If the string is any longer than that, my system will crash. It's entirely possible, however, that a system with more RAM might survive.

Note that this is TM1 10.1.

[Duncan P]

As you are doing repeated allocation of variables you may also be suffering a bit from this effect - address space fragmentation. I am assuming here that you are running 32 bit TM1.

[Harvey]

No, it's 64-bit. I think there are a number of possibilities for what might be going wrong. Memory allocation issues are possible, but it could also be related to running multiple SUBST commands.

Here's the code in the calling process if you're interested:

Code: Select all

StringGlobalVariable('OutputFileString');
ExecuteProcess('LoadFile', 'pFilePath', 'C:\Temp', 'pFilename', 'Test1.txt');
OutputText = OutputFileString;

ChunkSize = 65535;
CurrentOutputIndex = 0;
CurrentOutputByteCount =0;

WHILE(CurrentOutputByteCount < LONG(OutputText));
    ASCIIOUTPUT('C:\Temp\Output.txt', SUBST(OutputText, CurrentOutputByteCount + 1, ChunkSize));
    CurrentOutputIndex = CurrentOutputIndex + 1;
    CurrentOutputByteCount = CurrentOutputIndex * ChunkSize;
END;

The LoadFile process simply sets the text file as a data source and concatenates the lines into a single global variable.

From watching the status, the performance issue is during the file load, which simply concatenates a string variable multiple times.