bhinbest wrote:Hello All,
I need to create a process that can retrieve data from .txt files in a specific folder. I believe their is a way to get TM1 to automatically retreive the data from the latest file(s) in the folder on a nightly basis.
For example, every night a program is going to create a new .txt file with data (it may be possible that more than one file is created per night, in which case all of these new files need to be loaded) with a naming convention that has yet to be devised, but that I can create to facilitate this process. I need TM1 to access this 'newest' file(s) and upload the data, ignoring all other files that have already been uploaded.
Also, I may need to access these files through something called "FTP", which I'm not familiar with. Anyone know what this means and how to set-up a process to access a file using "FTP"?
Any thoughts, or points of clarification? Thanks in advance for any and all help!!
Holy heaven, you ain't asking much.
OK, to start with, I'd suggest that you use a datestamp to name your files. Your bigger problem is that you don't know how many files there will be, so I'd suggest making a numeric addition at the end of the datestamp for this purpose. That means that your filenames will come out in the form, say:
08-08-21_01.cma
08-08-21_02.cma
and so on. I'm going to assume that there will be no more than 9 of them each day; if there are, the string manipulations get a little more tricky but not by much. (I've left a leading 0 in case the numbers do eventually extend beyond 10; it will probably be easier to change the TI code than change the process which names the files. If it's more than 99 files, though... time to rethink your file generation method.)
With your files named by date you can use the Today() function to work out the file name. After that, you can use the FileExists() function to check for the files and to filter out any which relate to other days.
The next step is to define a TI process which will process the files. That will be called by your parent TI process using the ExecuteProcess function. If you're on a version which predates that function, you're screwed. Well, not really but it would be much easier if you ARE on a version which has that function.
In the second TI process, you need to define a string parameter defined to receive the file name. You do that on the Parameters tab of the Advanced tab, if you haven't done it before.
In short then, you have two processes:
(a) One which loops through the list of possible file names. For each file found, it calls;
(b) Another TI process which processes the actual file.
Your basic loop in the parent process (in the Prolog sub-tab of the Advanced tab) will therefore be something like this:
Code: Select all
# Path to the data files; this could be picked up from a control cube instead,
# but let's keep it simple.
SC_PATH = 'D:\TM1\Temp\';
s_Today = Today;
l_FileCounter = 1;
While ( l_FileCounter <= 9);
s_FileName = s_Today | '_0' | Trim ( Str ( l_FileCounter, 1, 0) ) | '.cma';
If ( FileExists ( SC_PATH | s_FileName ) = 1);
# Found a file by that name. This is where we do the main code.
# This is done by creating a process to process the file,
# creating a parameter to receive the file name,
# then changing the data source to that file name.
# In this example z080821B is the name of the other process.
ExecuteProcess('z080821B', 'FileName', SC_PATH | s_FileName);
Else;
#No more files found, exit the loop.
l_FileCounter = 9;
EndIf;
l_FileCounter = l_FileCounter + 1;
End;
In the second process, you'll have something like this in the Prolog tab. (I've assumed that the Parameter has been defined using the name "FileName"):
That will change the data source for the second TI process to the filename that you passed it.
This method enables you to use a single process to run through multiple data files without having to know their names in advance, or even how many of them there will be.
FTP stands for File Transfer Protocol. Just as a Windows server can run a www service to serve out HTML pages, it can also run an FTP service to serve out files. This is run out of IIS, which you'll come across if you ever inflict Web on yourselves.
To connect to an FTP server you typically need an FTP client. Most of them are freeware, but those aren't designed to do automated file pulls as you'll probably need.
The one I use is Core FTP; you may want to take a look at it. You'll probably need one of the industrial strength versions rather than the Lite one for what you have in mind though:
http://www.coreftp.com/