TI does not recognize character set?

[fleaster]

Hi all,
bit of a bizarre one - we're trying to import a txt file via TI, however it seems to have a hiccup whenever it encounters a particular character:

ΫΫΫΫΫ

...in addition, it appears it cannot be copied or pasted - the only time it can be seen is when it is pasted in Word. Questions...

1/ has anyone ever encountered this before, and how did you get around it?

2/ i know there is a "setinputcharacterset" function in TI, but am wondering what the original character set for TI is, and which character set the above character belongs to...?

Matt

[Duncan P]

Is the text file small enough (or could you cut it down to be small enough) to attach? You could for example load it in Notepad, remove all but the characters concerned and save it under a new name. In notepad you might have to set the font to a unicode font, such as Arial Unicode or Lucida Sans. Then it might be possible to see from how the characters are represented in the source what encoding is being used.

When you say it has a hiccup, what exact behaviour are you seeing? Is there an error message for instance?

[RJ!]

Hi Duncan P,

Just a quick background, our current process is to copy a .data file from our source systems, there is scripting that moves that .data file to another location and renames it to a .txt file ready to import. We then use MS Access to do the necessary "Import to Table" / "text to colomns" from there, though we want to automate this via a TI Process and this particular "issue" is a show stopper for us.

Please find attached and example file of what Fleaster is working on, the issue is with the 3rd line, in the text file it looks to be "Blank" after the letters "EP". If you open the file in Word and turn on the "Show Formatting Symbols" you can see what was "Blank" is actually ÿÿÿÿÿÿÿÿ.

This seems to be a special character that is causing our Import via TI to basically skip the whole line in the import process of our .txt file.
The code we have to pick up that field is: REF=Trim(Subst(V1,69,9));

Interestingly if we use the original .data file that this is created, it creates the line in our table but will not populate any of the fields to the right of the ÿÿÿÿÿÿÿÿ. (for example: AMOUNT_L=Trim(Subst(V1,79,18));) If we use the .txt file it seems to just drop the line altogether.


I'm more leaning towards creating a VB script that that does a "Find & Replace" on the file and fix it that way, but since this seems to be a special character, I can't seem to paste what I'm seeing into any code

[Duncan P]

Hi RJ!,

Thanks for the background. I loaded the file in a hex editor and the bytes corresponding to your characters are all 0. This is normally what C programs use to represent the character after the end of a string, and it does not represent a character in any recognised character set. For some reason Word is choosing to use the character "ÿ" to show you instead, but that is Word's choice and has nothing to do with the underlying file.

You can see the section outlined in red in the following picture.

Text file shown in HEX editor

null_chars.png (18.07 KiB) Viewed 5775 times

I suggest you find out if it is the same in the .data file and if so go back to the source system that provided it to investigate there. I would be surprised if you could ever get TI to import it properly. It might be worth trying your VBScript search and replace though.

Good luck,
Duncan.

[RJ!]

Hi Duncan,

Thanks for having a look, if we were to create a script, what would the "Find" be?

Getting our source system changed is an unlikely scenario unfortunately.

[Duncan P]

Even if you are unlikely to get the source system changed I would still try to find out why you are getting these NULL characters in your export. That is almost definitely a bug in the source system export process, as these are just not valid text characters. Knowing exactly why they are occurring will also give you some idea of in what patterns they could occur in the future. What you need to be searching for in VBScript is the character given by the VBScript function CHR(0), although I don't know whether VBScript will load these correctly from the file.