Create a process with TM1 REST API where data source contains accented characters

[Wim Gielis]

Hello all,

I'm stumped on this one !

It started with running a TI process that would create a drill process for a certain cube, in an automated way. The TI process woud create a view and subsets, and the TM1 REST API is used to create the custom }Drill process. In the middle, in between TI and TM1 REST API, there is PowerShell to execute the REST API call. The drill stays within the same cube, for the sake of illustration: the drill consists of having all level 0 descendants of the chosen combination.

It works fine for regular cube names. However, as soon as the cube name contains accented characters, it breaks down.
In fact, there is quite some code but I narrowed it down to the below.

I have a code in PowerShell to call the REST API. The script grabs the body for the call from a text file.
If I use Postman to execute the REST API call, it all works perfectly even with a cubename that contains accents in the name. I merely copy/paste the text file contents, execute in Postman and the process is created.

However, executing the PowerShell script, manually, will after about 5 seconds also create a process, but the accented character is replaced / unknown / invisible.

Having said this, below are my codes to reproduce the issue. The code is as small as possible. With a few small steps / changes you can reproduce it on your system.

The cube is called 'Financiële rapportering', which is 'Financial reporting' in Dutch Please create that cube with dummy dimensions and also create a Default view.

Next, the text file contents of 'D:\test.txt':

Code: Select all

{
"Name": "test",
"HasSecurityAccess": true,
"DataSource": {
"Type": "TM1CubeView",
"dataSourceNameForServer": "Financiële rapportering",
"dataSourceNameForClient": "Financiële rapportering",
"view": "Default"
},
"Parameters": [],
"Variables": [],
"PrologProcedure": "\r\n\r\n",
"MetadataProcedure": "\r\n\r\n",
"DataProcedure": "\r\n\r\n",
"EpilogProcedure": "\r\n\r\n"
}

Executing this in Postman works absolutely fine. Lastly, the PowerShell file:

Code: Select all

$TM1User = "wim"
$TM1PW = "test"

$Secure_String_Pwd = $TM1PW | ConvertTo-SecureString -AsPlainText -Force
$TM1Password = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($Secure_String_Pwd))

$headers = @{"Authorization" = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($TM1User):$($TM1Password)"));}
$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession

$body = Get-Content "D:\test.txt" -raw

$TM1_Instance_url = 'http://AEX199:8001/api/v1/Processes'

$TM1_create_process = Invoke-RestMethod -Method Post -uri $TM1_Instance_url -WebSession $session -Headers $headers -Body $body -ContentType "application/json"

Please change a few things like username/password/SSL or not/httpportnumber.

If I execute the PowerShell file manually or through TI, it takes about 5 seconds. And still the datasource is not 'Financiële rapportering' but rather 'FinanciWRONG_CHARACTERle rapportering'

04.png (24.77 KiB) Viewed 7656 times

03.png (24.3 KiB) Viewed 7656 times

02.png (21.17 KiB) Viewed 7656 times

So I guess the questions center around:
- what happens when PowerShell calls the REST API such that it produces wrong characters ?
- what do others use to execute REST API scripts starting from TI ? Always Python and TM1py ?

Does anyone have a clue please ? Thanks !

[PavoGa]

Have you tried aliases on the cube to get around the accented characters?

[Wim Gielis]

Have you tried aliases on the cube to get around the accented characters?

Hello ty,

Good idea, no not yet. I am still in the process of understanding what happens and correcting it.

[dr.nybble]

Have you tried the -Encoding flag on Get-Content?

What encoding is your text file in?

[Wim Gielis]

I figured this must indeed be the next step to look at.
When I open D:\test.txt with Notepad++, it says encoded in ANSI.
I'll start experimenting shortly.

[dr.nybble]

Try to convert it to UTF8 with no BOM (which is the default for Get-Content) which Notepad++ can do

[Wim Gielis]

EDIT: gotcha, it's UTF-8, my bad. Now it works fine !
I will bring it into the bigger process of creating drills and make it work.
Many thanks.



----

I have done that, no luck yet.

02.png (116.06 KiB) Viewed 7617 times

On the left, the PowerShell call of with the test.txt file on the right (with encoding now changed).
Then the TI process with missing accent character.

[Wim Gielis]

Still having an issue with the character set.
This small TI process:

Code: Select all

sFileName = 'tester.txt';
SetOutputCharacterSet( sFileName, 'TM1CS_UTF8' );
TextOutput( sFileName, 'Financiële rapportering' );

gives me:

03.png (25.18 KiB) Viewed 7602 times

While changing the character set to the below gives me:

Code: Select all

sFileName = 'tester.txt';
SetOutputCharacterSet( sFileName, 'TM1CS_UTF16ESC' );
TextOutput( sFileName, 'Financiële rapportering' );

gives me:

04.png (26.56 KiB) Viewed 7602 times

Source for character sets:
https://www.ibm.com/support/knowledgece ... aracterset

What should I use to see the right character WITH the right encoding set of UTF-8 in Notepad++ ?
I cannot change the encoding manually as the process of creating the TI process is automated.

Thanks !

[Wim Gielis]

While trying to get the accented (in fact, non-ASCII) characters to work, I found this interesting article on encodings: http://kunststube.net/encoding
It certainly helps in getting this straight, after which the translation to PowerShell needs to be done.