Page 1 of 1

REST API Dimension Clone

Posted: Tue Nov 14, 2023 4:58 pm
by dan.kelleher
Hello,

We have a simple Powershell script to hot promote TI processes, and am looking to implement something similar for dimension metadata.

I'm looking to replicate a dimension between 2 services using the REST API as opposed to TI. It should be an exact copy, including attributes. Does anyone know if this is possible without deleting/ recreating consolidated elements and re-processing the edges? i.e. is there a simple shortcut I'm not aware of?

http://${"host"}:${"port"}/api/v1/Dimensions('from')/Hierarchies('from')/Elements
http://${"host"}:${"port"}/api/v1/Dimensions('from')/Hierarchies('from')/Edges

I am aware of how this can be done by exporting/ importing using file-based approach, and am also aware of git integration as a potential solution, although I'd prefer to be able to trigger this from the source TM1 service in isolation:

https://www.ibm.com/docs/en/planning-an ... ntegration.

Many thanks for looking,

Dan

Re: REST API Dimension Clone

Posted: Tue Nov 14, 2023 6:41 pm
by Wim Gielis
AFAIK, indeed you need the elements and the edges.
No direct endpoint to do this although that would be useful.

Re: REST API Dimension Clone

Posted: Wed Nov 15, 2023 8:17 am
by lotsaram
There's no single endpoint to duplicate a dimension. You can get the elements and structure, but you then need separate calls to create the attributes and then copy the data.

Re: REST API Dimension Clone

Posted: Wed Nov 15, 2023 6:34 pm
by tm123
dan.kelleher wrote: Tue Nov 14, 2023 4:58 pm Hello,

We have a simple Powershell script to hot promote TI processes, and am looking to implement something similar for dimension metadata.

I'm looking to replicate a dimension between 2 services using the REST API as opposed to TI. It should be an exact copy, including attributes. Does anyone know if this is possible without deleting/ recreating consolidated elements and re-processing the edges? i.e. is there a simple shortcut I'm not aware of?

http://${"host"}:${"port"}/api/v1/Dimensions('from')/Hierarchies('from')/Elements
http://${"host"}:${"port"}/api/v1/Dimensions('from')/Hierarchies('from')/Edges

I am aware of how this can be done by exporting/ importing using file-based approach, and am also aware of git integration as a potential solution, although I'd prefer to be able to trigger this from the source TM1 service in isolation:

https://www.ibm.com/docs/en/planning-an ... ntegration.

Many thanks for looking,

Dan
Can you provide some details about your Powershell Script you use to hot promote TI Processes?
I am trying to do the same thing but I cant find any help online

Re: REST API Dimension Clone

Posted: Wed Nov 15, 2023 10:14 pm
by Wim Gielis
tm123 wrote: Wed Nov 15, 2023 6:34 pm
Can you provide some details about your Powershell Script you use to hot promote TI Processes?
I am trying to do the same thing but I cant find any help online
Where are you in your TM1 REST api journey ?
Do you use a tool like Postman to execute requests and capture the response ?
Do you use Python scripting and tm1py to do the heavy lifting ?
What programming language or tool do you use to launch the requests ?

To hot promote TIs, you would typically do a GET request towards Processes(‘nameoftheprocess’) (source TM1 server). Empty body in the request.
Then a POST or PATCH request towards Processes to create or update a process (destination TM1 server). Body contains the JSON representation of the process contents (reworked from the GET output above)

If you show what you now have, people might be able to provide more help.

Re: REST API Dimension Clone

Posted: Thu Nov 16, 2023 2:11 am
by tm123
Wim Gielis wrote: Wed Nov 15, 2023 10:14 pm
tm123 wrote: Wed Nov 15, 2023 6:34 pm
Can you provide some details about your Powershell Script you use to hot promote TI Processes?
I am trying to do the same thing but I cant find any help online
Where are you in your TM1 REST api journey ?
Do you use a tool like Postman to execute requests and capture the response ?
Do you use Python scripting and tm1py to do the heavy lifting ?
What programming language or tool do you use to launch the requests ?

To hot promote TIs, you would typically do a GET request towards Processes(‘nameoftheprocess’) (source TM1 server). Empty body in the request.
Then a POST or PATCH request towards Processes to create or update a process (destination TM1 server). Body contains the JSON representation of the process contents (reworked from the GET output above)

If you show what you now have, people might be able to provide more help.
I have used Postman but I was hoping I can use powershell script to do the following

Lets say I have all the processes I want to promote in a folder. Loop through all the processes and execute a GET Request for each of them in the Source Server, and the execute a POST/Patch request for that process in Target Server.

I clearly need some help, so if someone can share a powershell script that does something similar, i would really appreciate it

Re: REST API Dimension Clone

Posted: Thu Nov 16, 2023 12:16 pm
by Wim Gielis
tm123 wrote: Thu Nov 16, 2023 2:11 am
I clearly need some help, so if someone can share a powershell script that does something similar, i would really appreciate it
Hello, I do hope that someone can post code here.
Until then, I would advise to start yourself because if no one does, you still have nothing.
Build it gradually, hardcoded in the beginning, then later on more advanced and dynamic.
In general Google or Chat GPT can be good staring points or even more.

Re: REST API Dimension Clone

Posted: Fri Nov 17, 2023 3:02 pm
by WilliamSmith
tm123 wrote: Wed Nov 15, 2023 6:34 pm I clearly need some help, so if someone can share a powershell script that does something similar, i would really appreciate it
I'm willing to dig into this to see what programmatic solution is possible. Can you give me a definition of "hot promote" in this context, that's one that's not in my vocabulary :D

Re: REST API Dimension Clone

Posted: Fri Nov 17, 2023 6:08 pm
by tm123
WilliamSmith wrote: Fri Nov 17, 2023 3:02 pm
tm123 wrote: Wed Nov 15, 2023 6:34 pm I clearly need some help, so if someone can share a powershell script that does something similar, i would really appreciate it
I'm willing to dig into this to see what programmatic solution is possible. Can you give me a definition of "hot promote" in this context, that's one that's not in my vocabulary :D
Thank you William,

What I am trying to achieve is migrate a Process from DEV to QA for example, without having to restart the QA Instance. I was hoping I can do that with Rest API / Powershell

Re: REST API Dimension Clone

Posted: Fri Nov 17, 2023 6:36 pm
by gtonkin
This series may help you get familiar with some of the commands and then those concepts could be translated to Powershell.

YKud’s article may be very helpful too…

Re: REST API Dimension Clone

Posted: Wed Nov 22, 2023 4:01 pm
by Edward Stuart
I've not toyed with Powershell for some time but a historical script I have used looked like this:

Code: Select all

$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", <Authentication Method - TM1/ CAM/ PA On Cloud/ AWS?>)

$csvfolder = <MyExportLocation>
$csvfile = $csvfolder + '<MyExportFileName>.csv'

$response = Invoke-RestMethod 'http://<ServerName>:<HttpPortNumber>/api/v1/Dimensions(''<DimName>'')/Hierarchies(''<DimName>'')?$select=Name&$expand=Elements' -Method 'GET' -Headers $headers
$response | ConvertTo-Json
$response.Elements | Export-Csv -Delimiter "," -Encoding "UTF8" -path $csvfile -NoTypeInformation

Write-Host $response

I assume it still works! But should get you started

There are a few guides around on Authentication but George's code covers some of this but if you are hitting issues let us know

Re: REST API Dimension Clone

Posted: Wed Dec 27, 2023 1:57 pm
by dan.kelleher
We use something like this

Code: Select all

#.\hot_promote_process.ps1 my_process_name DEV UAT PATCH

$processName  =$args[0]
$sourceEnv =$args[1]
$targetEnv = $args[2]
$method = $args[3]

function Set-Header {
    param (
        $EncodedText
    )
    
    $header = @{
        "Content-Type"="application/json; odata.metadata=none; odata.streaming=true; charset=utf-8"
        "Authorization"="Basic ${EncodedText}"
    }
    
    return $header
}

$sourceServer = switch ( $sourceEnv )
{
    "DEV" { "ip address/ host name"}
    "UAT" { "ip address/ host name"}
}

$targetServer = switch ( $targetEnv )
{
    "DEV" { "ip address/ host name"}
    "UAT" { "ip address/ host name"}
}

$sourcePort = switch ( $sourceEnv )
{
    "DEV" { "1234"}
    "UAT" { "5678"}
}

$targetPort = switch ( $targetEnv )
{
    "DEV" { "1234"}
    "UAT" { "5678"}
}

$Bytes = [System.Text.Encoding]::UTF8.GetBytes("user:pass")
$EncodedText =[Convert]::ToBase64String($Bytes)

$url = $url = "http://${sourceServer}:${sourcePort}/api/v1/Processes('$processName')"
$header = Set-Header $EncodedText
$response = Invoke-WebRequest -Uri $url -Method Get -Headers $header

$x = $response.content | ConvertFrom-Json
$x = $x | Select-Object * -ExcludeProperty Attributes
$body = $x | ConvertTo-Json

$url = $url = "http://${targetServer}:${TargetPort}/api/v1/Processes('$processName')"
$header = Set-Header $targetEncodedText
$response = Invoke-WebRequest -Uri $url -Method $method -Headers $header -Body $body

Re: REST API Dimension Clone

Posted: Thu Dec 28, 2023 12:20 am
by Wim Gielis
Thanks Dan.Kelleher.

After some modifications, this works fine for me:

Code: Select all

#.\hot_promote_process.ps1 my_process_name DEV PRD PATCH

$processName = $args[0]
$sourceEnv = $args[1]
$targetEnv = $args[2]
$method = $args[3]

function Set-Header {
    param (
        $EncodedText
    )

    $header = @{
        "Content-Type"="application/json; odata.metadata=none; odata.streaming=true; charset=utf-8"
        "Authorization"="Basic ${EncodedText}"
    }

    return $header
}

$sourceServer = switch ( $sourceEnv )
{
    "DEV" { "20.123.178.99" }
    "PRD" { "ip address/host name" }
    "UAT" { "ip address/host name" }
}

$targetServer = switch ( $targetEnv )
{
    "DEV" { "ip address/host name" }
    "PRD" { "20.123.178.99" }
    "UAT" { "ip address/host name" }
}

$sourcePort = switch ( $sourceEnv )
{
    "DEV" { "8003" }
    "PRD" { "1234" }
    "UAT" { "1234" }
}

$targetPort = switch ( $targetEnv )
{
    "DEV" { "5678" }
    "PRD" { "6002" }
    "UAT" { "5678" }
}

$sourceUser = switch ( $sourceEnv )
{
    "DEV" { "admin" }
    "PRD" { "wim" }
    "UAT" { "wim" }
}

$targetUser = switch ( $targetEnv )
{
    "DEV" { "wim" }
    "PRD" { "admin" }
    "UAT" { "wim" }
}

$sourcePassword = switch ( $sourceEnv )
{
    "DEV" { "apple" }
    "PRD" { "apple" }
    "UAT" { "apple" }
}

$targetPassword = switch ( $targetEnv )
{
    "DEV" { "apple" }
    "PRD" { "apple" }
    "UAT" { "apple" }
}

$sourceUseSSL = switch ( $sourceEnv )
{
    "DEV" { 0 }
    "PRD" { 0 }
    "UAT" { 0 }
}

$targetUseSSL = switch ( $targetEnv )
{
    "DEV" { 1 }
    "PRD" { 1 }
    "UAT" { 1 }
}

$sourceProtocol = if ("${sourceUseSSL}" -eq 1) { "s" } else { "" }
$targetProtocol = if ("${targetUseSSL}" -eq 1) { "s" } else { "" }

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}


# The source side

$sourceBytes = [System.Text.Encoding]::UTF8.GetBytes("${sourceUser}:${sourcePassword}")
# $sourceBytes = [System.Text.Encoding]::UTF8.GetBytes("${sourceUser}:${sourceUser}:${sourceNamespace}")
$sourceEncodedText = [Convert]::ToBase64String($sourceBytes)

$sourceURL = "http${sourceProtocol}://${sourceServer}:${sourcePort}/api/v1/Processes('$processName')"
$header = Set-Header $sourceEncodedText
$sourceResponse = Invoke-WebRequest -Uri $sourceURL -Method Get -Headers $header


# The target side

$x = $sourceResponse.content | ConvertFrom-Json
$x = $x | Select-Object * -ExcludeProperty Attributes
$body = $x | ConvertTo-Json

$targetBytes = [System.Text.Encoding]::UTF8.GetBytes("${targetUser}:${targetPassword}")
# $targetBytes = [System.Text.Encoding]::UTF8.GetBytes("${targetUser}:${targetUser}:${targetNamespace}")
$targetEncodedText = [Convert]::ToBase64String($targetBytes)


$header = Set-Header $targetEncodedText

if ("${method}" -eq "POST") { 
$targetURL = "http${targetProtocol}://${targetServer}:${targetPort}/api/v1/Processes"
} else { 
$targetURL = "http${targetProtocol}://${targetServer}:${targetPort}/api/v1/Processes('$processName')"
}

$targetResponse = Invoke-WebRequest -Uri $targetURL -Method $method -Headers $header -Body $body

Re: REST API Dimension Clone

Posted: Thu Dec 28, 2023 10:33 pm
by ardi
Here is the version of the code I use to promote TIs from one TM1 Instance to another.
I have a config file where I save all the connection details and I created a function to get all the information i need from the config file
Also, I check if the Process exists in Target Instance. If it exists, I use PATCH method, otherwise, I use POST method

Code: Select all

$InputFolder = $args[0]
$srcEnv = $args[1]
$tgtEnv = $args[2]


$Directory = $InputFolder + "\pro\"

function Get-ConfigIni 
{  
    param(  
        [parameter(Mandatory = $true)] [string] $filePath  
    )  
    
    $anonymous = "NoSection"
  
    $ini = @{}  
    switch -regex -file $filePath  
    {  
        "^\[(.+)\]$" # Get Sections from Ini File 
        {  
            $section = $matches[1]  
            $ini[$section] = @{}  
            $CommentCount = 0  
        }  

        "^(;.*)$" # Comment  
        {  
            if (!($section))  
            {  
                $section = $anonymous  
                $ini[$section] = @{}  
            }  
            $value = $matches[1]  
            $CommentCount = $CommentCount + 1  
            $name = "Comment" + $CommentCount  
            $ini[$section][$name] = $value  
        }   

        "(.+?)\s*=\s*(.*)" # Key  
        {  
            if (!($section))  
            {  
                $section = $anonymous  
                $ini[$section] = @{}  
            }  
            $name,$value = $matches[1..2]  
            $ini[$section][$name] = $value  
        }  
    }  

    return $ini  
}  

$configIni = Get-ConfigIni .\config.ini

$srcServer = $configIni.$srcEnv.admin_host
$srcPort = $configIni.$srcEnv.port_number
$srcUser = $configIni.$srcEnv.user
$srcPassword = $configIni.$srcEnv.password
$srcUseSSL = $configIni.$srcEnv.useSSL

$tgtServer = $configIni.$tgtEnv.admin_host
$tgtPort = $configIni.$tgtEnv.port_number
$tgtUser = $configIni.$tgtEnv.user
$tgtPassword = $configIni.$tgtEnv.password
$tgtUseSSL = $configIni.$tgtEnv.useSSL

function Set-Header {
    param (
        $EncodedText
    )

    $header = @{
        "Content-Type"="application/json; odata.metadata=none; odata.streaming=true; charset=utf-8"
        "Authorization"="Basic ${EncodedText}"
    }

    return $header
}


$srcHttp = if ("${srcUseSSL}" -eq 1) { "https" } else { "http" }
$tgtHttp = if ("${tgtUseSSL}" -eq 1) { "https" } else { "http" }

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}


# The source side

$srcBytes = [System.Text.Encoding]::UTF8.GetBytes("${srcUser}:${srcPassword}")
# $srcBytes = [System.Text.Encoding]::UTF8.GetBytes("${srcUser}:${srcUser}:${srcNamespace}")
$srcEncodedText = [Convert]::ToBase64String($srcBytes)

foreach ($FileName in Get-ChildItem $Directory) {
	$FileType = $FileName.Extension
	if ("${FileType}" -eq ".pro") { 
		$processName = $FileName.Basename
		##
		$srcURL = "${srcHttp}://${srcServer}:${srcPort}/api/v1/Processes('$processName')"
		$header = Set-Header $srcEncodedText

		#$srcResponse = Invoke-WebRequest -Uri $srcURL -Method Get -Headers $header

		$srcErrorCode = 0
		try {
			$srcResponse = Invoke-WebRequest -Uri $srcURL -Method Get -Headers $header
		} catch {
			$StatusCode = [int]$_.Exception.Response.StatusCode
			$srcErrorCode = $StatusCode

		}
		if ("${srcErrorCode}" -eq 0) 
		{ 
			$srcErrorCode = $(Invoke-WebRequest -Uri $srcURL -Method Get -Headers $header).statuscode
		}

		# The target side
		if ("${srcErrorCode}" -eq 200) 
		{
			$x = $srcResponse.content | ConvertFrom-Json
			$x = $x | Select-Object * -ExcludeProperty Attributes
			$body = $x | ConvertTo-Json

			$tgtBytes = [System.Text.Encoding]::UTF8.GetBytes("${tgtUser}:${tgtPassword}")
			# $tgtBytes = [System.Text.Encoding]::UTF8.GetBytes("${tgtUser}:${tgtUser}:${tgtNamespace}")
			$tgtEncodedText = [Convert]::ToBase64String($tgtBytes)


			$header = Set-Header $tgtEncodedText

			#== Check if Process Exists in Target Server. If it Exists, use PATCH, otherwise, use POST

			$tgtErrorCode = 0
			$tgtURLCheck = "${tgtHttp}://${tgtServer}:${tgtPort}/api/v1/Processes('$processName')"
			try {
				$tgtResponse = Invoke-WebRequest -Uri $tgtURLCheck -Method Get -Headers $header
			} catch {
				$StatusCode = [int]$_.Exception.Response.StatusCode
				$tgtErrorCode = $StatusCode

			}

			if ("${tgtErrorCode}" -eq 0) 
			{ 
				$tgtErrorCode = $(Invoke-WebRequest -Uri $tgtURLCheck -Method Get -Headers $header).statuscode
			}	
			if ("${tgtErrorCode}" -eq 200 ) 
			{ 
				$method = "PATCH"
				Write-Output "Process $processName exists, therefore, using $method method"
			}
			else
			{
				$method = "POST"
				Write-Output "Process $processName does not exist, therefore, using $method method"
			}

			if ("${method}" -eq "POST") { 
				$tgtURL = "${tgtHttp}://${tgtServer}:${tgtPort}/api/v1/Processes"
			} else { 
				$tgtURL = "${tgtHttp}://${tgtServer}:${tgtPort}/api/v1/Processes('$processName')"
			}
			if ("${method}" -eq "POST") { 
				$tgtResponse = Invoke-WebRequest -Uri $tgtURL -Method $method -Headers $header -Body $body
			} else { 
				$tgtResponse = Invoke-WebRequest -Uri $tgtURL -Method $method -Headers $header -Body $body
			}
		}
		else
		{
			Write-Output "Unable to Hot promote process $processName"
		}
##
		
	}
    
}



Re: REST API Dimension Clone

Posted: Sat Dec 30, 2023 10:10 pm
by Wim Gielis
Thank you Ardian. Good additions.
Maybe useful to post a sample file for 'config.ini' (though probably standard and not too difficult to do it ourselves) ?
Of course without sensitive information but just the structure and some examples of entries.

I would also skip the If test for the PRO extension:

Code: Select all

foreach ($FileName in Get-ChildItem $Directory -Filter "*.pro") {
	$processName = $FileName.Basename

Re: REST API Dimension Clone

Posted: Sun Dec 31, 2023 4:34 pm
by ardi
Wim Gielis wrote: Sat Dec 30, 2023 10:10 pm Thank you Ardian. Good additions.
Maybe useful to post a sample file for 'config.ini' (though probably standard and not too difficult to do it ourselves) ?
Of course without sensitive information but just the structure and some examples of entries.

I would also skip the If test for the PRO extension:

Code: Select all

foreach ($FileName in Get-ChildItem $Directory -Filter "*.pro") {
	$processName = $FileName.Basename
Thanks Wim,

Here is the sample config.ini file

Code: Select all

[DEV]
admin_host=localhost
port_number=8881
user=admin
password=apple
useSSL=0

[UAT]
admin_host=localhost
port_number=8882
user=admin
password=apple
useSSL=0
Good suggestion regarding the if statement, the folder though is supposed to have only pro files and in my code, i write to an audit file the names of files that should not be on that folder.

Re: REST API Dimension Clone

Posted: Mon Jan 01, 2024 11:23 am
by Wim Gielis
Thank you. Happy new year to all 🥳