Introduction
In this post, I will demonstrate how to do a simple but useful scenario when managing Azure Machine Learning Workspaces and Experiments, copying all the experiments under one workspace, deploy a new workspace using ARM (Azure Resource Manager) in another region, and then copy the experiment under the newly deployed workspace.
Preparation
You will need to install a couple of things:
[powershell]
# Install the Azure Resource Manager modules from the PowerShell Gallery
Install-Module AzureRM -Scope CurrentUser
# Install the Azure Service Management modules from the PowerShell Gallery
Install-Module Azure -Scope CurrentUser
[/powershell]
Installing AMLPS, is not yet as easy but it is not too hard :).
[powershell]
#Unblock the downloaded dll file so Windows can trust it.
Unblock-File .\AzureMLPS.dll
#import the PowerShell module into current session
Import-Module .\AzureMLPS.dll
[/powershell]
Of course, you will also need to have an Azure Machine Learning account.
Configure
Now that all the modules are installed and imported, let’s configure our session.
First making sure we authenticate to Azure RM and SM and can list our workspaces.
[powershell]
# Authenticate (enter your credentials in the pop-up window)
Login-AzureRmAccount
# List all workspaces
Get-AzureRmResource |? { $_.ResourceType -Like "*MachineLearning/workspaces*"}
[/powershell]
At this point, you should see all your workspaces listed.
We are now ready to configure AMLPS, for this you will need to retrieve the workspace ID, and the authorization token (detailed step by step for AMLPS configuration). We will update config.json (in c:\amlps) for simplicity. From the above list of workspaces, select the one you want to duplicate. In my example, the workspace name is “workspaceus”.
[powershell]
# Select workspace with the name “workspaceus”
$wsp = Get-AzureRmResource |? { $_.Name -Like "workspaceus"}
# Get the workspaceId
$wid = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Properties.workspaceId
# Get the primary token
$wpt = (Invoke-AzureRmResourceAction -ResourceId $wsp.ResourceId -Action listworkspacekeys -Force).primaryToken
# Get the location of the workspace
$wil = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Location
# Create the JSON config file
(New-Object psobject | Add-Member -PassThru NoteProperty Location $wil | Add-Member -PassThru NoteProperty WorkspaceId $wid | Add-Member -PassThru NoteProperty AuthorizationToken $wpt) | ConvertTo-Json > config.json
[/powershell]
Deploy
Now that we have installed and configured all the tools, we can start our example. We will copy a workspace and its experiment located in “South Central US” and copy it to “West Europe”. Below is an illustration representing the different steps in our journey.
Get workspace information
We will export all experiment graph as a JSON file so we can import them back on the new workspace.
[powershell]
# Create folder for export
New-Item -Name "Export" -ItemType "directory" -Force
# Export all experiments in the workspace
Get-AmlExperiment |% {$i=0}{Export-AmlExperimentGraph -ExperimentId $_.ExperimentId -OutputFile "c:\AzureMLPS\export\exp$i.json"; $i++}
[/powershell]
Deploy new workspace in another location
To deploy the new workspace, you can refer to this more detailed article to get a sample ARM template to deploy a new machine learning workspace.
[powershell]
# Create a new resource group in West Europe.
$rg = New-AzureRmResourceGroup -Name "uniquenamerequired723" -Location "West Europe"
# Deploy a Resource Group, TemplateFile is the location of the JSON template.
$rgd = New-AzureRmResourceGroupDeployment -Name "demo" -TemplateFile "mlworkspace.json" -ResourceGroupName $rg.ResourceGroupName
[/powershell]
Copy the experiment into the new workspace
First, we need to update the configuration for AMLPS to the new location.
[powershell]
# Select workspace just created
$wsp = Get-AzureRmResource |? { $_.ResourceGroupName -Like $rgd.ResourceGroupName -AND $_.ResourceType -Like "Microsoft.MachineLearning/Workspaces"}
# Get the workspaceId
$wid = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Properties.workspaceId
# Get the primary token
$wpt = (Invoke-AzureRmResourceAction -ResourceId $wsp.ResourceId -Action listworkspacekeys -Force).primaryToken
# Get the location of the workspace
$wil = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Location
# Create the JSON config file
(New-Object psobject | Add-Member -PassThru NoteProperty Location $wil | Add-Member -PassThru NoteProperty WorkspaceId $wid | Add-Member -PassThru NoteProperty AuthorizationToken $wpt) | ConvertTo-Json > config.json
[/powershell]
Now we are able to import the experiments we copied from the previous workspace.
[powershell]
Get-ChildItem export\* -Include *.json |% {Import-AmlExperimentGraph -InputFile $_ }
[/powershell]
Test that all is working correctly
The simplest thing to do at this point is to list all the experiments under the workspace.
[powershell]
Get-AmlExperiment
[/powershell]
The result of this command should be all the experiment listed under your newly deployed workspace.
If you have no idea where to start with you Machine Learning experiment, you can have a look at a tutorial I wrote a while ago about getting started with Azure Machine Learning. You should also check the Cortana Intelligence Gallery where plenty of experiments are available for free.