Azure Machine Learning Deployment at Scale Using ARM and AMLPS.

By | September 9, 2016

Introduction

In this post, I will demonstrate how to do a simple but useful scenario when managing Azure Machine Learning Workspaces and Experiments, copying all the experiments under one workspace, deploy a new workspace using ARM (Azure Resource Manager) in another region, and then copy the experiment under the newly deployed workspace.

Preparation

You will need to install a couple of things:

  • Azure Resource Manager PowerShell modules
  • Azure Service Management PowerShell modules
  • Azure Machine Learning PowerShell modules (via github).
  • [powershell]
    # Install the Azure Resource Manager modules from the PowerShell Gallery
    Install-Module AzureRM -Scope CurrentUser

    # Install the Azure Service Management modules from the PowerShell Gallery
    Install-Module Azure -Scope CurrentUser
    [/powershell]
    Installing AMLPS, is not yet as easy but it is not too hard :).

  • Download the latest zip file from https://github.com/hning86/azuremlps/releases (as of today, the beta is 0.2.8), unzip to a folder (let’s say c:\amlps) – this is a manual step.
  • Navigate to c:\amlps
  • Unblock the file
  • Load the module
  • [powershell]
    #Unblock the downloaded dll file so Windows can trust it.
    Unblock-File .\AzureMLPS.dll

    #import the PowerShell module into current session
    Import-Module .\AzureMLPS.dll
    [/powershell]

    Of course, you will also need to have an Azure Machine Learning account.

    Configure

    Now that all the modules are installed and imported, let’s configure our session.
    First making sure we authenticate to Azure RM and SM and can list our workspaces.
    [powershell]
    # Authenticate (enter your credentials in the pop-up window)
    Login-AzureRmAccount

    # List all workspaces
    Get-AzureRmResource |? { $_.ResourceType -Like "*MachineLearning/workspaces*"}
    [/powershell]
    At this point, you should see all your workspaces listed.
    We are now ready to configure AMLPS, for this you will need to retrieve the workspace ID, and the authorization token (detailed step by step for AMLPS configuration). We will update config.json (in c:\amlps) for simplicity. From the above list of workspaces, select the one you want to duplicate. In my example, the workspace name is “workspaceus”.
    [powershell]
    # Select workspace with the name “workspaceus”
    $wsp = Get-AzureRmResource |? { $_.Name -Like "workspaceus"}

    # Get the workspaceId
    $wid = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Properties.workspaceId

    # Get the primary token
    $wpt = (Invoke-AzureRmResourceAction -ResourceId $wsp.ResourceId -Action listworkspacekeys -Force).primaryToken

    # Get the location of the workspace
    $wil = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Location

    # Create the JSON config file
    (New-Object psobject | Add-Member -PassThru NoteProperty Location $wil | Add-Member -PassThru NoteProperty WorkspaceId $wid | Add-Member -PassThru NoteProperty AuthorizationToken $wpt) | ConvertTo-Json > config.json
    [/powershell]

    Deploy

    Now that we have installed and configured all the tools, we can start our example. We will copy a workspace and its experiment located in “South Central US” and copy it to “West Europe”. Below is an illustration representing the different steps in our journey.
    machinelearningworkspace

    Get workspace information

    We will export all experiment graph as a JSON file so we can import them back on the new workspace.
    [powershell]
    # Create folder for export
    New-Item -Name "Export" -ItemType "directory" -Force

    # Export all experiments in the workspace
    Get-AmlExperiment |% {$i=0}{Export-AmlExperimentGraph -ExperimentId $_.ExperimentId -OutputFile "c:\AzureMLPS\export\exp$i.json"; $i++}
    [/powershell]

    Deploy new workspace in another location

    To deploy the new workspace, you can refer to this more detailed article to get a sample ARM template to deploy a new machine learning workspace.
    [powershell]
    # Create a new resource group in West Europe.
    $rg = New-AzureRmResourceGroup -Name "uniquenamerequired723" -Location "West Europe"

    # Deploy a Resource Group, TemplateFile is the location of the JSON template.
    $rgd = New-AzureRmResourceGroupDeployment -Name "demo" -TemplateFile "mlworkspace.json" -ResourceGroupName $rg.ResourceGroupName
    [/powershell]

    Copy the experiment into the new workspace

    First, we need to update the configuration for AMLPS to the new location.
    [powershell]
    # Select workspace just created
    $wsp = Get-AzureRmResource |? { $_.ResourceGroupName -Like $rgd.ResourceGroupName -AND $_.ResourceType -Like "Microsoft.MachineLearning/Workspaces"}

    # Get the workspaceId
    $wid = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Properties.workspaceId

    # Get the primary token
    $wpt = (Invoke-AzureRmResourceAction -ResourceId $wsp.ResourceId -Action listworkspacekeys -Force).primaryToken

    # Get the location of the workspace
    $wil = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Location

    # Create the JSON config file
    (New-Object psobject | Add-Member -PassThru NoteProperty Location $wil | Add-Member -PassThru NoteProperty WorkspaceId $wid | Add-Member -PassThru NoteProperty AuthorizationToken $wpt) | ConvertTo-Json > config.json
    [/powershell]
    Now we are able to import the experiments we copied from the previous workspace.
    [powershell]
    Get-ChildItem export\* -Include *.json |% {Import-AmlExperimentGraph -InputFile $_ }
    [/powershell]

    Test that all is working correctly

    The simplest thing to do at this point is to list all the experiments under the workspace.
    [powershell]
    Get-AmlExperiment
    [/powershell]
    The result of this command should be all the experiment listed under your newly deployed workspace.

    If you have no idea where to start with you Machine Learning experiment, you can have a look at a tutorial I wrote a while ago about getting started with Azure Machine Learning. You should also check the Cortana Intelligence Gallery where plenty of experiments are available for free.

     
    3 Kudos
    Don't
    move!

    Thoughts?