Skip to content

Azure Orchestrator

Enterprise

Image$DATAOPS_AZURE_RUNNER_IMAGE

The Azure orchestrator includes the following functionality:

  • Responsive interaction with Microsoft Azure Services using the built-in Azure CLI tools
  • DataOps Vault functionality that allows scripts to retrieve variables from the vault
  • DataOps native tools that allow the development of custom scripts that interact with Azure
  • The following added tools:
  • git
  • curl
  • ssh-client
  • perl
  • sshpass
  • unzip
  • terraform

Usage

The first use case described here is typical for this orchestrator; that is, to start an Azure instance to perform a task in the pipeline:

pipelines/includes/local_includes/azure_jobs/my_azure_job.yml
"My Azure Job":
  extends:
    - .agent_tag
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  stage: "Batch Ingestion"
  variables:
    # use one of the following connection methods
    # identity inheritance
    DATAOPS_USE_IDENTITY: 1

    # or default vault expansion
    SET_AZ_KEYS_TO_ENV: 1

    # or custom vault expansion
    AZURE_USER: DATAOPS_VAULT(PATH.TO.USERNAME.IN.VAULT)
    AZURE_PASSWORD: DATAOPS_VAULT(PATH.TO.PASSWORD.IN.VAULT)

    # or service principal
    DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL: 1
    TENANT_ID: <tenant_id>
    AZURE_APP_ID: DATAOPS_VAULT(PATH.TO.APPLICATION_ID.IN.VAULT)
    AZURE_CLIENT_SECRET: DATAOPS_VAULT(PATH.TO.CLIENT_SECRET.IN.VAULT)
    DISABLE_ALLOW_NO_SUBSCRIPTIONS: 1
  script:
    - /dataops
    - az .. # your azure cli command
  icon: ${AZURE_ICON}

Additionally, the following use cases demonstrate how to connect to Azure from a DataOps pipeline:

Connecting to Azure

The Azure orchestrator supports different methods of connecting to Azure. Only ever use one!

1. Username and password

To use the Azure orchestrator, you must provide your Azure username and password to the DataOps pipeline to connect to the Azure services. Setting the environment variables AZURE_USER and AZURE_PASSWORD achieves this.

We recommend that you keep your third-party credentials in the DataOps Vault. Storing them at the default paths AZURE.DEFAULT.USER and AZURE.DEFAULT.PASSWORD allows you to retrieve them by setting the environment variable SET_AZ_KEYS_TO_ENV.

Use the DATAOPS_VAULT() functionality to retrieve your credentials if you have stored your credentials at different vault paths.

"My Azure Job":
  variables:
    AZURE_USER: DATAOPS_VAULT(PATH.TO.USERNAME.IN.VAULT)
    AZURE_PASSWORD: DATAOPS_VAULT(PATH.TO.PASSWORD.IN.VAULT)

2. Inheriting permissions from the virtual machine

The Azure orchestrator also supports using the Virtual Machine's identity to connect to Azure. To utilize this feature, set the variable DATAOPS_USE_IDENTITY.

"My Azure Job":
  variables:
    DATAOPS_USE_IDENTITY: 1

3. Using a service principal

The Azure orchestrator also supports using a service principal to connect to Azure. To utilize this feature, set the variable DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL and provide the additional parameters.

"My Azure Job":
  variables:
    DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL: 1
    TENANT_ID: <tenant_id>
    AZURE_APP_ID: DATAOPS_VAULT(PATH.TO.APPLICATION_ID.IN.VAULT)
    AZURE_CLIENT_SECRET: DATAOPS_VAULT(PATH.TO.CLIENT_SECRET.IN.VAULT)
    DISABLE_ALLOW_NO_SUBSCRIPTIONS: 1

Supported parameters

Parameter Required/Default Description
AZURE_USER Optional Username for Azure
AZURE_PASSWORD Optional Password for Azure
DATAOPS_USE_IDENTITY Optional If set, uses inherited Azure permission from VM
SET_AZ_KEYS_TO_ENV Optional If set, it exports the Azure username and password from the DataOps Vault to the environment
DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL Optional If set, Azure sign-in will be done using a service principal
AZURE_APP_ID Optional The Application (client) ID associated with the service principal. It is required if DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL is set to 1.
AZURE_CLIENT_SECRET Optional The Client credentials associated with the service principal. It is required if DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL is set to 1.
DISABLE_ALLOW_NO_SUBSCRIPTIONS Optional If set, it disable the ALLOW_NO_SUBSCRIPTIONS. It is required if DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL is set to 1.
TENANT_ID Optional If set, it associates with a unique tenant ID. It is required if DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL is set to 1.
DATAOPS_AZURE_RETRY_ATTEMPTS Optional, defaults to 1 If set, configures the number or retries in case of connection failures.
DATAOPS_AZURE_RETRY_INTERVAL Optional, defaults to 10 If set, configures the number of seconds to wait between retries.

Example jobs

Here are a few example jobs that describe the most common use cases:

1. Run shell script in an Azure context

You can create shell scripts in your project repository, such as /scripts/myscript.sh, and run them from inside a job. For example:

pipelines/includes/local_includes/azure_jobs/my_azure_job.yml
"My Azure Job":
  extends:
    - .should_run_ingestion
    - .agent_tag
  stage: "Batch Ingestion"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    DATAOPS_USE_IDENTITY: 1
  script:
    - /scripts/myscript.sh
  icon: ${AZURE_ICON}

2. Run Azure CLI

It is more common to call an Azure CLI command directly than to run a shell script in an Azure context:

pipelines/includes/local_includes/azure_jobs/my_azure_cli_job.yml
"My Azure CLI job":
  extends:
    - .should_run_ingestion
    - .agent_tag
  stage: "Batch Ingestion"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    DATAOPS_USE_IDENTITY: 1
  script:
    - az vm list
  icon: ${AZURE_ICON}

3. Run Azure CLI with DataOps vault enabled

In most cases, you will need to leverage context from other jobs within a DataOps pipeline, and you need access to the DataOps.live vault. Therefore, the best practice is to include /dataops in your script tag before executing the Azure CLI. For example:

pipelines/includes/local_includes/azure_jobs/list_vms.yml
"List Azure Virtual Machines":
  extends:
    - .should_run_ingestion
    - .agent_tag
  stage: "Additional Configuration"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    DATAOPS_USE_IDENTITY: 1
  script:
    - /dataops
    - az vm list
  icon: ${AZURE_ICON}

Including /dataops allows you to read from, and write to, the vault.

4. List available storage containers

pipelines/includes/local_includes/azure_jobs/list_storage_containers.yml
"List Storage Containers":
  extends:
    - .agent_tag
  stage: "Additional Configuration"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    AZURE_USER: DATAOPS_VAULT(PATH.TO.USERNAME.IN.VAULT)
    AZURE_PASSWORD: DATAOPS_VAULT(PATH.TO.PASSWORD.IN.VAULT)
  script:
    - /dataops
    - az storage container list
  icon: ${AZURE_ICON}

5. Run Azure CLI using the service principal authorization

pipelines/includes/local_includes/azure_jobs/Sign_in_with_a_service_principal_using_Azure_CLI.yml
"Sign in with a service principal using Azure CLI":
  extends:
    - .agent_tag
  stage: "Additional Configuration"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL: 1
    TENANT_ID: <tenant_id>
    AZURE_APP_ID: DATAOPS_VAULT(PATH.TO.APPLICATION_ID.IN.VAULT)
    AZURE_CLIENT_SECRET: DATAOPS_VAULT(PATH.TO.CLIENT_SECRET.IN.VAULT)
    DISABLE_ALLOW_NO_SUBSCRIPTIONS: 1
  script:
    - /dataops
  icon: ${AZURE_ICON}