Azure Orchestrator

Enterprise

Image	$DATAOPS_AZURE_RUNNER_IMAGE

The Azure orchestrator includes the following functionality:

Responsive interaction with Microsoft Azure Services using the built-in Azure CLI tools
DataOps Vault functionality that allows scripts to retrieve variables from the vault
DataOps native tools that allow the development of custom scripts that interact with Azure
The following added tools:
- git
- curl
- ssh-client
- perl
- sshpass
- ansible
- unzip
- terraform

Usage

The first use case described here is typical for this orchestrator; that is, to start an Azure instance to perform a task in the pipeline:

pipelines/includes/local_includes/azure_jobs/my_azure_job.yml
"My Azure Job":
  extends:
    - .agent_tag
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  stage: "Batch Ingestion"
  variables:
    # use one of the following connection methods
    # identity inheritance
    DATAOPS_USE_IDENTITY: 1

    # or default vault expansion
    SET_AZ_KEYS_TO_ENV: 1

    # or custom vault expansion
    AZURE_USER: DATAOPS_VAULT(PATH.TO.USERNAME.IN.VAULT)
    AZURE_PASSWORD: DATAOPS_VAULT(PATH.TO.PASSWORD.IN.VAULT)

    # or service principal
    DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL: 1
    TENANT_ID: <tenant_id>
    AZURE_APP_ID: DATAOPS_VAULT(PATH.TO.APPLICATION_ID.IN.VAULT)
    AZURE_CLIENT_SECRET: DATAOPS_VAULT(PATH.TO.CLIENT_SECRET.IN.VAULT)
    DISABLE_ALLOW_NO_SUBSCRIPTIONS: 1
  script:
    - /dataops
    - az .. # your azure cli command
  icon: ${AZURE_ICON}

Additionally, the following use cases demonstrate how to connect to Azure from a DataOps pipeline:

Connecting to Azure

The Azure orchestrator supports different methods of connecting to Azure. Only ever use one!

1. Username and password

To use the Azure orchestrator, you must provide your Azure username and password to the DataOps pipeline to connect to the Azure services. Setting the environment variables AZURE_USER and AZURE_PASSWORD achieves this.

We recommend that you keep your third-party credentials in the DataOps Vault. Storing them at the default paths AZURE.DEFAULT.USER and AZURE.DEFAULT.PASSWORD allows you to retrieve them by setting the environment variable SET_AZ_KEYS_TO_ENV.

Use the DATAOPS_VAULT() functionality to retrieve your credentials if you have stored your credentials at different vault paths.

"My Azure Job":
  variables:
    AZURE_USER: DATAOPS_VAULT(PATH.TO.USERNAME.IN.VAULT)
    AZURE_PASSWORD: DATAOPS_VAULT(PATH.TO.PASSWORD.IN.VAULT)

2. Inheriting permissions from the virtual machine

The Azure orchestrator also supports using the Virtual Machine's identity to connect to Azure. To utilize this feature, set the variable DATAOPS_USE_IDENTITY.

"My Azure Job":
  variables:
    DATAOPS_USE_IDENTITY: 1

3. Using a service principal

The Azure orchestrator also supports using a service principal to connect to Azure. To utilize this feature, set the variable DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL and provide the additional parameters.

"My Azure Job":
  variables:
    DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL: 1
    TENANT_ID: <tenant_id>
    AZURE_APP_ID: DATAOPS_VAULT(PATH.TO.APPLICATION_ID.IN.VAULT)
    AZURE_CLIENT_SECRET: DATAOPS_VAULT(PATH.TO.CLIENT_SECRET.IN.VAULT)
    DISABLE_ALLOW_NO_SUBSCRIPTIONS: 1

Supported parameters

Parameter	Required/Default	Description
`AZURE_USER`	Optional	Username for Azure
`AZURE_PASSWORD`	Optional	Password for Azure
`DATAOPS_USE_IDENTITY`	Optional	If set, uses inherited Azure permission from VM
`SET_AZ_KEYS_TO_ENV`	Optional	If set, it exports the Azure username and password from the DataOps Vault to the environment
`DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL`	Optional	If set, Azure sign-in will be done using a service principal
`AZURE_APP_ID`	Optional	The Application (client) ID associated with the service principal. It is `required` if `DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL` is set to 1.
`AZURE_CLIENT_SECRET`	Optional	The Client credentials associated with the service principal. It is `required` if `DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL` is set to 1.
`DISABLE_ALLOW_NO_SUBSCRIPTIONS`	Optional	If set, it disable the ALLOW_NO_SUBSCRIPTIONS. It is `required` if `DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL` is set to 1.
`TENANT_ID`	Optional	If set, it associates with a unique tenant ID. It is `required` if `DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL` is set to 1.
`DATAOPS_AZURE_RETRY_ATTEMPTS`	Optional, defaults to 1	If set, configures the number or retries in case of connection failures.
`DATAOPS_AZURE_RETRY_INTERVAL`	Optional, defaults to 10	If set, configures the number of seconds to wait between retries.

Example jobs

Here are a few example jobs that describe the most common use cases:

1. Run shell script in an Azure context

You can create shell scripts in your project repository, such as /scripts/myscript.sh, and run them from inside a job. For example:

pipelines/includes/local_includes/azure_jobs/my_azure_job.yml
"My Azure Job":
  extends:
    - .should_run_ingestion
    - .agent_tag
  stage: "Batch Ingestion"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    DATAOPS_USE_IDENTITY: 1
  script:
    - /scripts/myscript.sh
  icon: ${AZURE_ICON}

2. Run Azure CLI

It is more common to call an Azure CLI command directly than to run a shell script in an Azure context:

pipelines/includes/local_includes/azure_jobs/my_azure_cli_job.yml
"My Azure CLI job":
  extends:
    - .should_run_ingestion
    - .agent_tag
  stage: "Batch Ingestion"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    DATAOPS_USE_IDENTITY: 1
  script:
    - az vm list
  icon: ${AZURE_ICON}

3. Run Azure CLI with DataOps vault enabled

In most cases, you will need to leverage context from other jobs within a DataOps pipeline, and you need access to the DataOps.live vault. Therefore, the best practice is to include /dataops in your script tag before executing the Azure CLI. For example:

pipelines/includes/local_includes/azure_jobs/list_vms.yml
"List Azure Virtual Machines":
  extends:
    - .should_run_ingestion
    - .agent_tag
  stage: "Additional Configuration"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    DATAOPS_USE_IDENTITY: 1
  script:
    - /dataops
    - az vm list
  icon: ${AZURE_ICON}

Including /dataops allows you to read from, and write to, the vault.

4. List available storage containers

pipelines/includes/local_includes/azure_jobs/list_storage_containers.yml
"List Storage Containers":
  extends:
    - .agent_tag
  stage: "Additional Configuration"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    AZURE_USER: DATAOPS_VAULT(PATH.TO.USERNAME.IN.VAULT)
    AZURE_PASSWORD: DATAOPS_VAULT(PATH.TO.PASSWORD.IN.VAULT)
  script:
    - /dataops
    - az storage container list
  icon: ${AZURE_ICON}

5. Run Azure CLI using the service principal authorization

pipelines/includes/local_includes/azure_jobs/Sign_in_with_a_service_principal_using_Azure_CLI.yml
"Sign in with a service principal using Azure CLI":
  extends:
    - .agent_tag
  stage: "Additional Configuration"
  image: $DATAOPS_AZURE_RUNNER_IMAGE
  variables:
    DATAOPS_AZURE_LOGIN_AS_SERVICE_PRINCIPAL: 1
    TENANT_ID: <tenant_id>
    AZURE_APP_ID: DATAOPS_VAULT(PATH.TO.APPLICATION_ID.IN.VAULT)
    AZURE_CLIENT_SECRET: DATAOPS_VAULT(PATH.TO.CLIENT_SECRET.IN.VAULT)
    DISABLE_ALLOW_NO_SUBSCRIPTIONS: 1
  script:
    - /dataops
  icon: ${AZURE_ICON}

Enterprise

Usage​

Connecting to Azure​

1. Username and password​

2. Inheriting permissions from the virtual machine​

3. Using a service principal​

Supported parameters​

Example jobs​

1. Run shell script in an Azure context​

2. Run Azure CLI​

3. Run Azure CLI with DataOps vault enabled​

4. List available storage containers​

5. Run Azure CLI using the service principal authorization​