AWS Orchestrator
Enterprise¶
| Image | $DATAOPS_AWS_RUNNER_IMAGE |
|---|
The AWS orchestrator includes the following features:
- The ability to interact with the AWS services using the built-in AWS CLI tools provides a wide range of features focused on infrastructure automation
- DataOps Vault functionality that allows scripts to retrieve variables from the vault
- DataOps native tools that allow the development of custom scripts that interact with AWS
- The following additional tools:
- git
- curl
- ssh-client
- perl
- sshpass
- unzip
- terraform
Usage¶
The first use case described here is typical for this orchestrator; that is, to start an EC2 instance to perform a task in the pipeline:
"My AWS Job":
extends:
- .should_run_ingestion
- .agent_tag
stage: "Batch Ingestion"
image: $DATAOPS_AWS_RUNNER_IMAGE
variables:
# use one of the following connection methods
# EC2 instance role inheritance - no variables to set / default
#
# or default vault expansion for from vault keys `AWS.DEFAULT.S3_KEY` and `AWS.DEFAULT.S3_SECRET`
SET_AWS_KEYS_TO_ENV: 1
# or custom vault expansion for access key / secret
AWS_ACCESS_KEY_ID: DATAOPS_VAULT(PATH.TO.ACCESS_KEY_ID.IN.VAULT)
AWS_SECRET_ACCESS_KEY: DATAOPS_VAULT(PATH.TO.SECRET_ACCESS_KEY.IN.VAULT)
script:
- /dataops
- aws ... # your AWS CLI command
icon: ${AWS_ICON}
Additionally, the following use cases demonstrate how to connect to AWS from a DataOps pipeline:
Connecting to AWS¶
To connect to AWS, the AWS orchestrator supports the following five methods:
1. Inheriting the IAM role from the DataOps runner's EC2 instance¶
No additional configuration is necessary for the AWS orchestrator to use the AWS EC2 instance IAM role on which the DataOps runner is deployed. This behavior is the default behavior and what we recommend.
2. AWS access key ID and secret access key¶
Instead, if you are required to use the AWS access key ID and secret access key, provide them to the DataOps pipeline by setting the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
Note
Additional environment variables may be required for your use case. For more information, refer to the AWS CLI documentation on environment variables.
3. Using the standard DataOps vault credentials¶
When you want to use the standard DataOps Vault credentials, the following details apply:
- The vault path details are stored in
AWS.DEFAULT.S3_KEYandAWS.DEFAULT.S3_SECRET - Use the pipeline variable
SET_AWS_KEYS_TO_ENVto retrieve the values of these two keys from the vault - The scope of these exposed environment variables is limited to the entry point
/dataops - All the scripts that run from the
/dataopsdirectory also fall within this scope
Note
If you need access to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from outside /dataops, you must reimport them by retrieving them from the DATAOPS_SOURCE_FILE. For example:
4. Using custom DataOps vault credentials¶
When you want to use your custom DataOps Vault credentials, use the DATAOPS_VAULT() function to retrieve credentials stored in a different vault path. For example:
variables:
AWS_ACCESS_KEY_ID: DATAOPS_VAULT(PATH.TO.ACCESS_KEY_ID.IN.VAULT)
AWS_SECRET_ACCESS_KEY: DATAOPS_VAULT(PATH.TO.SECRET_ACCESS_KEY.IN.VAULT)
5. Exchanging an OIDC token for temporary credentials¶
A job can assume an IAM role with no stored key. With id_tokens, DataOps.live gives the job a signed OpenID Connect (OIDC) token, which the job exchanges with AWS STS for the role's temporary credentials.
In AWS, create an IAM OIDC identity provider with the provider URL https://app.dataops.live and the audience sts.amazonaws.com. Then create a role that trusts it. In a branch pipeline, the token's sub claim is project_path:<group>/<project>:ref_type:branch:ref:<branch>, so the trust policy can admit one branch of one project:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<account-id>:oidc-provider/app.dataops.live"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"app.dataops.live:aud": "sts.amazonaws.com",
"app.dataops.live:sub": "project_path:<group>/<project>:ref_type:branch:ref:main"
}
}
}
]
}
To admit every branch, use StringLike with project_path:<group>/<project>:ref_type:branch:ref:*.
The job exchanges the token in its own script::
"My AWS OIDC Job":
extends:
- .agent_tag
stage: "Batch Ingestion"
image: $DATAOPS_AWS_RUNNER_IMAGE
id_tokens:
AWS_OIDC_TOKEN:
aud: sts.amazonaws.com
variables:
ROLE_ARN: arn:aws:iam::<account-id>:role/<role-name>
AWS_DEFAULT_REGION: <region>
script:
- |
credentials=$(aws sts assume-role-with-web-identity --role-arn "$ROLE_ARN" \
--role-session-name "dataops-$CI_PIPELINE_ID" --web-identity-token "$AWS_OIDC_TOKEN" \
--query Credentials --output json)
export AWS_ACCESS_KEY_ID=$(jq -r .AccessKeyId <<< "$credentials")
export AWS_SECRET_ACCESS_KEY=$(jq -r .SecretAccessKey <<< "$credentials")
export AWS_SESSION_TOKEN=$(jq -r .SessionToken <<< "$credentials")
unset credentials
aws s3 ls s3://<bucket>/
icon: ${AWS_ICON}
The credentials are environment variables of the job's shell and expire when the AWS role session ends. Don't write them to DATAOPS_SOURCE_FILE, to a file, or to the log.
Troubleshooting¶
When using the secrets manager with a DataOps Runner deployed on AWS EC2 review the necessary Instance Metadata Service Version 2 (IMDSv2) configuration changes.
Supported parameters¶
| Parameter | Required/Default | Description |
|---|---|---|
SET_AWS_KEYS_TO_ENV |
Optional | If set, export the AWS access key ID AWS_ACCESS_KEY_ID) and secret access key AWS_SECRET_ACCESS_KEY by retrieving it from the DataOps Vault keys AWS.DEFAULT.S3_KEY and AWS.DEFAULT.S3_SECRET respectively. |
Example jobs¶
You can create scripts that wrap your AWS usage in your project repository like /scripts/myawsscript.sh. Then run the script from inside your job. For example:
"My AWS Job":
extends:
- .should_run_ingestion
- .agent_tag
stage: "Batch Ingestion"
image: $DATAOPS_AWS_RUNNER_IMAGE
variables:
script:
- /scripts/my_aws_script.sh
icon: ${AWS_ICON}
For single AWS interactions, you can call an AWS CLI command directly:
"My AWS CLI job":
extends:
- .should_run_ingestion
- .agent_tag
stage: "Batch Ingestion"
image: $DATAOPS_AWS_RUNNER_IMAGE
variables:
script:
- aws s3 ls s3://mybucket/
icon: ${AWS_ICON}
If you need access to a vault and other DataOps features, include /dataops in your script tag. For example:
My AWS Job:
extends:
- .should_run_ingestion
- .agent_tag
stage: "Batch Ingestion"
image: $DATAOPS_AWS_RUNNER_IMAGE
variables:
script:
- /dataops
- scripts/myawsscript.sh
icon: ${AWS_ICON}
Lastly, use the following YAML file when you need to use the AWS Access Key ID and Secret Access Key to connect:
"My AWS Job":
extends:
- .should_run_ingestion
- .agent_tag
stage: "Batch Ingestion"
image: $DATAOPS_AWS_RUNNER_IMAGE
variables:
- SET_AWS_KEYS_TO_ENV: 1
- DATAOPS_SOURCE_FILE: $CI_PROJECT_DIR/env.sh
script:
- /dataops
- source $DATAOPS_SOURCE_FILE
- scripts/myawsscript.sh
icon: ${AWS_ICON}