Skip to main content

Python2 Orchestrator

TypeFlexible
Image$DATAOPS_PYTHON2_RUNNER_IMAGE
Feature Status

The Python2 Orchestrator is a flexible orchestrator and includes Python2 as well as the following libraries:

  • requests
  • responses
  • NumPy
  • pandas

Usage

This orchestrator is used to create scripts that run jobs. Therefore, you can create your Python script inside your DataOps project, such as python2 my-script.py, and run it from a job. In order to pass variables into the job, set them in the YAML config file's variables block. For example:

pipelines/includes/local_includes/python2_jobs/my_python2_job.yml
"My Python2 Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_PYTHON2_RUNNER_IMAGE
variables:
SCRIPT_PATH: $CI_PROJECT_DIR/scripts/my-script.py
script: python2 my-script.py
icon: ${PYTHON_ICON}

Supported Parameters

ParameterRequired or Optional (and default)Description
DATAOPS_RUN_PYTHON_SCRIPTOPTIONALProject path to a Python2 script
FILEOPTIONAL - defaults to requirements.txtName of the file used to run pip install
SPECIFIED_PATHOPTIONAL - defaults to $CI_PROJECT_DIRPath to requirements.txt file

Example Jobs

The following example shows how to execute a Python2 script:

pipelines/includes/local_includes/python2_jobs/execute_python2_script.yml
"Execute Python2 Script":
extends:
- .agent_tag
stage: "Additional Configuration"
image: $DATAOPS_PYTHON2_RUNNER_IMAGE
variables:
SCRIPT_PATH: $CI_PROJECT_DIR/scripts/my-script.py
script: python2 $SCRIPT_PATH
icon: ${PYTHON_ICON}

In order to retrieve the secrets PATH.TO.VAR1 and PATH.TO.VAR2 from the DataOps Vault, add them as variables to the YAML config file as follows:

pipelines/includes/local_includes/python2_jobs/extract_from_vault.yml
"Extract From Vault":
extends:
- .agent_tag
image: $DATAOPS_PYTHON2_RUNNER_IMAGE
stage: "Additional Configuration"
variables:
DATAOPS_RUN_PYTHON_SCRIPT: $CI_PROJECT_DIR/scripts/my-script.py
VAR1: DATAOPS_VAULT(PATH.TO.VAR1)
VAR2: DATAOPS_VAULT(PATH.TO.VAR2)
script:
- /dataops
icon: ${PYTHON_ICON}

Project Resources

None

Host Dependencies (and Resources)

None