Python2 Orchestrator

Type	Flexible
Image	$DATAOPS_PYTHON2_RUNNER_IMAGE
Feature Status	EoL

The Python2 orchestrator is a flexible orchestrator and includes Python2 as well as the following libraries:

• requests
• responses
• NumPy
• pandas

Usage

This orchestrator is used to create scripts that run jobs. Therefore, you can create your Python script inside your DataOps project, such as python2 my-script.py, and run it from a job. In order to pass variables into the job, set them in the YAML config file's variables block. For example:

pipelines/includes/local_includes/python2_jobs/my_python2_job.yml

"My Python2 Job":
  extends:
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_PYTHON2_RUNNER_IMAGE
  variables:
    SCRIPT_PATH: $CI_PROJECT_DIR/scripts/my-script.py
  script: python2 my-script.py
  icon: ${PYTHON_ICON}

Supported parameters

Parameter	Required or Optional (and default)	Description
DATAOPS_RUN_PYTHON_SCRIPT	Optional	Project path to a Python2 script
FILE	Optional. Defaults to requirements.txt	Name of the file used to run pip install
SPECIFIED_PATH	Optional. Defaults to $CI_PROJECT_DIR	Path to requirements.txt file

Example jobs

The following example shows how to execute a Python2 script:

pipelines/includes/local_includes/python2_jobs/execute_python2_script.yml

"Execute Python2 Script":
  extends:
    - .agent_tag
  stage: "Additional Configuration"
  image: $DATAOPS_PYTHON2_RUNNER_IMAGE
  variables:
    SCRIPT_PATH: $CI_PROJECT_DIR/scripts/my-script.py
  script: python2 $SCRIPT_PATH
  icon: ${PYTHON_ICON}

To retrieve the secrets PATH.TO.VAR1 and PATH.TO.VAR2 from the DataOps Vault, add them as variables to the YAML config file as follows:

pipelines/includes/local_includes/python2_jobs/extract_from_vault.yml

"Extract From Vault":
  extends:
    - .agent_tag
  image: $DATAOPS_PYTHON2_RUNNER_IMAGE
  stage: "Additional Configuration"
  variables:
    DATAOPS_RUN_PYTHON_SCRIPT: $CI_PROJECT_DIR/scripts/my-script.py
    VAR1: DATAOPS_VAULT(PATH.TO.VAR1)
    VAR2: DATAOPS_VAULT(PATH.TO.VAR2)
  script:
    - /dataops
  icon: ${PYTHON_ICON}