Skip to main content

DataOps Docker Runner Prerequisites

DataOps.live account​

To get started, you'll need a DataOps account. If you don't have one, you can set one up by logging in to your Snowflake tenant, clicking on Partner Connect, and then selecting DataOps. Alternatively, contact our Support team.

Snowflake account​

Each DataOps pipeline accesses Snowflake using a service account user, which defaults to user DATAOPS_SOLE_ADMIN and role DATAOPS_SOLE_ADMIN_ROLE. The required privileges will vary based on your requirements, yet ensure you review the required Snowflake privileges before proceeding.

Physical infrastructure​

Install the DataOps runner on a Linux server, host, or virtual machine in a location with Snowflake access and access to all the other systems/tools you need to connect to from DataOps jobs.

The exact nature of the server/virtual machine is up to you and will differ between bare metal, AWS, or Azure.

Minimum production specifications:

  • 4 CPU cores
  • 16GB RAM
  • Minimum 50GB Disk/Storage (300GB recommended)
  • As a guide, for most use cases, an AWS t3a.xlarge (or equivalent) which uses EBS-only storage by default
  • A sudo user

Minimum PoC/Pilot specifications:

  • 2 CPU cores
  • 8GB RAM
  • Minimum 50GB Disk/Storage (300GB recommended)
  • As a guide, for most use cases, an AWS t3a.large (or equivalent) which uses EBS-only storage by default
  • A sudo user

Network access​

This server/host requires outbound internet access, as a minimum, to:

See the DataOps architecture documentation for a general overview of the system architecture and the DataOps Security and Compliance section for a detailed discussion on networking.

Docker Linux distribution​

Install Docker following the instructions at the Docker site for your operating system of choice.

Ubuntu

We recommend not installing Docker via the default Ubuntu repository, as this is often quite old.

Then (if you didn't do it as part of the Docker installation instructions):

  • Run sudo usermod -aG docker $USER (this allows you to run docker without being root)
  • Log out and log in

To test your docker install, run the hello-world container:

docker run hello-world

Secret manager access​

The instance you will create requires access to read from a secret manager.

For more information, see the supported secrets managers in our Secret Manager orchestrator documentation and the Azure official documentation about Azure Key Vault Provider.

Last updated on