DataOps Docker runner Prerequisites
DataOps Account
To get started, you'll need a DataOps account. If you don't have one, you can set one up by logging in to your Snowflake tenant and clicking on "Partner Connect," then select "DataOps" or contact us at support@dataops.live.
Physical Infrastructure
The DataOps Runner must be installed on a Linux server, host, or virtual machine in a location with Snowflake access and access for all the other systems/tools you need to connect to from DataOps jobs.
The exact nature of the server/VM is up to you and will differ between bare metal, AWS, or Azure.
Minimum production specifications:
- Ubuntu 20.04 (18.04 is possible but not recommended)
- 4 CPU cores
- 16GB RAM
- Minimum 50GB Disk/Storage (300GB recommended)
- As a guide, for most use cases an AWS
t3a.xlarge
(or equivalent) - A
sudo
user
Minimum PoC/Pilot specifications:
- Ubuntu 20.04 (18.04 is possible but not recommended)
- 2 CPU cores
- 8GB RAM
- Minimum 50GB Disk/Storage (300GB recommended)
- As a guide, for most use cases an AWS
t3a.large
(or equivalent) - A
sudo
user
Network access
This Server/Host requires outbound internet access, as a minimum to :
- The DataOps app at https://app.dataops.live
- Docker Hub at https://hub.docker.com/
- The applicable Snowflake instances at https://...prefix....snowflakecomputing.com/
See the DataOps Architecture doc for a general overview of system architecture and the DataOps Security and Governance Appendix for a detailed discussion on networking.
Docker
Docker should be installed following the instructions at the Docker site for your OS of choice.
- Docker for Ubuntu
We recommend not installing Docker via the default Ubuntu repository, as this is often quite old.
Then (if you didn't do it as part of the Docker installation instructions):
- Run
sudo usermod -aG docker $USER
(this allows you to run docker without being root) - Log out and log in
To test your docker install run:
docker run hello-world