DataOps Kubernetes Sizing Recommendations

Kubernetes sizing guidance

When running jobs, the underlying runner requirements will vary based on what the jobs in your pipelines are doing.

We expect a typical pipeline workload to be more CPU intensive than memory. The exception will be jobs where you are processing large amounts of data as part of the job instead of orchestrating data processing with a 3rd party service. The Data Prep Orchestrator is one example of a memory-bound orchestrator.

Our recommendations for the physical infrastructure still apply, yet for cloud-specific recommendations, see below.

AWS recommendation

On AWS, for a managed cluster, we recommend an instance type of t3a.xlarge or better for cluster nodes. We recommend at least 4 Cores and 16GB of memory.

Azure recommendation

On Azure Kubernetes Service, we recommend a VM size of Standard_D4s_v3.