Skip to main content

DataOps Kubernetes Sizing Recommendations

Kubernetes Sizing Guidance

When running jobs the underlying runner requirements will vary based on what the jobs in your pipelines are doing.

We expect a typical pipeline workload to be more CPU intensive than memory. The exception will be jobs where you are processing large amounts of data as part of the job instead of orchestrating the processing of data with a 3rd party service. This can happen when you are using the Data Prep Orchestrator.

Our recommendations for the physical infrastructure still apply yet for cloud specific recommendations see below.

AWS Recommendation

On AWS, for a managed cluster we recommend an instance type of t3a.xlarge or better for cluster nodes. We recommend at least 4 Cores and 16GB of memory.

Azure Recommendation

On Azure Kubernetes Service we recommend a VM size of Standard_D4s_v3.