DataOps Sample Development Workflow
As DataOps follows the core principles of dev-ops, a vital part of that is the development workflow, promoting code through the standard environments before the final release into production.
This is a simplified, idealized workflow based on the most typical implementations of DataOps. You may wish to incorporate DataOps processes into their existing development workflows, so this sample procedure may not match your account's configuration exactly.
Whether adding a feature or fixing a bug, development activity usually starts with a feature branch*, created from the main development branch (typically dev* in DataOps).
- Create a feature branch from dev - often done by opening the dev branch in the WebIDE and then committing the changes to a new branch.
- Develop and test in the feature branch, committing as needed.
- When the code is ready, create a Merge Request (MR) from the feature branch into dev.
- The development team reviews the MR and, if they're happy with the code and test results, merges it into dev, deleting the feature branch.
- Perform integration testing of the new code - this often starts with a pipeline run immediately after the MR is merged. Other features/fixes will usually be included in this process.
- When a set of development activities are ready, usually at the end of a development cycle (e.g., sprint), an MR is raised to promote the new code into the Quality Assurance environment (qa branch).
Quality assurance environment
- The QA team will review the MR's code and tests before merging it into their qa branch if acceptable. Otherwise, there is usually some kind of rework cycle between the QA and development teams to resolve any issues.
- Quality assurance test procedures are run. This may include existing unit tests, but dedicated QA test routines will often use separate infrastructure. This can be included as part of standard DataOps pipelines using configured jobs to only run in the qa branch.
- Once all QA testing is passed, the new code is ready for release into production. A merge request will be raised from the qa branch into master.
- Production operators will review the MR and schedule the release if the code and QA results meet the required production criteria. As the release schedule dictates, the MR will be merged into production-ready for the next pipeline run.