Releases: Orchestrator Versions
Tech Target defines a software release as follows:
A release is the distribution of the final version or the newest version of a software application.
In the DataOps context, a release is the distribution of the latest version of existing features or new features added to our data product platform. There are two types of releases relevant to the DataOps Platform:
- Major release
- Minor release
Major release
A major release can introduce significant changes to existing functionality like the upgrade to the data product platform UI. Or/and it can introduce new features such as new orchestrators added to the existing orchestrators. It is indicated by a change in the first digit of the version number. For example, a major release moves from v4 (orchestrator version 4-stable) to v5 (orchestrator version 5-stable).
A major release can include breaking changes, requiring manual changes from customers to update their projects to run.
The following tags are always included with each major release:
- x-latest
- x-stable
For example, the -stable tag (4-stable) points to the latest stable orchestrator release. This is the recommended tag to use within projects, as it will track the versions released after they have been through the QA process.
On the other hand, the -latest tag (4-latest) points to the newest development orchestrator release. This release may have new functionality but has not yet been through the entire QA release cycle.
The -latest tag should not be used in production environments.
Standard customer changes
For customers updating to <new-major-version>, there are several always-required changes, such as:
- Re-pointing the customer project to the
v<major-version>Reference Project tag
The above can be achieved as a standard development process. In other words, start in a feature branch, test, merge through dev, qa, and finally into production.
Release-specific customer changes
Every new major release must have a documentation page called <major-version> to <new-major-version> Upgrade Guide, detailing how customers must update their projects.
DataOps runner
The DataOps Runner, represented by the dataops-runner container, is a long-running container that connects to the data product platform and looks for jobs to execute.
This runner is versioned alongside the data product platform and provides the link between the platform and the other orchestrators. Therefore, you should always use this runner's latest tag version.
The only change that can't be made using the standard development process is updating the dataops-runner image. However, you can easily achieve this by deploying a separate runner image (on the same host or a different host) and assigning it to the branches by changing to /pipelines/includes/config/agent_tag.yml in the feature branch and then promoting it through the stack. Each time the branch is promoted, it starts to use the new runner. Once the changes reach main and a safe tie has elapsed, you can remove the old runner.
Major release guide
Minor release
A minor release is juxtaposed to a major release in that it introduces small (minor) improvements or upgrades to the data product Platform's functionality that are backward compatible and non-breaking.
The rule of thumb is that a minor release should not include new features. However, they can be added, ensuring that they do not break existing functionality. But this release type usually just includes modifications made to existing features.
The minor release is indicated by a change in any of the numbers below the first digit, for example, from 4.17.1 to 4.17.2 or 4.18.1.
Customer changes for orchestrator major and minor releases
Customers do not have to make any changes to receive updates. Updates to <major-version>-stable functional orchestrators will automatically be deployed if the pull policy includes "always."
In other words, the DataOps Runner will always look for new images with the same tag as shown in the following code:
concurrent = 16
check_interval = 10
[session_server]
session_timeout = 1800
[[runners]]
name = "...."
url = "https://app.dataops.live"
token = "...."
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.docker]
pull_policy = ["always", "if-not-present"]
tls_verify = false
image = "dataopslive/dataops-utils-runner"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/srv/dataops-runner-truedataops5-bdl/local_config:/local_config", "/srv/dataops-runner-truedataops5-bdl/agent_cache:/agent_cache", "/srv/dataops-runner-truedataops5-bdl/secrets:/secrets", "/var/run/docker.sock:/var/run/docker.sock", "/cache"]
shm_size = 0
dataops-runner:<major-version>-stable
dataops-runner:<major-version>-stableThe dataops-runner:<major-version>-stable upgrade is effectively run manually (unless setup using Docker Compose or something else client-defined). There is no automated way of forcing a new version of dataops-runner:<major-version>-stable to be pulled and used.
This is rarely an issue since this image is straightforward and rarely changes, but if it is ever needed, this is achieved in the following way:
docker pull dataopslive/dataops-runner:<major-version>-stable
docker restart <runner-name>
DataOps.live changes
- Reference projects don't need to change since they use the
<major-version>-stablerunner tag. - Within the Reference Project, the
v<major-version>repo tag remains stable - Template projects don't need to change since they reference the
v<major-version>Reference Project tag - QA projects don't need to change since they reference the
v<major-version>Reference Project tag - Demo projects don't need to change since they reference the
v<major-version>Reference Project tag - Vertically-integrated application template projects don't need to change since they reference the
v<major-version>Reference Project tag