Skip to main content

DataOps Template Rendering

Template Rendering is a software technology that takes a template built using a template language like Jinja and injects (or passes) data into it. The result is a piece of content such as a static web application, text document, or configuration file. It is a good way to substitute values in template or configuration files and forms an integral part of the DataOps platform architecture, as follows:

Therefore, it is worth looking at the principles of Template Rendering to understand how DataOps leverages its power.

Jinja Template Rendering

DataOps uses Jinja Template Rendering to inject data into YAML configuration files and for reporting purposes.

How?

Templates are developed using the Jinja Templating Language. In fact, a Jinja template is simply a text file containing variables, expressions, and tags. When the template is rendered, these variables and expressions get replaced with values, while the tags control the template's logic.

For instance, the Jinja Template Designer Documentation uses the following example to show what a simple template looks like:

<!DOCTYPE html>
<html lang="en">
<head>
<title>My Webpage</title>
</head>
<body>
<ul id="navigation">
{% for item in navigation %}
<li><a href="{{ item.href }}">{{ item.caption }}</a></li>
{% endfor %}
</ul>

<h1>My Webpage</h1>
{{ a_variable }}

{# a comment #}
</body>
</html>

Jinja Basics

Now that we understand what Jinja looks like, let's take a deep dive into some of the Jinja basics.

As highlighted in the Jinja documentation, Jinja is a templating language similar to Python. It is primarily used in MATE to write functional SQL, as well as other functions, including documentation, macros, and packages.

While the best place to learn Jinja is its Jinja Template Designer documentation, it is important to review the basics that are most often used in MATE.

Delimiters

There are three delimiters to be aware of:

1. {% %} statement delimiters

This is used to denote statements. And, they are used for any functional programming such as starting and ending loops, if-else statements, and setting variables.

For instance:

  • A variable is set as follows:
{% set animals = ['horse', 'cat', 'dog'] %}
  • The for loop construct is as follows:
{% set animals = ['horse', 'cat', 'dog'] %}

{% for animal in animals %}
loop through animal list
{% endfor %}

  • The if-else construct is as follows:
{% if i = 1 %}
do something
{% else %}
do something else
{% endif %}

2. {{ }} expression delimiters

This delimiter contains expressions that are printed to the rendered file. For instance:

{% set my_companion = 'Dutch Shepherd'] %}

I have a companion dog. It is my {{ my_companion }}.

When this script is rendered, it reads as follows:

I have a companion dog. It is my Dutch Shepherd.

3. {# #} comment delimiters

This delimiter is used to denote comments. For example:

{# this script describes my companion #}

{% set my_companion = 'Dutch Shepherd'] %}

I have a companion dog. It is my {{ my_companion }}.

Jinja Features

The most common features used in Jinja are dictionaries, lists, for loops, if-else statements, and macros. Let's look at each feature individually:

1. Dictionaries

Dictionaries in Jinja are data structures comprising key-value pairs. The best way to describe a dictionary is to look at the following code snippet:

{% set customer = {
'customer_id': 'A001',
'name': 'My Name',
'address': 'This is my address'
} %}

The data in this data structure is then accessed as follows:

{{ customer.customer_id }}

A001

{{ customer.name }}

My Name

{{ customer.address }}

This my address

2. Lists

Lists are similar to dictionaries in that they are also data structures. However, they also differ in that they are ordered and indexed by integers, starting from 0.

For instance:

{% set my_pronouns = ['me', 'myself'] %}

{{ my_pronouns[0] }}

me

{{ my_pronouns[1] }}

myself

3. if-else statements

Even though the if-else construct has been highlighted in the [delimiters][jinjaDelimiters] section, it is worth taking another look.

if-else statements are essentially control statements that allow the system to make decisions based on clear criteria. For instance:

{% set color = 'blue' %}

{% if color = 'blue' %}
{{ The sky is blue }}
{% else %}
{{ The grass is green }}
{% endif %}

Because the variable color is set to 'blue' at the top of this script, it will always render as

The sky is blue

However, if we modify this if-else statement to use elif (else if), the script might render differently.

{% set color = 'blue' %}

{% if color = 'blue' %}
{{ The sky is blue }}
{% elif color = 'green' %}
{{ The grass is green }}
{% else %}
{{ There is no color at all! }}
{% endif %}

The expected outcomes are as follows:

  • If color is set to 'blue', the template will render as The sky is blue
  • If color is set to 'green', the template will render as The grass is green
  • If color is null or has no value, the template will render as There is no color at all!

4. for loops

For loops provide the functionality where the template renderer will repeat a code block while passing in different values each time it iterates through the loop.

For example:

{% set products = ['chocolate', 'ice cream', 'candy'] %}

{% for product in products %}
My favorite food is {{ product }}
{% endfor %}

This template will render as follows:

My favorite food is chocolate

My favorite food is ice cream

My favorite food is candy

5. Macros

In Jinja (and MATE), a macro is a way to write functions, similar to functions in any other programming language. We can use macros to group together a set of statements and then reference these statements throughout our codebase.

For instance:

{% macro my_color_products(color, product = 'toys') %}

I stock {{ color }} {{ product }} on my eCommerce store.

{% endmacro %}

Once the macro has been written, the next step is to use it. In this sample scenario, I am using this macro to describe the color products in my online store. The template and rendered text are as follows:

{{ my_color_products(red) }}

I stock red toys on my eCommerce store.

{{ my_color_products(blue, trains) }}

I stock blue trains on my eCommerce store.

6. Packages

Packages form a vital component for importing models, macros, and test suites into your DataOps project, especially in MATE. See the MATE docs for more information on using packages in DataOps.

Whitespace Control

Whitespace control is a vital part of the Jinja templating language. By default, Jinja renders its output with large blocks of whitespace between each line of code. For example, looking at the following code snippet; we will see that when compiled, the output includes big blocks of whitespace between each line of code.

{% set animals = ['tiger', 'snake', 'lion', 'leopard'] %}

{% for animal in animals %}
{% if animal = 'snake' %}
{% set animal_type = 'reptile' %}
{% else %}
{% set animal_type = 'mammal' %}
{% endif %}

The {{ animal}} is a {{ animal_type }}.
{% endfor %}

This template will actually render as follows:

The tiger is a mammal.









The snake is a reptile.









The lion is a mammal.









The leopard is a mammal.

This is because when Jinja compiles and evaluates each of the blocks it needs to evaluate, it will include a whitespace line for every line inside the Jinja code even if it doesn't print anything. Therefore, because there are eight lines of Jinja code, there will be eight lines of whitespace between each line of rendered code.

How do we fix this?

We add minus signs (-) next to the % signs in the control blocks, eliminating the whitespace immediately before and after each control block.

For example:

{%- set animals = ['tiger', 'snake', 'lion', 'leopard'] -%}

{% for animal in animals -%}
{%- if animal = 'snake' -%}
{%- set animal_type = 'reptile' -%}
{%- else -%}
{%- set animal_type = 'mammal' -%}
{%- endif -%}

The {{ animal}} is a {{ animal_type }}.
{% endfor %}

The Jinja template will now render as follows:

The tiger is a mammal.

The snake is a reptile.

The lion is a mammal.

The leopard is a mammal.
note

Removing whitespace is quite tricky. Therefore, it's a case of trial and error.

Template Rendering: The DataOps Way

Template files are identified by inserting the word .template into the filename, typically before the extension (e.g. config.template.yml). However, some older templates may still use a suffix format (e.g., config.yml.template).

note

The suffix format is now deprecated and should not be used in new projects.

Let's now look at two examples of how Template Rendering is used in DataOps:

1. YAML Config Files

As described in the DataOps Reference Project, storing YAML templates in the repository and rendering them at runtime using variables from the DataOps vault or calculated in an earlier job is a critical function of the overall DataOps platform.

For instance, let's assume that your login credentials are needed in a YAML config file to run a DatOps pipeline job. However, it is not a good idea (or DataOps best practice) to add these credentials to the YAML file at development time, posing a challenge of how to access these credentials at runtime.

The best solution is to develop a YAML template file with placeholder variables, extract these credentials from the DataOps Vault at runtime, and inject them into the template file at the beginning of the job, allowing the job to successfully run while preserving the integrity and the security of this data.

The code snippet for this template file is as follows:

/secrets/login_credentials.template.yml
LOGIN_CREDENTIALS:
USERNAME: {{ PERSON.NAME }}
PASSWORD: {{ PERSON.PASSWORD }}

The resulting YAML config file, after it has been rendered using our Template Rendering process, will look like this:

/secrets/login_credentials.yml
LOGIN_CREDENTIALS:
USERNAME: John
PASSWORD: John's Password

In some cases, having both the template file and the resulting processed file may cause issues, particularly if an application scans a directory and processes all files found. In this case, the variable DATAOPS_REMOVE_RENDERED_TEMPLATES can be set, which causes the template renderer to remove the template once it has been processed.

2. Static Web Application

DataOps also uses template rendering for static web page generation. In summary, report output files are generated by different DataOps pipeline jobs and rendered as static web applications at runtime by the DataOps platform.

Jobs leverage standard DataOps variables like CI_JOB_ID and CI_JOB_NAME to generate output as an artifact or static web application. See the Job Artifacts doc page for more information.

info

The output does not have to be generated by the same process that renders the report. The template rendering process also can render output generated by other jobs/processes.

To expand on this topic, let's look at how DataOps renders the HTML page index.html at runtime by combining the template titled index.template.html and a CSV output file.

This example does not include the output file generation process. It picks up where the values are injected into the placeholder variables in the index.template.html template file as follows:

index.template.html
<!DOCTYPE html>
<html lang="en">
<head>
<title>{{ env.CI_JOB_NAME }}</title>
<meta name="author" content="DataOps">
<meta content="Display any CSV file as a searchable, filterable, pretty HTML table">
</head>
<body>

<h1>All about {{ env.CI_JOB_NAME }}</h1>

<p>The main content of the static web page gets generated from the input file <b>data/{{ env.CI_JOB_ID }}_output.out</b></p>
</body>
</html>

Assuming we used this template in a DataOps job named My Pipeline Results and the pipeline execution assigned the JOB_ID as 47845454 to this DataOps job, the rendered output is as the following HTML:

index.html
<!DOCTYPE html>
<html lang="en">
<head>
<title>My Pipeline Results</title>
<meta name="author" content="DataOps">
<meta content="Display any CSV file as a searchable, filterable, pretty HTML table">
</head>
<body>

<h1>All about My Pipeline Results</h1>

<p>The main content of the static web page gets generated from the input file <b>data/47845454_output.out</b></p>
</body>
</html>

Or as web page in the browser as:

rendered-index-html