Skip to main content

Fivetran Orchestrator

TypePre-Set
Image$DATAOPS_FIVETRAN_RUNNER_IMAGE

The Fivetran Orchestrator is a preset orchestrator, allowing a DataOps pipeline to interact with Fivetran as a job in a DataOps pipeline, triggering a Fivetran operation such as ingesting data from a Fivetran pipeline.

Usage

A typical Fivetran Orchestrator workflow is as follows:

  • If FIVETRAN_ACTION has one of STATUS, START, TEST or FORCE_RESYNC, one of the following actions occurs:

    • Update the connector schedule_type to manual if FIVETRAN_DONT_DISABLE_SYNC has no value
    • If set to STATUS, print the JSON response in the console
    • If set START, log the connector's final status
    • If set to TEST, log the summary of tests passed or failed
    • If set FORCE_RESYNC, log the connector's final status

Connector ID

Fetch the Fivetran Connector ID for the Fivetran Orchestrator to work. To do so:

  • Go to your connector instance in the Fivetran User Interface (UI)):

    Connectors

  • Search for an event of type api_call:

    API-Call

  • The string after the standard-config is the Connector ID (shrank_paparazzi in the example given).

See the Fivetran REST API Group Management Docs for more information.

Supported Parameters

ParameterRequired or Optional (and default)Description
FIVETRAN_ACTIONREQUIREDMust be one of STATUS, START, TEST or FORCE_RESYNC
CONNECTOR_IDREQUIREDIdentifier for the connector. See the Get Connector ID section for fetching the connector ID
SET_FIVETRAN_KEYS_TO_ENVOptionalIf a value is set to this variable, fetches values for FIVETRAN_API_KEY and FIVETRAN_API_SECRET from vault with vault keys FIVETRAN.DEFAULT.API_KEY and FIVETRAN.DEFAULT.SECRET respectively
FIVETRAN_API_KEYREQUIRED unless SET_FIVETRAN_KEYS_TO_ENV is setFivetran API access token (see the Fivetran docs)
FIVETRAN_API_SECRETREQUIRED unless SET_FIVETRAN_KEYS_TO_ENV is setFivetran API access token (see the Fivetran docs)
TABLEREQUIRED in FORCE_RESYNC actionName of the table to force re-sync
SCHEMAREQUIRED in FORCE_RESYNC actionName of the schema to force re-sync
FIVETRAN_DONT_DISABLE_SYNCOptionalIf a value is set to this variable, schedule_type will not updated to manual

Example Jobs

The following four examples describe the most common uses of the Fivetran Orchestrator:

1. Check the Connector Status

This example sends a request to the Fivetran pipeline to check its connector status and prints the JSON response on the console.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_status.yml
"Fivetran Connector Status":
extends:
- .agent_tag
stage: "Check Status"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: STATUS
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}

2. Start a Connector

This example sends a request to Fivetran to start the specified connector.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_start.yml
"Fivetran Connector Start":
extends:
- .agent_tag
stage: "Start Connector"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: START
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}

3. Test the Connector

This example sends a request to Fivetran to test the specified connector.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_test.yml
"Fivetran Connector Test":
extends:
- .agent_tag
stage: "Test Connector"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: TEST
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}

4. Force Resync of a Specific Table

This example sends a request to Fivetran to force resync the table specified in the YAML config file.

pipelines/includes/local_includes/fivetran-jobs/fivetran_force-resync.yml
"Fivetran Connector Force Resync":
extends:
- .agent_tag
stage: "Force Resync Table"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: FORCE_RESYNC
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
TABLE: <table name>
SCHEMA: <schema name>
script:
- /dataops
icon: ${FIVETRAN_ICON}

Project Resources

None

Host Dependencies (and Resources)

None