Skip to main content

Fivetran Orchestrator

TypePre-Set
Image$DATAOPS_FIVETRAN_RUNNER_IMAGE

The Fivetran orchestrator is a preset orchestrator, allowing a DataOps pipeline to interact with Fivetran as a job in a DataOps pipeline, triggering a Fivetran operation such as ingesting data from a Fivetran pipeline.

Usage

A typical Fivetran orchestrator workflow is as follows:

  • If FIVETRAN_ACTION has one of STATUS, START, TEST or FORCE_RESYNC, one of the following actions occurs:

    • Update the connector schedule_type to manual if FIVETRAN_DONT_DISABLE_SYNC has no value
    • If set to STATUS, print the JSON response in the console
    • If set START, log the connector's final status
    • If set to TEST, log the summary of tests passed or failed
    • If set FORCE_RESYNC, log the connector's final status

Connector ID

Fetch the Fivetran Connector ID for the Fivetran orchestrator to work. To do so:

  1. Go to your connector instance in the Fivetran User Interface (UI).

    Fivetran Connectors Log screenshot __shadow__

  2. Search for an event of type api_call.

    Fivetran log search for the keyword api_call __shadow__

The string after the standard-config is the Connector ID, shrank_paparazzi in the example given.

See the Fivetran REST API Group Management Docs for more information.

Connecting to Fivetran

To connect to Fivetran API services, you need to generate API keys. You can use these API keys in the following two ways:

API keys saved in DataOps vault at a specific path

  1. Save the API key and API secret in the DataOps Vault at the paths FIVETRAN.DEFAULT.API_KEY and FIVETRAN.DEFAULT.API_SECRET respectively.

    FIVETRAN:
    DEFAULT:
    API_KEY: "XXXXXXXXXX"
    API_SECRET: "XXXXXXXXXXXXXXXXXXXX"
  2. Set the pipeline/job variable SET_FIVETRAN_KEYS_TO_ENV to retrieve the values from the vault and use them in the Fivetran ingestion job.

API keys saved in DataOps vault at any path

  1. Save the API key and API secret in the DataOps Vault at any path.

    INGESTION:
    FIVETRAN:
    PROD:
    API_KEY: "XXXXXXXXXX"
    API_SECRET: "XXXXXXXXXXXXXXXXXXXX"
  2. Set values in the variables FIVETRAN_API_KEY and FIVETRAN_API_SECRET following the DATAOPS_VAULT functionality.

    "fivetran-orchestrator Ingestion Start":
    variables:
    FIVETRAN_ACTION: START
    CONNECTOR_ID: splint_shopkeepers
    FIVETRAN_API_KEY: DATAOPS_VAULT(INGESTION.FIVETRAN.PROD.API_KEY)
    FIVETRAN_API_SECRET: DATAOPS_VAULT(INGESTION.FIVETRAN.PROD.API_KEY)

Supported parameters

ParameterRequired or Optional (and default)Description
FIVETRAN_ACTIONREQUIREDMust be one of STATUS, START, TEST or FORCE_RESYNC
CONNECTOR_IDREQUIREDIdentifier for the connector. See the Get Connector ID section for fetching the connector ID.
SET_FIVETRAN_KEYS_TO_ENVREQUIRED if default DataOps Vault paths are usedIf a value is set to this variable, values for FIVETRAN_API_KEY and FIVETRAN_API_SECRET are fetched from the vault with the vault keys FIVETRAN.DEFAULT.API_KEY and FIVETRAN.DEFAULT.SECRET respectively. See API keys saved at a specific path for more info.
FIVETRAN_API_KEYOptionalFivetran API access token (see the Fivetran docs).
Note: This parameter becomes required if you don't set SET_FIVETRAN_KEYS_TO_ENV.
FIVETRAN_API_SECRETOptionalFivetran API access token (see the Fivetran docs).
Note: This parameter becomes required if you don't set SET_FIVETRAN_KEYS_TO_ENV.
TABLEREQUIRED in FORCE_RESYNC actionName of the table to force resync
SCHEMAREQUIRED in FORCE_RESYNC actionName of the schema to force resync
FIVETRAN_DONT_DISABLE_SYNCOptionalIf a value is set to this variable, schedule_type will not update to manual
FIVETRAN_TIMEOUTOptional, defaults to 300 secondsTime in seconds to wait for the completion of the Fivetran job, after which the operation will fail. We recommend you put a higher value if you have a large data volume to avoid a false positive failure where the system only needs some more time to complete the operation.

Example jobs

The following four examples describe the most common uses of the Fivetran orchestrator:

1. Check the connector status

This example sends a request to the Fivetran pipeline to check its connector status and prints the JSON response on the console.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_status.yml
"Fivetran Connector Status":
extends:
- .agent_tag
stage: "Check Status"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: STATUS
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}

2. Start a connector

This example sends a request to Fivetran to start the specified connector.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_start.yml
"Fivetran Connector Start":
extends:
- .agent_tag
stage: "Start Connector"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: START
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}

3. Test the connector

This example sends a request to Fivetran to test the specified connector.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_test.yml
"Fivetran Connector Test":
extends:
- .agent_tag
stage: "Test Connector"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: TEST
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}

4. Force resync of a specific table

This example requests Fivetran to force-resync the table specified in the YAML config file.

pipelines/includes/local_includes/fivetran-jobs/fivetran_force-resync.yml
"Fivetran Connector Force Resync":
extends:
- .agent_tag
stage: "Force Resync Table"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: FORCE_RESYNC
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
TABLE: <table name>
SCHEMA: <schema name>
script:
- /dataops
icon: ${FIVETRAN_ICON}