Fivetran Orchestrator

Type	Pre-Set
Image	$DATAOPS_FIVETRAN_RUNNER_IMAGE

The Fivetran Orchestrator is a preset orchestrator, allowing a DataOps pipeline to interact with Fivetran as a job in a DataOps pipeline, triggering a Fivetran operation such as ingesting data from a Fivetran pipeline.

Usage

A typical Fivetran Orchestrator workflow is as follows:

• If FIVETRAN_ACTION has one of STATUS, START, TEST or FORCE_RESYNC, one of the following actions occurs:

  • Update the connector schedule_type to manual if FIVETRAN_DONT_DISABLE_SYNC has no value
  • If set to STATUS, print the JSON response in the console
  • If set START, log the connector's final status
  • If set to TEST, log the summary of tests passed or failed
  • If set FORCE_RESYNC, log the connector's final status

Connector ID

Fetch the Fivetran Connector ID for the Fivetran Orchestrator to work. To do so:

• Go to your connector instance in the Fivetran User Interface (UI)):

  

• Search for an event of type api_call:

  

• The string after the standard-config is the Connector ID (shrank_paparazzi in the example given).

See the Fivetran REST API Group Management Docs for more information.

Supported Parameters

Parameter	Required or Optional (and default)	Description
FIVETRAN_ACTION	REQUIRED	Must be one of STATUS, START, TEST or FORCE_RESYNC
CONNECTOR_ID	REQUIRED	Identifier for the connector. See the Get Connector ID section for fetching the connector ID
SET_FIVETRAN_KEYS_TO_ENV	Optional	If a value is set to this variable, fetches values for FIVETRAN_API_KEY and FIVETRAN_API_SECRET from vault with vault keys FIVETRAN.DEFAULT.API_KEY and FIVETRAN.DEFAULT.SECRET respectively
FIVETRAN_API_KEY	REQUIRED unless SET_FIVETRAN_KEYS_TO_ENV is set	Fivetran API access token (see the Fivetran docs)
FIVETRAN_API_SECRET	REQUIRED unless SET_FIVETRAN_KEYS_TO_ENV is set	Fivetran API access token (see the Fivetran docs)
TABLE	REQUIRED in FORCE_RESYNC action	Name of the table to force re-sync
SCHEMA	REQUIRED in FORCE_RESYNC action	Name of the schema to force re-sync
FIVETRAN_DONT_DISABLE_SYNC	Optional	If a value is set to this variable, schedule_type will not updated to manual

Example Jobs

The following four examples describe the most common uses of the Fivetran Orchestrator:

1. Check the Connector Status

This example sends a request to the Fivetran pipeline to check its connector status and prints the JSON response on the console.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_status.yml

"Fivetran Connector Status":
  extends: 
    - .agent_tag
  stage: "Check Status"
  image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
  variables:
    FIVETRAN_ACTION: STATUS
    FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
    FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
    CONNECTOR_ID: <your connector id>
  script:
    - /dataops
  icon: ${FIVETRAN_ICON}

2. Start a Connector

This example sends a request to Fivetran to start the specified connector.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_start.yml

"Fivetran Connector Start":
  extends: 
    - .agent_tag
  stage: "Start Connector"
  image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
  variables:
    FIVETRAN_ACTION: START
    FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
    FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
    CONNECTOR_ID: <your connector id>
  script:
    - /dataops
  icon: ${FIVETRAN_ICON}

3. Test the Connector

This example sends a request to Fivetran to test the specified connector.

pipelines/includes/local_includes/fivetran-jobs/fivetran_connector_test.yml

"Fivetran Connector Test":
  extends: 
    - .agent_tag
  stage: "Test Connector"
  image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
  variables:
    FIVETRAN_ACTION: TEST
    FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
    FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
    CONNECTOR_ID: <your connector id>
  script:
    - /dataops
  icon: ${FIVETRAN_ICON}

4. Force Resync of a Specific Table

This example sends a request to Fivetran to force resync the table specified in the YAML config file.

pipelines/includes/local_includes/fivetran-jobs/fivetran_force-resync.yml

"Fivetran Connector Force Resync":
  extends: 
    - .agent_tag
  stage: "Force Resync Table"
  image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
  variables:
    FIVETRAN_ACTION: FORCE_RESYNC
    FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
    FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
    CONNECTOR_ID: <your connector id>
    TABLE: <table name>
    SCHEMA: <schema name>
  script:
    - /dataops
  icon: ${FIVETRAN_ICON}

Project Resources

None

Host Dependencies (and Resources)

None