Fivetran Orchestrator
Type | Pre-Set |
---|---|
Image | $DATAOPS_FIVETRAN_RUNNER_IMAGE |
The Fivetran Orchestrator is a preset orchestrator, allowing a DataOps pipeline to interact with Fivetran as a job in a DataOps pipeline, triggering a Fivetran operation such as ingesting data from a Fivetran pipeline.
Usage
A typical Fivetran Orchestrator workflow is as follows:
If
FIVETRAN_ACTION
has one ofSTATUS
,START
,TEST
orFORCE_RESYNC
, one of the following actions occurs:- Update the connector
schedule_type
to manual ifFIVETRAN_DONT_DISABLE_SYNC
has no value - If set to
STATUS
, print the JSON response in the console - If set
START
, log the connector's final status - If set to
TEST
, log the summary of tests passed or failed - If set
FORCE_RESYNC
, log the connector's final status
- Update the connector
Connector ID
Fetch the Fivetran Connector ID for the Fivetran Orchestrator to work. To do so:
Go to your connector instance in the Fivetran User Interface (UI)):
Search for an event of type
api_call
:The string after the standard-config is the Connector ID (shrank_paparazzi in the example given).
See the Fivetran REST API Group Management Docs for more information.
Supported Parameters
Parameter | Required or Optional (and default) | Description |
---|---|---|
FIVETRAN_ACTION | REQUIRED | Must be one of STATUS, START, TEST or FORCE_RESYNC |
CONNECTOR_ID | REQUIRED | Identifier for the connector. See the Get Connector ID section for fetching the connector ID |
SET_FIVETRAN_KEYS_TO_ENV | Optional | If a value is set to this variable, fetches values for FIVETRAN_API_KEY and FIVETRAN_API_SECRET from vault with vault keys FIVETRAN.DEFAULT.API_KEY and FIVETRAN.DEFAULT.SECRET respectively |
FIVETRAN_API_KEY | REQUIRED unless SET_FIVETRAN_KEYS_TO_ENV is set | Fivetran API access token (see the Fivetran docs) |
FIVETRAN_API_SECRET | REQUIRED unless SET_FIVETRAN_KEYS_TO_ENV is set | Fivetran API access token (see the Fivetran docs) |
TABLE | REQUIRED in FORCE_RESYNC action | Name of the table to force re-sync |
SCHEMA | REQUIRED in FORCE_RESYNC action | Name of the schema to force re-sync |
FIVETRAN_DONT_DISABLE_SYNC | Optional | If a value is set to this variable, schedule_type will not updated to manual |
Example Jobs
The following four examples describe the most common uses of the Fivetran Orchestrator:
1. Check the Connector Status
This example sends a request to the Fivetran pipeline to check its connector status and prints the JSON response on the console.
"Fivetran Connector Status":
extends:
- .agent_tag
stage: "Check Status"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: STATUS
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}
2. Start a Connector
This example sends a request to Fivetran to start the specified connector.
"Fivetran Connector Start":
extends:
- .agent_tag
stage: "Start Connector"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: START
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}
3. Test the Connector
This example sends a request to Fivetran to test the specified connector.
"Fivetran Connector Test":
extends:
- .agent_tag
stage: "Test Connector"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: TEST
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
script:
- /dataops
icon: ${FIVETRAN_ICON}
4. Force Resync of a Specific Table
This example sends a request to Fivetran to force resync the table specified in the YAML config file.
"Fivetran Connector Force Resync":
extends:
- .agent_tag
stage: "Force Resync Table"
image: $DATAOPS_FIVETRAN_RUNNER_IMAGE
variables:
FIVETRAN_ACTION: FORCE_RESYNC
FIVETRAN_API_KEY: <your key here or in the DataOps Vault>
FIVETRAN_API_SECRET: <your secret here or in the DataOps Vault>
CONNECTOR_ID: <your connector id>
TABLE: <table name>
SCHEMA: <schema name>
script:
- /dataops
icon: ${FIVETRAN_ICON}
Project Resources
None
Host Dependencies (and Resources)
None