Skip to main content

dbt Integration

Overview

Omni integrates closely with dbt (both dbt Core and Cloud). These are the features the dbt integration supports (and more coming soon!):

  1. Schema refreshes to sync dbt changes
  2. Pull dbt context and metadata into Omni
  3. Author dbt models from Omni queries
  4. Dynamic schema switching

Steps for Connecting dbt to Omni

  1. Inside your Omni connection, navigate to the dbt tab:
    • Git SSH URL can be found from your git provider (make sure to use the SSH one)
    • Git branch is typically main or master
    • Target schema is the schema you used to configure dbt
      • If you're using dbt Cloud, you can find this under Deployment Environments > Production Prod > Settings > Deployment Credentials > Schema
      • If you're using dbt Core, you can often find this from the profiles.yml file under 'Schema' Note, for BigQuery, 'schema' is called 'dataset' instead.
    • Hit ‘Submit’ and Omni will provide you a public key.
  1. Follow the instructions from dbt to add a deploy key to your git provider. If you want to use Omni’s ‘push to dbt’ integration, make sure you allow write access. You’ve now connected Omni to your dbt repo.

  2. Back on the Omni connection page, hit ‘Refresh Schema.’ This will sync the metadata from the dbt schema.yml file to your Omni model. The schema.yml file is what dbt uses to generate their documentation. Example dbt schema.yml file:

  1. dbt model descriptions will be added to the matching Omni view files. These are also exposed in Omni’s field picker
  1. If metadata changes in dbt, you’ll need to refresh the schema again in Omni (either from the connection page, or the model) to resync. We will be automating this in the future.

Sync dbt changes with schema refresh

Schema refresh syncs downstream changes from dbt into your Omni model, pulling in field changes, new fields, new tables, etc. This eliminates the need to manually update the Omni model every time something changes in the layer below. To kick off a refresh, select 'Refresh Schema' from the IDE model menu, or use our API endpoint. Learn more about schema refreshes here.

Pull dbt context and metadata into Omni

Pulling metadata from dbt into Omni makes it easier for analytics engineers, analysts, and business users to share context about what fields and tables mean, ensuring they're using the right things in their queries (and making tracing logic throughout the data stack very easy!). Omni pulls the following information from dbt:

  • dbt model and field descriptions are brought into the corresponding Omni view files. Descriptions also become viewable in the workbook field picker
  • dbt SQL code is brought into the corresponding Omni view files. This makes it easy to trace logic between the tools
  • dbt dependencies are also brought into the corresponding Omni view files

Author dbt models from Omni queries

Omni is great for both building analyses and data models. Every time you construct a query - select columns, aggregate, filter, add joins - you're creating the building blocks of a model. While not every query needs to be pushed down to dbt, when you do create a query where you want to promote the logic into dbt, you can easily do so from within Omni.

  • Within a workbook tab, there is an option under the Model menu to ‘Push to dbt’. This will open a new pull request in your dbt git repository with the SQL used to generate the query.
  • Once merged in dbt, the new model will show up inside Omni upon refreshing schema from the Omni IDE.

Dynamic schemas

Dynamic schemas allow users to switch the environment that their dbt integration is pointing to within Omni. Users can easily switch between development and production environments to test changes made in the model and how they impact content. Users can...

  • Run the content validator while pointing at a dbt development schema to determine the possible impact ahead of merging changes.
  • Create new queries or create visualizations against the dbt dev schema to validate the model changes work as expected.

To switch schemas, users must be in branch mode. To switch schemas, modify the Target Schema in the branch.