Skip to main content
Omni can connect to your Databricks workspace in Amazon Web Services (AWS) or Azure via PrivateLink. In this configuration, Omni establishes an encrypted connection from our network to your Databricks workspace over a private network.

​
Requirements

To follow the steps in this guide, you’ll need:
  • A Databricks workspace configured for PrivateLink access. Refer to the setup section for your cloud provider for the specific requirements.
  • Permissions in your cloud environment to retrieve PrivateLink configuration details
  • On Azure, permissions to approve private endpoint connections
  • Organization Admin permissions in Omni

​
Setup

Follow the steps for the cloud provider that hosts your Databricks workspace.

​
AWS setup

Your workspace must have AWS PrivateLink enabled.
1
Collect the following details:
  • Your Databricks workspace URL. For example, dbc-abcd1234-5678.cloud.databricks.com
  • The AWS region where your workspace is deployed
  • The VPC endpoint service name, which starts with com.amazonaws.vpce
2
Contact Omni support with:
  • The details from step 1
  • Technical contact details, in case of connection difficulties
3
Omni will create a VPC endpoint that connects to your workspace.

​
Azure setup

Your workspace must be VNet-injected, which means that it is deployed into a custom VNet. A workspace without this cannot accept a private endpoint.

​
Workspace storage and large result sets

Databricks sends large result sets through CloudFetch. The workspace writes the rows to its managed storage account, then gives Omni’s driver temporary URLs to download them directly from that account. This traffic does not use the workspace endpoint. If your workspace storage account blocks public network access, Omni must create a second private endpoint for it. Without this endpoint, small queries succeed and large queries fail. Refer to Troubleshooting. To find the storage account, open your workspace’s managed resource group in the Azure portal. Then check Security + networking > Networking > Firewalls and virtual networks on the account. Under Public network access:
  • Enabled from all networks - No extra endpoint is necessary.
  • Enabled from selected virtual networks and IP addresses or Disabled - Send the storage account resource ID to Omni support in step 1.
This applies only to Azure. On AWS, Databricks downloads result sets from Amazon S3 through a gateway endpoint, which Omni’s network already permits.

​
Steps

1
Collect the following details:
  • Your Databricks workspace URL. For example, adb-1234567890123456.7.azuredatabricks.net
  • The Azure region where your workspace is deployed
  • The Databricks workspace resource ID, which contains /providers/Microsoft.Databricks/workspaces/
  • The workspace storage account resource ID, which contains /providers/Microsoft.Storage/storageAccounts/. Include this only if the storage account blocks public network access.
2
Contact Omni support with:
  • The details from step 1
  • Technical contact details, in case of connection difficulties
3
Omni will create a private endpoint that connects to your workspace. If you sent a storage account resource ID, Omni will also create a private endpoint for that account.
4
Approve each private endpoint connection. Omni creates them as manual connection requests and will tell you when they are ready.
  • Approve the workspace endpoint in your Databricks workspace, under Networking > Private endpoint connections.
  • Approve the storage account endpoint on the storage account, under Security + networking > Networking > Private endpoints.
Each endpoint is approved on the resource that it connects to. Approval of the workspace endpoint does not approve the storage account endpoint.

​
Troubleshooting

​
Large queries fail, but small queries succeed

On Azure, this usually means that CloudFetch cannot reach the workspace storage account. Queries that return a small number of rows come back through the workspace endpoint and are not affected. To correct this:
  1. Confirm that the workspace storage account blocks public network access. Refer to Workspace storage and large result sets.
  2. Send the storage account resource ID to Omni support.
  3. Approve the private endpoint connection on the storage account after Omni creates it.

​
What’s next?

After Omni creates the private endpoints and you approve them, Omni support will add the new database connection to your Omni instance. Omni will notify you when the connection is ready to use. The connection can then be used like any other database connection. You’ll need to provide your Databricks authentication credentials (Personal Access Token or OAuth M2M) when setting up the connection in Omni.