Configuring an ADF Linked Service Connection for Azure Databricks Delta Lake
For information on how to enable a connection for use with linked services, see Configuring a Linked Service Connection.
Field labels in this walkthrough follow the underlying Azure Data Factory property names, with the connection column that carries each value named alongside. The exact labels shown in the BimlFlex Connection Editor may differ.
After selecting Azure Databricks Delta Lake from the Linked Service Type dropdown, the form required for creating an Azure Databricks Delta Lake Linked Service will appear.
An Azure Databricks Delta Lake connection is used as a source. It is also available as a target for the Persistent Staging Area, the Data Vault and the Business Data Vault.
Datasets and Sources
The Azure Databricks Delta Lake dataset takes the table from the TargetObjectName parameter.
The copy source is an Azure Databricks Delta Lake source. It reads each object with a source query and takes the TargetSchemaName and TargetObjectName parameters.
Choosing an Authentication Method
| Method | Credentials | Additional Settings |
|---|---|---|
| Access Token | Access Token | clusterId |
Complete Example: Access Token
| Field | Example Value |
|---|---|
| Connect via Integration Runtime | AutoResolveIntegrationRuntime |
| Domain | https://adb-1234567890123456.7.azuredatabricks.net |
| Cluster Id | 0123-456789-abcdefgh |
| Access Token | The token generated from the Azure Databricks workspace |
Required Fields
Common Required Fields
An Azure Databricks Delta Lake Linked Service connection uses a single Authentication Type, so the same fields are required on every connection.
These common required fields are:
Connect via Integration Runtime
Connect via Integration Runtime names the integration runtime for an Azure Databricks Delta Lake Linked Service connection.
The default value is AutoResolveIntegrationRuntime.
To use another runtime, select it from the dropdown or choose + New to add one.
When a custom value is saved in the linked service form, it will be added to the custom integration runtimes.
The custom values that appear in this dropdown can be maintained in Settings under Azure - AzureIntegrationRuntime.
The runtime is stored in the connection's LS_ConnectVia_IntegrationRuntime column. Any value other than AutoResolveIntegrationRuntime is written to the linked service as the runtime it connects through.
Domain
The Domain is the Azure Databricks workspace URL (for example, https://adb-1234567890123456.7.azuredatabricks.net).
It is stored in the connection's LS_Domain column and maps to the ADF domain property.
Cluster Id
The Cluster Id identifies an existing interactive cluster in the Azure Databricks workspace (for example, 0123-456789-abcdefgh).
The Cluster Id is stored in the connection's Additional Settings (LS_AdditionalSettings) under the key clusterId, and maps to the ADF clusterId property.
Access Token
The Access Token authenticates Azure Data Factory to Azure Databricks. It is generated from the Azure Databricks workspace.
The Access Token field takes the token as plaintext. It is stored in the connection's LS_AccessToken column and maps to the ADF accessToken property.
Credential
When the connection's LS_Credential column names an Azure Data Factory credential for a user-assigned managed identity, it is written to the linked service as the credential reference.
Azure Data Factory Linked Service Additional Information
For additional information on ADF Azure Databricks Delta Lake Linked Service and its connection requirements see the Azure Data Factory Azure Databricks Delta Lake Connector documentation.