Configuring an ADF Linked Service Connection for HDFS
For information on how to enable a connection for use with linked services, see Configuring a Linked Service Connection.
Field labels in this walkthrough follow the underlying Azure Data Factory property names, with the connection column that carries each value named alongside. The exact labels shown in the BimlFlex Connection Editor may differ.
After selecting HDFS from the Linked Service Type dropdown, the form required for creating an HDFS Linked Service will appear.
The HDFS option is offered when the connection's Connection Type is Azure File Server.
File Format System Types
An HDFS Linked Service is selected on a connection whose System Type is a file format. The file format System Types that offer it are Delimited Format, Avro Format, Binary Format, JSON Format, ORC Format, Parquet Format and XML Format.
Choosing an Authentication Method
| Method | Credentials | Additional Settings |
|---|---|---|
| Anonymous | None | None |
| Windows | Username and Password | None |
Recommendation: Store the Password in Azure Key Vault rather than entering it as plaintext.
Complete Example: Windows Authentication with Azure Key Vault
| Field | Example Value |
|---|---|
| Connect via Integration Runtime | bfx-selfhosted-ir |
| Url | http://hdfs.contoso.com:50070/webhdfs/v1/ |
| Authentication Type | Windows |
| Username | [email protected] |
| Azure Key Vault Linked Service | bfx-kv |
| Secret Name (for Password) | BFX-SRC-HdfsPassword |
Required Fields
Common Required Fields
Most required fields for an HDFS Linked Service connection depend on the Authentication Type used. There are some fields that are required regardless of Authentication Type.
These common required fields are:
Authentication Type Dependent Required Fields
The remaining field requirements are dependent on Authentication Type.
Required fields for Anonymous:
- None
Required fields for Windows:
- Username
- Password (or Azure Key Vault)
Connect via Integration Runtime
Connect via Integration Runtime names the integration runtime for an HDFS Linked Service connection.
The default value is AutoResolveIntegrationRuntime.
To use another runtime, select it from the dropdown or choose + New to add one.
When a custom value is saved in the linked service form, it will be added to the custom integration runtimes.
The custom values that appear in this dropdown can be maintained in Settings under Azure - AzureIntegrationRuntime.
The runtime is stored in the connection's LS_ConnectVia_IntegrationRuntime column. Any value other than AutoResolveIntegrationRuntime is written to the linked service as the runtime it connects through.
Url
The Url is the address of the HDFS service (for example, http://hdfs.contoso.com:50070/webhdfs/v1/).
It is stored in the connection's LS_Url column and maps to the ADF url property.
Authentication Type
The HDFS Linked Service connection can use Anonymous or Windows authentication.
The selected value is stored in the connection's LS_AuthenticationType column and maps to the ADF authenticationType property.
When a connection has no Authentication Type, the BimlFlex Connection Editor sets it to Anonymous.
For Anonymous, no credentials are required.
When using Windows, a Username and Password are required.
When the connection's LS_EncryptedCredential column holds an Encrypted Credential value, it is also written to the linked service.
The BimlFlex Connection Editor shows no field for this setting.
Anonymous
Anonymous authentication connects to HDFS without a Username or Password.
Windows
Windows authentication signs in to HDFS with a Username and Password.
For Kerberos authentication, the Username takes the form [email protected] (for example, [email protected]).
The Username is stored in the connection's LS_Username column and maps to the ADF userName property.
The Password field accepts the password as plaintext in the LS_Password column, or an Azure Key Vault reference held in the LS_PasswordKVS_ fields that resolves to the password at runtime. It maps to the ADF password property.
It is suggested that Azure Key Vault be used in place of manually entering the Password.
Azure Data Factory Linked Service Additional Information
For additional information on ADF HDFS Linked Service and its connection requirements see the Azure Data Factory HDFS Connector documentation.