Configuring an ADF Linked Service Connection for Amazon S3
For information on how to enable a connection for use with linked services, see Configuring a Linked Service Connection.
Field labels in this walkthrough follow the underlying Azure Data Factory property names, with the connection column that carries each value named alongside. The exact labels shown in the BimlFlex Connection Editor may differ.
After selecting Amazon S3 from the Linked Service Type dropdown, the form required for creating an Amazon S3 Linked Service will appear.
File Format Connections
Amazon S3 is selected as the Linked Service Type of a connection whose System Type is one of the file formats Delimited Format, Avro Format, Binary Format, JSON Format, ORC Format, Parquet Format, XML Format or Delta Lake.
The Amazon S3 option appears in the Linked Service Type dropdown when the connection's Connection Type is Amazon S3.
Choosing an Authentication Method
| Method | Credentials | Additional Settings |
|---|---|---|
| Access Key | Access Key ID and Secret Access Key | None |
| Temporary Security Credentials | Access Key ID, Secret Access Key and Session Token | None |
Recommendation: Store the Secret Access Key and the Session Token in Azure Key Vault rather than entering them as plaintext.
Complete Example: Access Key with Azure Key Vault
| Field | Example Value |
|---|---|
| Connect via Integration Runtime | AutoResolveIntegrationRuntime |
| Authentication Type | AccessKey |
| Access Key ID | <access-key-id-of-iam-user> |
| Azure Key Vault Linked Service | bfx-kv |
| Secret Name (for Secret Access Key) | BFX-LND-AmazonS3SecretAccessKey |
Required Fields
Common Required Fields
The required fields for an Amazon S3 Linked Service connection are the same for every Authentication Type. The Session Token is used only with Temporary Security Credentials.
These common required fields are:
Optional fields are:
Authentication Type Dependent Fields
Fields for Temporary Security Credentials:
- Session Token (or Azure Key Vault)
Connect via Integration Runtime
Connect via Integration Runtime names the integration runtime for an Amazon S3 Linked Service connection.
The default value is AutoResolveIntegrationRuntime.
To use another runtime, select it from the dropdown or choose + New to add one.
When a custom value is saved in the linked service form, it will be added to the custom integration runtimes.
The custom values that appear in this dropdown can be maintained in Settings under Azure - AzureIntegrationRuntime.
The runtime is stored in the connection's LS_ConnectVia_IntegrationRuntime column. Any value other than AutoResolveIntegrationRuntime is written to the linked service as the runtime it connects through.
Authentication Type
The Amazon S3 Linked Service connection can use Access Key or Temporary Security Credentials authentication.
The selected value is stored in the connection's LS_AuthenticationType column. A new Amazon S3 connection starts with AccessKey.
When the value is TemporarySecurityCredentials, it is written to the ADF authenticationType property together with the Session Token.
When the value is AccessKey, the linked service carries no authenticationType property, and Azure Data Factory uses access key authentication, its default.
Access Key
Access Key authentication signs in with the access key of an AWS Identity and Access Management (IAM) account. It uses the Access Key ID and the Secret Access Key.
Temporary Security Credentials
Temporary Security Credentials authentication signs in with AWS temporary security credentials: the Access Key ID, the Secret Access Key and a Session Token.
The Session Token field appears when the Authentication Type is TemporarySecurityCredentials.
The Session Token is entered as plaintext in the LS_SessionToken column, or as an Azure Key Vault reference held in the LS_SessionTokenKVS_ fields that resolves to the token at runtime. It maps to the ADF sessionToken property.
AWS temporary credentials expire between 15 minutes and 36 hours after they are issued, depending on their settings. A session token stored in Azure Key Vault can be refreshed there so that it is valid when the pipeline runs.
It is suggested that Azure Key Vault be used in place of manually entering the Session Token.
Access Key ID
The Access Key ID identifies the access key the connector signs in with.
It is stored in the connection's LS_AccessKeyId column and maps to the ADF accessKeyId property.
Secret Access Key
The Secret Access Key field accepts the secret access key as plaintext in the LS_SecretAccessKey column, or an Azure Key Vault reference held in the LS_SecretAccessKeyKVS_ fields that resolves to the key at runtime.
It maps to the ADF secretAccessKey property.
It is suggested that Azure Key Vault be used in place of manually entering the Secret Access Key.
Service URL
The Service URL sets a custom S3 endpoint in the form https://<service url>.
It is stored in the connection's LS_ServiceUrl column and maps to the ADF serviceUrl property.
The Service URL is optional. When the column is empty, the linked service carries no serviceUrl property.
Set it only to use a different service endpoint or to switch between https and http.
Azure Data Factory Linked Service Additional Information
For additional information on ADF Amazon S3 Linked Service and its connection requirements see the Azure Data Factory Amazon S3 Connector documentation.