Configuring an ADF Linked Service Connection for Spark

note

For information on how to enable a connection for use with linked services, see Configuring a Linked Service Connection.

note

Field labels in this walkthrough follow the underlying Azure Data Factory property names, with the connection column that carries each value named alongside. The exact labels shown in the BimlFlex Connection Editor may differ.

After selecting Spark from the Linked Service Type dropdown, the form required for creating a Spark Linked Service will appear.

BimlFlex writes the Spark linked service at version 2.0.

Source Only Connector

A Spark connection is used as a source. It is not available as a target.

Each Spark object is read with a source query on the copy source.

Choosing an Authentication Method

Method	Credentials	Additional Settings
Anonymous	None	None
UsernameAndPassword	Username and Password	None
WindowsAzureHDInsightService	Username and Password	None

Recommendation: Store the Password in Azure Key Vault rather than entering it as plaintext.

Complete Example: WindowsAzureHDInsightService with Azure Key Vault

Field	Example Value
Connect via Integration Runtime	AutoResolveIntegrationRuntime
Host	contoso-spark.azurehdinsight.net
Port	443
Server Type	SparkThriftServer
Thrift Transport Protocol	HTTP
Enable SSL	Checked
Authentication Type	WindowsAzureHDInsightService
Username	bfx_reader
Azure Key Vault Linked Service	bfx-kv
Secret Name (for Password)	BFX-SRC-SparkPassword

Required Fields

Common Required Fields

Most required fields for a Spark Linked Service connection depend on the Authentication Type used. There are some fields that are required regardless of Authentication Type.

These common required fields are:

• Connect via Integration Runtime
• Host
• Port
• Authentication Type

Authentication Type Dependent Required Fields

The remaining field requirements are dependent on Authentication Type.

Required fields for UsernameAndPassword and WindowsAzureHDInsightService:

• Username
• Password (or Azure Key Vault)

Optional fields are:

• Server Type
• Thrift Transport Protocol
• HTTP Path
• Enable SSL
• Enable Server Certificate Validation

Connect via Integration Runtime

Connect via Integration Runtime names the integration runtime for a Spark Linked Service connection. The default value is AutoResolveIntegrationRuntime. To use another runtime, select it from the dropdown or choose + New to add one. When a custom value is saved in the linked service form, it will be added to the custom integration runtimes. The custom values that appear in this dropdown can be maintained in Settings under Azure - AzureIntegrationRuntime.

The runtime is stored in the connection's LS_ConnectVia_IntegrationRuntime column. Any value other than AutoResolveIntegrationRuntime is written to the linked service as the runtime it connects through.

Host

The Host is the IP address or host name of the Spark server (for example, contoso-spark.azurehdinsight.net). It is stored in the connection's LS_Host column and maps to the ADF host property.

Port

The Port is the TCP port that the Spark server uses to listen for client connections. To connect to Azure HDInsight, set it to 443. It is stored in the connection's LS_Port column and maps to the ADF port property.

Server Type

The Server Type is the type of Spark server. The dropdown offers SharkServer, SharkServer2 and SparkThriftServer, and Azure Data Factory accepts SparkThriftServer for this linked service. It is stored in the connection's LS_ServerType column and maps to the ADF serverType property. BimlFlex writes it to the linked service when the column holds SharkServer2 or SparkThriftServer. When the column holds SharkServer or is empty, the property is not written.

Thrift Transport Protocol

The Thrift Transport Protocol is the transport protocol used in the Thrift layer. The dropdown offers Binary, SASL and HTTP, and Azure Data Factory accepts HTTP for this linked service. It is stored in the connection's LS_ThriftTransportProtocol column and maps to the ADF thriftTransportProtocol property. BimlFlex writes it to the linked service when the column holds SASL or HTTP. When the column holds Binary or is empty, the property is not written.

HTTP Path

The HTTP Path is the partial URL that corresponds to the Spark server. For the WindowsAzureHDInsightService Authentication Type, the default value is /sparkhive2. It is stored in the connection's LS_HttpPath column and maps to the ADF httpPath property. BimlFlex writes it to the linked service when the column is set.

Enable SSL

Enable SSL specifies whether the connections to the server are encrypted using TLS. It is stored in the connection's LS_EnableSsl column and maps to the ADF enableSsl property. When Enable SSL is checked, BimlFlex writes enableSsl as true. The build reads the column as true when it holds true, t, yes, y or 1, in any letter case. A cleared box stores false in the column. When the column does not read as true, the build reads the key enableSsl in the connection's Additional Settings (LS_AdditionalSettings), and a value of true, 1 or Y there also writes enableSsl as true. Otherwise the property is not written and Azure Data Factory uses its default value, true.

Enable Server Certificate Validation

Enable Server Certificate Validation specifies whether the server SSL certificate is validated when the connection opens. Validation uses the system trust store. The Spark form has no field for it. The build reads the connection's LS_EnableServerCertificateValidation column and maps it to the ADF enableServerCertificateValidation property. The build reads a filled column as true when it holds true, t, yes, y or 1, in any letter case, and as false for any other value. A column read as false writes enableServerCertificateValidation as false. A column read as true, or a blank column, writes no property, and Azure Data Factory uses its default value, true.

Use System Trust Store

The Spark form shows a Use System Trust Store box, stored in the connection's LS_UseSystemTrustStore column. The value entered in that box is not written to the linked service. The build writes no useSystemTrustStore, trustedCertPath, allowHostNameCNMismatch or allowSelfSignedServerCert property, and a value held in the LS_TrustedCertPath, LS_AllowHostNameCNMismatch or LS_AllowSelfSignedServerCert column is not written either. Server certificate validation uses the system trust store, as the Enable Server Certificate Validation section describes.

Authentication Type

The Spark Linked Service connection can use Anonymous, UsernameAndPassword or WindowsAzureHDInsightService authentication. The selected value is stored in the connection's LS_AuthenticationType column and maps to the ADF authenticationType property.

When using UsernameAndPassword or WindowsAzureHDInsightService, a Username and Password are required.

Anonymous

Anonymous authentication takes no Username or Password.

UsernameAndPassword

UsernameAndPassword authentication signs in to the Spark server with a Username and Password.

The Username is stored in the connection's LS_Username column and maps to the ADF username property. The Password field accepts the password as plaintext in the LS_Password column, or an Azure Key Vault reference held in the LS_PasswordKVS_ fields that resolves to the password at runtime. It maps to the ADF password property.

WindowsAzureHDInsightService

WindowsAzureHDInsightService authentication signs in with a Username and Password. The Username and Password are stored in the same columns as for UsernameAndPassword.

note

It is suggested that Azure Key Vault be used in place of manually entering the Password.

Azure Data Factory Linked Service Additional Information

For additional information on ADF Spark Linked Service and its connection requirements see the Azure Data Factory Spark Connector documentation.