Azure Resource Manager templates for BimlFlex

BimlFlex uses Azure Resource Manager (ARM) templates to reliably automate deployments to the Azure cloud.

ARM templates are the Microsoft preferred way of deploying infrastructure-as-code to Azure. ARM templates define objects, data types, names, and properties within a JSON file which is then understood by the ARM API.

The template uses declarative syntax to indicate what is to be deployed without having to first write an entire sequence of programming commands. Within the ARM template, resources are deployed and properties for resources are also specified.

ARM templates generated by BimlFlex will deploy all elements of a Data Factory, including pipelines, linked services, datasets, and triggers.

BimlFlex Handling of ARM templates

A significant ARM template limitation in Azure is that the overall size of the template can not be larger than 4MB.

In cases where the generated template exceeds the 4MB size threshold, BimlFlex will also create a linked template folder (to the root folder of the project) that contains files for individual data sets and manages the dependency paths for these files.

File 0 has no dependency.
File 1 depends on File 0.
File 2 depends on File 0 and File 1.
File 3 ...

Microsoft has provided a detailed account of ARM Template best practices here. This approach follows the Microsoft best practice standards for managing ARM template limitations.

BimlFlex ensures that when a file has a dependency, the preceding file will be deployed before or at the same time as the file that relies on it.

BimlFlex also provides a PowerShell script for deploying the linked templates. Users will be required to provide metadata, or additional variables within the PowerShell script, such as: storage account name, storage account container name, and storage account key, among others.

ARM templates will need to be uploaded separately into Blob storage, and then referenced individually by the URL in the main template when deployed. This is included in the generated deployment script for linked template ARM files.

Note

In addition to the Best Practices document above, Microsoft has published documentation on the creation, editing, and deployment of ARM templates, referenced here.

Excluding parts of ARM template from deployment

When deploying the ARM template, it is possible to exclude parts of the template through the defined parameters. The arm_template_parameters.json file contains an attribute for excluding entities from the deployment:

"excludeFromDeployment": {
  "value": []
},

Any resource defined in the ARM template that is also defined in the excludeFromDeployment list will be excluded from the deployment.