Data Factory Extension Points

The Data Factory category has the following available Extension Points defined.

Add Linked Service

Configure additional Linked Service references to the Data Factory

Parameters

Name	Type	Description
dataFactory	string	Contains the name of the related Data Factory.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfAddLinkedService" #>
<#@ property name="dataFactory" type="string" #>


<# 	CustomOutput.ObjectInherit = true; #>
<AzureBlobStorage Name="AdfLinkedService_SRC" AuthenticationMethod="AccountKey">
	<ConnectionStringKVS StoreName="AdfKeyVault" Name="AdfLinkedService-SRC-ConnectionString"/>
</AzureBlobStorage>

Deprecation Notice

This extension point is deprecated and no longer present in the BimlFlex manifest. Use AdfLinkedService to define new linked services or OverrideAdfLinkedService to override existing ones.

Archive Delete Landing Files

Override the archival process of Delete Detection Landing Files for Data Factory that involves retrieving, copying, and deleting source files.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object (e.g. table or file) to which the process will be added.
dependency	String	Contains the dependency name for the previous activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the batch.
OutputPathName	String	You must add CustomOutput.OutputPathName with the last task to connect it with the next Data Flow task.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfArchiveDeleteLandingFiles" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>


<# 	CustomOutput.ObjectInherit = false; #>

<#	var sourceTableObject = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC");
#>

What It Does

Overrides the default archival process for delete detection landing files in Azure Data Factory. Landing files in the delete detection context are the intermediate files created in the landing area during the delete detection extract process. This extension point controls what happens to those files after the delete detection staging process has consumed them -- whether they are archived to a separate folder, deleted, or retained.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the delete detection pipeline's file handling section, after the delete detection staging process has consumed the landing files. It replaces the default landing file archival activities for delete detection.

When to Use It

1. You need to archive delete detection landing files to a dated folder for audit trail purposes.
2. You want to retain delete detection landing files for reprocessing in case the delete detection logic needs to be re-run.
3. You need to copy delete detection landing files to a secondary storage for compliance before deletion.
4. You want to skip cleanup during development for easier debugging of delete detection logic. Use this instead of AdfArchiveLandingFiles when working specifically with delete detection landing files.

Prerequisites

• Delete detection must be enabled for the source object in BimlFlex metadata.
• The source must use file-based extraction that creates landing files for delete detection.
• The table and dependency parameters provide access to object metadata and the preceding activity.
• CustomOutput.OutputPathName must be set for dependency chaining.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the custom archival logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfArchiveDeleteLandingFiles" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = false; #>
<# var srcObj = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC"); #>
<# var outputPathName = "Archive_Del_Lnd_" + srcObj.SourceSsisSafeName; #>
<Copy Name="<#=outputPathName#>" EnableStaging="false">
  <BinarySource DatasetName="AZB_DelLanding" />
  <BinarySink DatasetName="AZB_Archive" />
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Copy>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the archival activity appears in the delete detection pipeline after the staging process.

Example

Before (default delete landing file handling):

{ "name": "Delete_DelLnd_SalesLT_Customer",
  "type": "Delete",
  "typeProperties": { "dataset": { "referenceName": "AZB_DelLanding" } }
}

After (archive to compliance folder):

{ "name": "Archive_Del_Lnd_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "source": { "type": "BinarySource" },
    "sink": { "type": "BinarySink" }
  }
}

Common Mistakes

1. Mistake: Confusing this with AdfArchiveLandingFiles. Symptom: Regular extraction landing files are archived instead of delete detection files. Fix: Use AdfArchiveDeleteLandingFiles specifically for delete detection landing files.
2. Mistake: Not setting CustomOutput.OutputPathName. Symptom: Downstream activities in the delete detection pipeline lose their dependency chain. Fix: Always set the output path name.
3. Mistake: Archiving landing files before the delete detection staging process has consumed them. Symptom: The staging process fails because landing files are no longer available. Fix: Ensure the dependency references the staging activity.

Related Extension Points

• AdfArchiveDeleteSourceFiles -- archives delete detection source files.
• AdfArchiveLandingFiles -- archives regular extraction landing files.
• AdfArchiveSourceFiles -- archives regular extraction source files.
• AdfProcessStaging -- the staging process that consumes landing files before archival.

Archive Delete Source Files

Override the archival process of Delete Detection Source Files for Data Factory that involves retrieving, copying, and deleting source files.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object (e.g. table or file) to which the process will be added.
dependency	String	Contains the dependency name for the previous activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the batch.
OutputPathName	String	You must add CustomOutput.OutputPathName with the last task to connect it with the next Data Flow task.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfArchiveDeleteSourceFiles" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>


<# 	CustomOutput.ObjectInherit = false; #>

<#	var sourceTableObject = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC");
#>

What It Does

Overrides the default archival process for delete detection source files in Azure Data Factory. Delete detection is a BimlFlex pattern that identifies rows deleted from the source system by comparing current source snapshots against previously loaded data. This extension point controls what happens to the source files used in the delete detection process after they have been copied to the landing area -- whether they are archived, deleted, or retained.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the delete detection pipeline's file handling section, after the delete detection source files have been successfully copied. It replaces the default source file archival activities for delete detection.

When to Use It

1. You need to archive delete detection source files to a dated folder for compliance auditing.
2. You want to retain delete detection source files for a specific period to enable reprocessing of delete detection logic.
3. You need to copy delete detection files to a secondary storage account before deletion.
4. You want to skip deletion of source files during development for debugging purposes. Use this instead of AdfArchiveSourceFiles when working specifically with delete detection source files rather than regular extraction source files.

Prerequisites

• Delete detection must be enabled for the source object in BimlFlex metadata.
• The source must be a file-based connection that produces delete detection files.
• The table and dependency parameters provide access to object metadata and the preceding activity.
• CustomOutput.OutputPathName must be set for dependency chaining.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the custom archival logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfArchiveDeleteSourceFiles" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = false; #>
<# var srcObj = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC"); #>
<# var outputPathName = "Archive_Del_Src_" + srcObj.SourceSsisSafeName; #>
<Copy Name="<#=outputPathName#>" EnableStaging="false">
  <BinarySource DatasetName="AZB_DelSource" />
  <BinarySink DatasetName="AZB_Archive" />
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Copy>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the archival activity appears in the delete detection pipeline.

Example

Before (default delete source file handling):

{ "name": "Delete_DelSrc_SalesLT_Customer",
  "type": "Delete",
  "typeProperties": { "dataset": { "referenceName": "AZB_DelSource" } }
}

After (archive to compliance folder):

{ "name": "Archive_Del_Src_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "source": { "type": "BinarySource" },
    "sink": { "type": "BinarySink" }
  }
}

Common Mistakes

1. Mistake: Confusing this with AdfArchiveSourceFiles. Symptom: Regular extraction source files are archived instead of delete detection files. Fix: Use AdfArchiveDeleteSourceFiles specifically for delete detection source files.
2. Mistake: Not setting CustomOutput.OutputPathName. Symptom: The delete detection pipeline's downstream activities lose their dependency chain. Fix: Always set the output path name.
3. Mistake: Archiving before confirming the copy succeeded. Symptom: Delete detection files are moved before landing, causing data loss. Fix: Use Condition="Succeeded" on the dependency.

Related Extension Points

• AdfArchiveDeleteLandingFiles -- archives delete detection landing files.
• AdfArchiveSourceFiles -- archives regular extraction source files.
• AdfArchiveLandingFiles -- archives regular extraction landing files.
• AdfPostCopy -- post-copy logic that may include archival steps.

Archive Landing Files

Override the archival process of Copy Activity Landing Files for Data Factory that involves retrieving, copying, and deleting source files.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object (e.g. table or file) to which the process will be added.
dependency	String	Contains the dependency name for the previous activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the batch.
OutputPathName	String	You must add CustomOutput.OutputPathName with the last task to connect it with the next Data Flow task.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfArchiveLandingFiles" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>


<# 	CustomOutput.ObjectInherit = false; #>

<#	var sourceTableObject = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC");
#>

What It Does

Overrides the default archival process for landing files after a Copy Activity completes in Azure Data Factory. Landing files are the intermediate files created in the landing area (e.g., Azure Blob Storage or ADLS Gen2) during the extract process. This extension point controls what happens to those landing files after the staging process has consumed them -- whether they are moved to an archive folder, deleted, or retained for reprocessing.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline's file handling section, after the staging process has consumed the landing files. It replaces the default landing file archival activities.

When to Use It

1. You need to move processed landing files to a dated archive folder for audit trail purposes (e.g., archive/landing/2025/03/26/SalesLT_Customer/).
2. You want to retain landing files for a specific retention period instead of deleting them immediately.
3. You need to copy landing files to a secondary storage account before deletion.
4. You want to skip landing file cleanup for debugging purposes during development. Use this instead of AdfArchiveSourceFiles when you need to control what happens to files in the landing area, not the original source files.

Prerequisites

• The source must be a file-based extraction process that creates landing files.
• The table and dependency parameters provide access to object metadata and the preceding activity.
• CustomOutput.OutputPathName must be set to connect the archival activity to the downstream dependency chain.
• Archive storage locations must be accessible from the ADF Integration Runtime.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the custom archival logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfArchiveLandingFiles" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = false; #>
<# var srcObj = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC"); #>
<# var outputPathName = "Archive_Landing_" + srcObj.SourceSsisSafeName; #>
<Copy Name="<#=outputPathName#>" EnableStaging="false">
  <BinarySource DatasetName="AZB_Landing" />
  <BinarySink DatasetName="AZB_Archive" />
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Copy>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the archival activity appears in the pipeline after the staging process.

Example

Before (default landing file deletion):

{ "name": "Delete_Landing_SalesLT_Customer",
  "type": "Delete",
  "typeProperties": { "dataset": { "referenceName": "AZB_Landing" } }
}

After (archive to dated folder):

{ "name": "Archive_Landing_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "source": { "type": "BinarySource" },
    "sink": { "type": "BinarySink" }
  }
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: Downstream activities lose their dependency chain. Fix: Always set the output path name.
2. Mistake: Confusing AdfArchiveLandingFiles with AdfArchiveSourceFiles. Symptom: Source files are archived instead of landing files. Fix: Use AdfArchiveLandingFiles for landing-side and AdfArchiveSourceFiles for source-side.
3. Mistake: Archiving landing files before the staging process has consumed them. Symptom: Staging fails because the landing files have been moved. Fix: Ensure the dependency references the staging activity, not the copy activity.
4. Mistake: Not cleaning up the archive over time. Symptom: Storage costs grow indefinitely. Fix: Implement a lifecycle management policy on the archive storage container.

Related Extension Points

• AdfArchiveSourceFiles -- archives the original source files.
• AdfArchiveDeleteLandingFiles -- archives delete detection landing files.
• AdfArchiveDeleteSourceFiles -- archives delete detection source files.
• AdfProcessStaging -- the staging process that consumes landing files before archival.

Archive Source Files

Override the archival process of Copy Activity Source Files for Data Factory that involves retrieving, copying, and deleting source files.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object (e.g. table or file) to which the process will be added.
dependency	String	Contains the dependency name for the previous activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the batch.
OutputPathName	String	You must add CustomOutput.OutputPathName with the last task to connect it with the next Data Flow task.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfArchiveSourceFiles" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>


<# 	CustomOutput.ObjectInherit = false; #>

<#	var sourceTableObject = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC");
#>

What It Does

Overrides the default archival process for source files processed by a Copy Activity in Azure Data Factory. This extension point controls what happens to the original source files after they have been successfully copied to the landing area. The default BimlFlex behavior typically moves or deletes the source files, but this extension point lets you implement custom archival logic such as moving files to a dated archive folder, renaming them, or applying retention policies.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline's file handling section, after the Copy Activity completes successfully. It replaces the default source file archival activities.

When to Use It

1. You need to move processed source files to a dated archive folder (e.g., archive/2025/03/26/) instead of deleting them.
2. You want to copy source files to a secondary storage account for compliance or audit requirements before deleting the originals.
3. You need to rename processed files with a timestamp suffix to prevent reprocessing.
4. You want to skip source file archival entirely for certain objects while keeping it for others. Use this instead of AdfArchiveLandingFiles when you need to control what happens to the original source files, not the landing-layer copies.

Prerequisites

• The source must be a file-based connection (e.g., Azure Blob Storage, ADLS Gen2, SFTP).
• The table and dependency parameters provide access to object metadata and the preceding activity.
• CustomOutput.OutputPathName must be set to connect the archival activity to the downstream dependency chain.
• Archive storage locations must be accessible from the ADF Integration Runtime.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the custom archival logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfArchiveSourceFiles" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = false; #>
<# var srcObj = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC"); #>
<# var outputPathName = "Archive_Source_" + srcObj.SourceSsisSafeName; #>
<Copy Name="<#=outputPathName#>" EnableStaging="false">
  <BinarySource DatasetName="AZB_Source" />
  <BinarySink DatasetName="AZB_Archive" />
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Copy>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the archival activity appears in the pipeline after the Copy Activity.

Example

Before (default source file handling):

{ "name": "Delete_Source_SalesLT_Customer",
  "type": "Delete",
  "typeProperties": { "dataset": { "referenceName": "AZB_Source" } }
}

After (copy to archive then delete):

{ "name": "Archive_Source_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "source": { "type": "BinarySource", "storeSettings": { "type": "AzureBlobStorageReadSettings" } },
    "sink": { "type": "BinarySink", "storeSettings": { "type": "AzureBlobStorageWriteSettings" } }
  }
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: Downstream activities lose their dependency chain. Fix: Always set the output path name.
2. Mistake: Archiving files before the copy is confirmed successful. Symptom: Source files are moved but the copy failed, causing data loss. Fix: Use Condition="Succeeded" on the dependency to ensure the copy completed.
3. Mistake: Confusing AdfArchiveSourceFiles with AdfArchiveLandingFiles. Symptom: Landing files are archived instead of source files. Fix: Use AdfArchiveSourceFiles for source-side archival and AdfArchiveLandingFiles for landing-side.

Related Extension Points

• AdfArchiveLandingFiles -- archives files in the landing area after copy.
• AdfArchiveDeleteSourceFiles -- archives delete detection source files.
• AdfArchiveDeleteLandingFiles -- archives delete detection landing files.
• AdfPostCopy -- adds post-copy logic that runs after the Copy Activity.

Batch Parameter

Configure Data Factory Batch Parameters.

Parameters

Name	Type	Description
batch	BimlFlexModelWrapper.BatchesWrapper	Contains all information about the Data Factory (Batch) Pipeline to which the Parameter will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchParameter" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>


<Parameter Name="MyAdfParameter" DataType="Bool">false</Parameter>

What It Does

Adds custom parameters to the batch-level ADF pipeline (the orchestration pipeline for a batch such as 01_EXT_AWLT_SRC_Batch). Batch pipeline parameters are read-only inputs that are set when the batch pipeline is triggered or called by a parent pipeline. They allow you to pass configuration values such as IsInitialLoad flags, environment identifiers, or processing mode settings into the batch orchestration at execution time.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point emits parameter definitions into the parameters section of the batch pipeline ARM template.

When to Use It

1. You need to pass an IsInitialLoad flag from a trigger or parent orchestration pipeline to control full-load vs. incremental behavior across all objects in the batch.
2. You want to pass an environment-specific configuration value (e.g., database server name) that differs between dev, test, and production.
3. You need to pass a date range filter to the batch pipeline that gets forwarded to individual object pipelines.
4. You want to add a DryRun parameter that skips actual data movement when set to true. Use this instead of AdfPipelineParameter when the parameter is needed at the batch level and potentially forwarded to child pipelines.

Prerequisites

• A batch must be defined in BimlFlex metadata.
• The batch parameter provides access to batch metadata.
• Parameter names must be unique within the batch pipeline.
• The trigger or parent pipeline must pass values for parameters that do not have defaults.

Implementation Steps

1. Create a new extension point targeting the batch.
2. Add the BimlScript directive with the parameter definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchParameter" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>

<Parameter Name="IsInitialLoad" DataType="Bool">false</Parameter>
<Parameter Name="TargetEnvironment" DataType="String">Production</Parameter>

3. Build and verify the parameters appear in the batch pipeline's parameters section.
4. Configure triggers (AdfBatchTrigger) or parent pipelines (AdfPipeline) to pass values for these parameters.

Example

Before (batch pipeline has only BimlFlex-generated parameters):

{ "name": "01_EXT_AWLT_SRC_Batch",
  "properties": { "parameters": {} }
}

After (custom parameters added):

{ "name": "01_EXT_AWLT_SRC_Batch",
  "properties": {
    "parameters": {
      "IsInitialLoad": { "type": "bool", "defaultValue": false },
      "TargetEnvironment": { "type": "string", "defaultValue": "Production" }
    }
  }
}

Common Mistakes

1. Mistake: Adding a batch parameter without passing it from the trigger or parent pipeline. Symptom: The parameter always uses its default value. Fix: Configure AdfBatchTrigger or AdfTrigger to pass the parameter value.
2. Mistake: Using AdfBatchParameter when the parameter is needed at the object pipeline level. Symptom: Individual object pipelines cannot access the batch parameter directly. Fix: Use AdfPipelineParameter for object-level parameters, and forward batch parameters using OverrideAdfExecutePipeline.
3. Mistake: Defining a parameter name that conflicts with a BimlFlex-generated parameter. Symptom: The build fails or the default parameter value is overwritten. Fix: Prefix custom parameters (e.g., EP_IsInitialLoad).

Related Extension Points

• AdfPipelineParameter -- adds parameters to object-level pipelines.
• AdfBatchVariable -- adds variables (read-write) to the batch pipeline.
• AdfBatchTrigger -- triggers that pass parameter values to the batch pipeline.
• OverrideAdfExecutePipeline -- forwards batch parameters to individual object pipelines.

Batch Variable

Configure Data Factory Batch Variables.

Parameters

Name	Type	Description
batch	BimlFlexModelWrapper.BatchesWrapper	Contains all information about the Data Factory (Batch) Pipeline to which the Parameter will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchVariable" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>


<Variable Name="MyAdfVariable" DataType="String"></Variable>

What It Does

Adds custom variables to the batch-level ADF pipeline (the orchestration pipeline for a batch such as 01_EXT_AWLT_SRC_Batch). Batch pipeline variables are scoped to the batch pipeline execution and can be used to hold intermediate values such as status flags, aggregate counts, or computed values throughout the batch's activity chain. Unlike parameters, variables can be set and updated during pipeline execution using SetVariable activities.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point emits variable definitions into the variables section of the batch pipeline ARM template.

When to Use It

1. You need a batch-scoped variable to hold a status flag that controls whether downstream object pipelines should execute (e.g., SkipExtraction).
2. You want to accumulate a total row count across object pipelines using SetVariable activities in AdfBatchPostProcess.
3. You need a variable to store the result of a pre-batch health check that other activities reference.
4. You want to pass a computed value between AdfBatchPreProcess and AdfBatchPostProcess activities. Use this instead of AdfPipelineVariable when the variable is needed at the batch pipeline level rather than individual object pipelines.

Prerequisites

• A batch must be defined in BimlFlex metadata.
• The batch parameter provides access to batch metadata.
• Variable names must be unique within the batch pipeline.

Implementation Steps

1. Create a new extension point targeting the batch.
2. Add the BimlScript directive with the variable definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchVariable" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>

<Variable Name="BatchStatus" DataType="String">pending</Variable>
<Variable Name="TotalRowCount" DataType="String">0</Variable>

3. Build and verify the variables appear in the batch pipeline's variables section.
4. Use SetVariable activities in AdfBatchPreProcess or AdfBatchPostProcess to update variable values.

Example

Before (batch pipeline has only BimlFlex-generated variables):

{ "name": "01_EXT_AWLT_SRC_Batch",
  "properties": { "variables": {} }
}

After (custom variables added):

{ "name": "01_EXT_AWLT_SRC_Batch",
  "properties": {
    "variables": {
      "BatchStatus": { "type": "String", "defaultValue": "pending" },
      "TotalRowCount": { "type": "String", "defaultValue": "0" }
    }
  }
}

Common Mistakes

1. Mistake: Using AdfBatchVariable when the variable is needed at the object pipeline level. Symptom: The variable is only available in the batch pipeline, not in individual object pipelines. Fix: Use AdfPipelineVariable for object-level variables.
2. Mistake: Defining a variable name that conflicts with a BimlFlex-generated variable. Symptom: The build fails with a duplicate variable error. Fix: Prefix custom variable names (e.g., EP_BatchStatus).
3. Mistake: Expecting variable values to persist between pipeline runs. Symptom: The variable resets to its default value on each run. Fix: Variables are scoped to a single pipeline execution. Use the BimlCatalog or external storage for persistent values.

Related Extension Points

• AdfPipelineVariable -- adds variables to object-level pipelines.
• AdfBatchParameter -- adds read-only parameters to the batch pipeline.
• AdfBatchPreProcess / AdfBatchPostProcess -- activities where SetVariable can be used to update batch variables.
• AdfSnowflakeScaleUp / AdfSnowflakeScaleDown -- Snowflake-specific batch activities that may use batch variables.

Batch Post Process

Configure logic that will be injected after the targeted main Batch Pipeline process.

Parameters

Name	Type	Description
batch	BimlFlexModelWrapper.BatchesWrapper	Contains all information about the Data Factory (Batch) Pipeline to which the Extension Point content will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchPostProcess" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>
<#@ property name="dependency" type="String" #>


<Wait Name="Wait for Ten Seconds" WaitTimeInSeconds="10">
	<Dependencies>
		<Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>"></Dependency>
	</Dependencies>
</Wait>

What It Does

Injects custom activities at the end of a batch pipeline in Azure Data Factory, after all object-level Execute Pipeline calls have completed. This is the batch-level equivalent of AdfPostProcess and runs before the batch logging finalization. Common uses include batch-level cleanup, aggregate notifications, triggering downstream processes, or running cross-object validation after all objects in the batch have been processed.

Where It Fires

Consumed in Code/DataFactoryHelper.cs and Code/BatchViewModel.cs. The extension point fires within the batch pipeline (e.g., 01_EXT_AWLT_SRC_Batch), after the last object-level ExecutePipeline activity and before LogExecutionEnd. The dependency parameter contains the name of the preceding activity.

When to Use It

1. You need to send a completion notification (e.g., email or webhook) after all objects in the 01_EXT_AWLT_SRC_Batch have been processed.
2. You want to trigger a downstream batch pipeline (e.g., 02_LOAD_BFX_DV_Batch) from within the current batch.
3. You need to run cross-object validation or reconciliation after all objects have loaded (e.g., comparing source and target row counts).
4. You want to run database maintenance tasks (e.g., rebuild indexes on landing tables) after the batch completes. Use this instead of AdfPostProcess when the logic should run once per batch rather than once per object.

Prerequisites

• A batch must be defined in BimlFlex metadata.
• The batch and dependency parameters provide access to batch metadata and the preceding activity.
• The dependency parameter must be used to chain the activity after the last object pipeline.

Implementation Steps

1. Create a new extension point targeting the batch.
2. Add the BimlScript directive with the post-process logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchPostProcess" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>
<#@ property name="dependency" type="String" #>

<Web Name="NotifyCompletion" Url="https://prod-logic-app.azurewebsites.net/api/notify"
     Method="POST">
  <Body>{"batch": "<#=batch.Name#>", "status": "completed"}</Body>
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Web>

3. Build and verify the activity appears in the batch pipeline after the last object ExecutePipeline and before LogExecutionEnd.

Example

Before (logging runs directly after object pipelines):

{ "name": "01_EXT_AWLT_SRC_Batch",
  "activities": [
    { "name": "EXT_AWLT_SRC_SalesLT_Customer", "type": "ExecutePipeline" },
    { "name": "LogExecutionEnd", "dependsOn": [{ "activity": "EXT_AWLT_SRC_SalesLT_Customer" }] }
  ]
}

After (notification inserted before logging):

{ "name": "01_EXT_AWLT_SRC_Batch",
  "activities": [
    { "name": "EXT_AWLT_SRC_SalesLT_Customer", "type": "ExecutePipeline" },
    { "name": "NotifyCompletion", "type": "WebActivity",
      "dependsOn": [{ "activity": "EXT_AWLT_SRC_SalesLT_Customer", "dependencyConditions": ["Succeeded"] }] },
    { "name": "LogExecutionEnd", "dependsOn": [{ "activity": "NotifyCompletion" }] }
  ]
}

Common Mistakes

1. Mistake: Using AdfBatchPostProcess when the logic should run per-object. Symptom: The activity runs once at the batch level instead of once per object. Fix: Use AdfPostProcess for per-object post-processing.
2. Mistake: Not referencing the dependency parameter. Symptom: The post-process runs in parallel with object pipelines. Fix: Always include <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>">.
3. Mistake: Adding a post-process activity that fails and blocks LogExecutionEnd. Symptom: The batch execution is never logged as complete. Fix: Use Condition="Completed" instead of Condition="Succeeded" if the post-process should not block logging.
4. Mistake: Confusing AdfBatchPostProcess with AdfPostProcess. Symptom: The logic runs at the wrong level (batch vs. object). Fix: Use AdfBatchPostProcess for batch-level and AdfPostProcess for object-level.

Related Extension Points

• AdfBatchPreProcess -- adds activities before object pipelines start in the batch.
• AdfPostProcess -- per-object post-processing within individual object pipelines.
• AdfSnowflakeScaleDown -- Snowflake-specific batch-level post-processing for warehouse scale-down.
• AdfBatchVariable -- defines batch variables that post-process activities can read.

Batch Pre Process

Configure logic that will be injected before the targeted main Batch Pipeline process.

Parameters

Name	Type	Description
batch	BimlFlexModelWrapper.BatchesWrapper	Contains all information about the Data Factory (Batch) Pipeline to which the Extension Point content will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchPreProcess" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- This extension point allows the addition of bespoke logic to be added as the first step in the targeted (Batch) ADF Pipeline. -->
<!-- The logic is added after the LogExecutionStart, so that the BimlCatalog can still record the initiation of the running of the Pipeline. -->
<!-- The below example shows 'Wait' step to be added before the main processing in the Pipeline begins. -->

<Wait Name="Wait for Ten Seconds" WaitTimeInSeconds="10">
	<Dependencies>
		<Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>"></Dependency>
	</Dependencies>
</Wait>

What It Does

Injects custom activities at the beginning of a batch pipeline in Azure Data Factory, immediately after the LogExecutionStart activity. This is the batch-level equivalent of AdfPreProcess and runs before any object-level Execute Pipeline calls begin. Common uses include batch-level health checks, environment preparation, notification triggers, or setting batch-wide variables before data processing starts.

Where It Fires

Consumed in Code/DataFactoryHelper.cs and Code/BatchViewModel.cs. The extension point fires within the batch pipeline (e.g., 01_EXT_AWLT_SRC_Batch), after LogExecutionStart and before the first ExecutePipeline activity. The dependency parameter contains the name of the preceding activity (typically LogExecutionStart).

When to Use It

1. You need to validate that target databases are accessible before starting the batch extraction (e.g., running a connectivity check against BFX_LND and BFX_STG).
2. You want to send a notification (e.g., email or Teams message via Logic App) at the start of a batch run.
3. You need to set batch-level variables based on external configuration before object pipelines execute.
4. You want to run a database maintenance task (e.g., update statistics) at the start of each batch. Use this instead of AdfPreProcess when the logic should run once per batch rather than once per object.

Prerequisites

• A batch must be defined in BimlFlex metadata (e.g., 01_EXT_AWLT_SRC_Batch).
• The batch and dependency parameters provide access to batch metadata and the preceding activity.
• The dependency parameter must be used to chain the activity after LogExecutionStart.

Implementation Steps

1. Create a new extension point targeting the batch.
2. Add the BimlScript directive with the pre-process logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchPreProcess" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>
<#@ property name="dependency" type="String" #>

<Script Name="BatchHealthCheck" Timeout="0.00:05:00" LinkedServiceName="BFX_STG">
  <Scripts>
    <Script ScriptType="Query">
      <Query>SELECT 1 AS HealthCheck</Query>
    </Script>
  </Scripts>
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Script>

3. Build and verify the activity appears in the batch pipeline after LogExecutionStart and before the object ExecutePipeline activities.

Example

Before (object pipelines start directly after LogExecutionStart):

{ "name": "01_EXT_AWLT_SRC_Batch",
  "activities": [
    { "name": "LogExecutionStart", "type": "SqlServerStoredProcedure" },
    { "name": "EXT_AWLT_SRC_SalesLT_Customer", "type": "ExecutePipeline",
      "dependsOn": [{ "activity": "LogExecutionStart" }] }
  ]
}

After (health check runs before object pipelines):

{ "name": "01_EXT_AWLT_SRC_Batch",
  "activities": [
    { "name": "LogExecutionStart", "type": "SqlServerStoredProcedure" },
    { "name": "BatchHealthCheck", "type": "Script",
      "dependsOn": [{ "activity": "LogExecutionStart", "dependencyConditions": ["Succeeded"] }] },
    { "name": "EXT_AWLT_SRC_SalesLT_Customer", "type": "ExecutePipeline",
      "dependsOn": [{ "activity": "BatchHealthCheck" }] }
  ]
}

Common Mistakes

1. Mistake: Using AdfBatchPreProcess when the logic should run per-object. Symptom: The activity runs once at the batch level instead of once per object pipeline. Fix: Use AdfPreProcess for per-object pre-processing.
2. Mistake: Not referencing the dependency parameter. Symptom: The pre-process activity runs in parallel with LogExecutionStart. Fix: Always include <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>">.
3. Mistake: Adding a long-running activity that delays all object pipelines. Symptom: The entire batch is delayed waiting for the pre-process to complete. Fix: Keep pre-process logic lightweight or use async patterns.
4. Mistake: Confusing AdfBatchPreProcess with AdfPreProcess. Symptom: The logic runs at the wrong level (batch vs. object). Fix: Use AdfBatchPreProcess for batch-level and AdfPreProcess for object-level.

Related Extension Points

• AdfBatchPostProcess -- adds activities after all object pipelines complete in the batch.
• AdfPreProcess -- per-object pre-processing within individual object pipelines.
• AdfBatchParameter / AdfBatchVariable -- define batch pipeline parameters and variables used by pre-process activities.
• AdfSnowflakeScaleUp -- Snowflake-specific batch-level pre-processing for warehouse scaling.

Dataset

Configure additional Dataset references to an Data Factory.

Parameters

Name	Type	Description
instance	BimlFlexModelWrapper	The metadata instance of which the Data Factory is part. There is no 'Data Factory' object in the metadata model that can directly be accessed, and the instance object allows for full access to all metadata to define the specific behaviour of the Linked Service that is added using this Extension Point.
dataFactoryName	String	The Data Factory name the Extension Point is applied to. E.g. where the Linked Service will be added.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfDataset" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>


<# 	CustomOutput.ObjectInherit = true; #>
<AzureBlob Name="AZB_AdfDataset_SRC" LinkedServiceName="AdfLinkedService_SRC" AdfFolder="Custom" FolderPath="@dataset().DatasetFolder" FileSystem="@dataset().DatasetContainer">
	<Binary  />
	<Parameters>
		<Parameter Name="DatasetContainer" DataType="String">DatasetContainerPlaceholder</Parameter>
		<Parameter Name="DatasetFolder" DataType="String">DatasetFolderPlaceholder</Parameter>
	</Parameters>
</AzureBlob>

What It Does

Adds entirely new Dataset definitions to the Azure Data Factory alongside the Datasets BimlFlex generates from connection metadata. Datasets in ADF define the structure and location of data -- the Linked Service, table or file path, schema, and parameterization. This extension point creates additional Dataset resources that can be referenced by Copy Activities, Lookup activities, or other pipeline activities in your extension points.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point emits Dataset JSON resources at the factory level in the generated ARM template. It is evaluated once per data factory during output generation.

When to Use It

1. You need a Dataset for a data store that is not represented as a BimlFlex connection (e.g., a custom Blob Storage container for file archives or staging).
2. You want a parameterized Dataset that multiple Copy Activities can share by passing different container and folder values at runtime.
3. You need a Dataset for a REST API or other connector type that BimlFlex does not auto-generate.
4. You want to create a Binary Dataset for file copy operations (e.g., archiving source files to a different storage location). Use this instead of OverrideAdfDataset when you need to create a new Dataset rather than modifying an existing one.

Prerequisites

• The Data Factory connection (adf-bimlflex-prod) must be configured in BimlFlex metadata.
• The instance and dataFactoryName parameters provide access to metadata and the factory name.
• The Linked Service referenced by the Dataset must exist (either generated by BimlFlex or added via AdfLinkedService).

Implementation Steps

1. Create a new extension point targeting the data factory.
2. Add the BimlScript directive with the Dataset definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfDataset" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>

<# CustomOutput.ObjectInherit = true; #>
<AzureBlob Name="AZB_Archive" LinkedServiceName="AdfLinkedService_StagingBlob"
    AdfFolder="Custom" FolderPath="@dataset().ArchiveFolder"
    FileSystem="@dataset().ArchiveContainer">
  <Binary />
  <Parameters>
    <Parameter Name="ArchiveContainer" DataType="String">archive</Parameter>
    <Parameter Name="ArchiveFolder" DataType="String">default</Parameter>
  </Parameters>
</AzureBlob>

3. Build and verify the Dataset appears in the ARM template output under Microsoft.DataFactory/factories/datasets.
4. Reference the Dataset in Copy Activities within other extension points (e.g., AdfPostCopy, AdfPipeline).

Example

Before (only BimlFlex-generated Datasets):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/datasets", "name": "SQLDB_AWLT_SRC" },
    { "type": "Microsoft.DataFactory/factories/datasets", "name": "SQLDB_BFX_LND" }
  ]
}

After (custom archive Dataset added):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/datasets", "name": "SQLDB_AWLT_SRC" },
    { "type": "Microsoft.DataFactory/factories/datasets", "name": "SQLDB_BFX_LND" },
    { "type": "Microsoft.DataFactory/factories/datasets", "name": "AZB_Archive",
      "properties": {
        "type": "AzureBlob",
        "linkedServiceName": { "referenceName": "AdfLinkedService_StagingBlob" },
        "typeProperties": {
          "folderPath": { "type": "Expression", "value": "@dataset().ArchiveFolder" }
        },
        "parameters": {
          "ArchiveContainer": { "type": "string", "defaultValue": "archive" },
          "ArchiveFolder": { "type": "string", "defaultValue": "default" }
        }
      }
    }
  ]
}

Common Mistakes

1. Mistake: Using AdfDataset to modify an existing auto-generated Dataset. Symptom: A duplicate Dataset is created. Fix: Use OverrideAdfDataset to modify existing Datasets.
2. Mistake: Creating a Dataset name that conflicts with an auto-generated one. Symptom: The build fails or the auto-generated Dataset is overwritten. Fix: Use a distinct naming convention (e.g., prefix with AZB_, EP_).
3. Mistake: Referencing a Linked Service that does not exist. Symptom: ADF deployment fails with a "linked service not found" validation error. Fix: Ensure the Linked Service is defined via AdfLinkedService or BimlFlex metadata.
4. Mistake: Not parameterizing the Dataset when it needs to support multiple tables or files. Symptom: Separate Datasets must be created for each table, leading to maintenance overhead. Fix: Use Dataset parameters for dynamic resolution (e.g., @dataset().TableName).

Related Extension Points

• OverrideAdfDataset -- overrides an existing auto-generated Dataset.
• AdfLinkedService -- defines the Linked Service that the Dataset references.
• OverrideAdfCopySource / OverrideAdfCopySink -- Copy Activity source/sink definitions that reference Datasets.
• AdfPipeline -- custom pipelines whose activities reference the Dataset.

External File Format

Configure the file format for PolyBase External Tables when using SSIS (does not apply to Data Factory Pipelines). This is used in SSIS for files that are uploaded to Blob Storage and then loaded through external tables into Synapse.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object to which the file format will be added.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="ExternalFileFormat" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>



<!-- This Extension Point overrides the 'Pre-copy script' property in the Sink definition of the targeted Copy Activity. -->
<!-- The below example populates a Pre-copy script. -->

<# 	CustomOutput.ObjectInherit = true; #>
<#	var sourceConnection = EntityHelper.GetSourceConnection(table);
	var sourceScopedName = table.GetSourceScopedName(sourceConnection);
	var sourceSsisSafeScopedName = sourceScopedName.MakeSsisSafe();#>
WITH
(
	LOCATION = '<#=sourceSsisSafeScopedName #>/'
	, DATA_SOURCE = dwhload_storage
	, FILE_FORMAT = pipe_zip_format
)

What It Does

Configures the external file format definition for PolyBase external tables when loading data into Azure Synapse Analytics (formerly SQL Data Warehouse) via SSIS. This extension point controls the WITH clause of the CREATE EXTERNAL TABLE statement, defining the file location, data source, and file format used to load data from Azure Blob Storage into Synapse through external tables. Note that this extension point applies to SSIS-based loads using PolyBase, not directly to ADF pipelines.

Where It Fires

Consumed in Code/ObjectViewModel.cs, Code/EntityHelper.cs, Code/ScriptProvider.cs, and 90.flx-i-get-azure-stage-on-extract.biml. The extension point is referenced by 12 files in the BimlFlex codebase. It fires for each source object during generation when IsSourceStagedFile == true, meaning the object uses file-based extraction through Blob Storage and external tables.

When to Use It

1. You need to change the file format from the default (e.g., pipe-delimited compressed) to Parquet or CSV for specific source objects.
2. You want to customize the external data source name or path structure for the PolyBase external table.
3. You need to change the compression type (e.g., from GZip to Snappy) for performance reasons.
4. You want to add additional WITH clause options such as REJECT_TYPE, REJECT_VALUE, or REJECT_SAMPLE_VALUE for error handling. Use this instead of AdfSnowflakeCreateStage when working with Synapse/SQL DW PolyBase rather than Snowflake stages.
5. Define a custom PolyBase external table format for pipe-delimited compressed files. Your Synapse Analytics (SQL DW) load uses PolyBase with external tables. Override the default file format to specify a custom LOCATION (derived from the source object's scoped name), DATA_SOURCE, and FILE_FORMAT (e.g., pipe_zip_format for pipe-delimited, gzip-compressed files) to match your staging blob structure.

Prerequisites

• The target must be Azure Synapse Analytics (SQL DW) using PolyBase external tables.
• The source object must be configured for file-based extraction (IsSourceStagedFile == true).
• The external data source and file format objects must exist on the Synapse database.
• The table parameter provides access to the source object metadata.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the external file format definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="ExternalFileFormat" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<# CustomOutput.ObjectInherit = true; #>
<#
var sourceConnection = EntityHelper.GetSourceConnection(table);
var sourceScopedName = table.GetSourceScopedName(sourceConnection);
var sourceSsisSafeScopedName = sourceScopedName.MakeSsisSafe();
#>
WITH
(
  LOCATION = '<#=sourceSsisSafeScopedName#>/'
  , DATA_SOURCE = dwhload_storage
  , FILE_FORMAT = parquet_format
  , REJECT_TYPE = VALUE
  , REJECT_VALUE = 0
)

3. Build and verify the CREATE EXTERNAL TABLE statement uses your custom WITH clause.

Example

Before (default pipe-delimited compressed format):

CREATE EXTERNAL TABLE [stg].[EXT_SalesLT_Customer]
( ... )
WITH
(
  LOCATION = 'SalesLT_Customer/'
  , DATA_SOURCE = dwhload_storage
  , FILE_FORMAT = pipe_zip_format
)

After (Parquet format with error handling):

CREATE EXTERNAL TABLE [stg].[EXT_SalesLT_Customer]
( ... )
WITH
(
  LOCATION = 'SalesLT_Customer/'
  , DATA_SOURCE = dwhload_storage
  , FILE_FORMAT = parquet_format
  , REJECT_TYPE = VALUE
  , REJECT_VALUE = 0
)

Common Mistakes

1. Mistake: Using this extension point for ADF pipelines. Symptom: The extension point has no effect because ADF does not use PolyBase external tables in the same way. Fix: For ADF-based loads into Synapse, use OverrideAdfCopySink with enableStaging and the COPY command.
2. Mistake: Referencing a file format object that does not exist on the Synapse database. Symptom: The CREATE EXTERNAL TABLE fails with "file format not found". Fix: Create the file format object on Synapse before running the load (e.g., CREATE EXTERNAL FILE FORMAT parquet_format WITH (FORMAT_TYPE = PARQUET)).
3. Mistake: Setting ObjectInherit = false when applying at batch level. Symptom: Only one object gets the custom file format. Fix: Set CustomOutput.ObjectInherit = true; when applying at batch level.
4. Mistake: Incorrect LOCATION path that does not match where files are uploaded. Symptom: The external table query returns zero rows. Fix: Use sourceSsisSafeScopedName to ensure the path matches the upload structure.

Related Extension Points

• AdfSnowflakeCreateStage -- the Snowflake equivalent for creating external stages (not PolyBase).
• OverrideAdfCopySink -- for ADF-based loads, override the sink to use COPY command with staging.
• OverrideAdfStagingSettings -- configure staging storage for ADF Copy Activities loading into Synapse.
• AdfProcessStaging -- the staging process that may rely on external tables.

Get Parameter

Override the parameters that are retrieved from metadata, and how they behave in the Data Factory Pipeline. Note that this extension point replaces all paramaters.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object (e.g. table or file) to which the process will be added.
dependency	String	Contains the dependency name for the previous activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the batch.
OutputPathName	String	You must add CustomOutput.OutputPathName with the last task to connect it with the next Data Flow task.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfGetParameter" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- This extension point allows the manipulation of metadata parameters that are used in the data pipelines, for example to create load windows or high-water mark values. -->
<!-- The most common use-case is to change the behaviour of the parameters that are already defined in the metadata. -->
<!-- In this case, the parameter code will be replaced by the contents of this Extension Point. -->

<!-- This extension point affects all Parameters that are added before the main 'Copy Data' activity in the targeted ADF Pipeline. -->
<!-- The below example shows how parameters (e.g. LkpLastLoadDate, LlkpNextLoadDate) are replaced by values retrieved from metadata. -->
<!-- Compared to the default template, this override uses a Lookup Activity to retrieve the target parameter value. -->

<# 
// Object inheritance is disabled for this extension point example.
// This means the target can be set either as individual object or parent objects (e.g. Batch-level), but that this extension will not be applied to any child objects.
CustomOutput.ObjectInherit = false;
#>

<#	
	var srcObj = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC");
	var parameters = srcObj.Parameters;
  
	if (!parameters.Any()) return null; 
  
	foreach(var parameter in parameters){
		if (parameter.IsNotPersisted == "Y") continue;
    
		var srcDatasetType = srcObj.SourceDatasetType;
		var parameterColumn = parameter.Column.RelatedItem != null ? parameter.Column.RelatedItem.Name : "";
		parameterColumn = string.IsNullOrWhiteSpace(parameterColumn)
			? parameter.Object.RelatedItem != null ? parameter.Object.RelatedItem.Name + "." + parameter.ParameterName : parameter.ParameterName
			: parameterColumn;
		var parameterSourceElement = srcObj.SourceAdfSourceElement;
#>
	<Lookup Name="Lkp<#=parameter.ParameterName#>">
    	<<#=parameterSourceElement#> StoredProcedure="[adf].[GetConfigVariable]" DatasetName="BimlCatalog">
			<StoredProcedureParameters>
				<Parameter Name="SystemName" DataType="String" IsNull="false"><#=srcObj.SourceConnection.Name#></Parameter>
				<Parameter Name="ObjectName" DataType="String" IsNull="false"><#=parameterColumn#></Parameter>
				<Parameter Name="VariableName" DataType="String" IsNull="false"><#=parameter.ParameterName#></Parameter>
				<Parameter Name="VariableValue" DataType="String" IsNull="false"><#=parameter.ParameterDefault#></Parameter>
				<Parameter Name="ExecutionID" DataType="Int64">@activity(&apos;LogExecutionStart&apos;).output.firstRow.ExecutionID</Parameter>
			</StoredProcedureParameters>
		</<#=parameterSourceElement#>>
	</Lookup>
	<Lookup Name="Lkp<#=parameter.ParameterToName#>" FirstRowOnly="true">
		<Dependencies>
			<Dependency DependsOnActivityName="Lkp<#=parameter.ParameterName#>" />
		</Dependencies>
		<<#=parameterSourceElement#> DatasetName="<#=srcObj.SourceDatasetName#>">
			<Query><#=parameter.GetSourceParameterSql() #></Query>
		</<#=parameterSourceElement#>>
	</Lookup>
	<SetVariable Name="Set<#=parameter.ParameterName#>" VariableName="<#=parameter.ParameterName#>">
		<Dependencies>
			<Dependency Condition="Succeeded" DependsOnActivityName="Lkp<#=parameter.ParameterName#>"></Dependency>
		</Dependencies>
		<Value>
			@activity('Lkp<#=parameter.ParameterName#>').output.firstRow.VariableValue
		</Value>
	</SetVariable>	
	<SetVariable Name="Set<#=parameter.ParameterToName#>" VariableName="<#=parameter.ParameterToName#>">
		<Dependencies>
			<Dependency Condition="Succeeded" DependsOnActivityName="Set<#=parameter.ParameterName#>"></Dependency>
			<Dependency Condition="Succeeded" DependsOnActivityName="Lkp<#=parameter.ParameterToName#>"></Dependency>
		</Dependencies>
		<Value>
			@{if(contains(activity('Lkp<#=parameter.ParameterToName#>').output,'firstRow'), activity('Lkp<#=parameter.ParameterToName#>').output.firstRow.<#=parameter.ParameterToName#>, variables('<#=parameter.ParameterName#>'))}
		</Value>
	</SetVariable>
<# CustomOutput.OutputPathName = $"Set{parameter.ParameterToName}";
}
#>

What It Does

Overrides the default logic that retrieves parameter values (such as high-water mark dates and load window boundaries) from the BimlCatalog at the start of a data pipeline. By default, BimlFlex generates Lookup and SetVariable activities that call [adf].[GetConfigVariable] to populate load parameters before the Copy Activity runs. This extension point replaces all of those retrieval activities, giving you full control over how parameters are sourced and set for incremental or windowed data extraction.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline, before the Copy Activity, in the parameter retrieval section of the MainActivity IfCondition. It replaces the Lookup and SetVariable activities for all parameters on the targeted object.

When to Use It

1. You need to change the stored procedure used to retrieve parameter values (e.g., using a custom lookup instead of [adf].[GetConfigVariable]).
2. You want to retrieve parameter values from a different source (e.g., an Azure SQL Database instead of the BimlCatalog).
3. You need to add additional logic before setting parameter values (e.g., date arithmetic to compute a lookback window).
4. You want to replace the Lookup-based approach with a Web Activity that calls a REST API for parameter values. Use this instead of AdfSetParameter when you need to control parameter retrieval, not the write-back.

Prerequisites

• Parameters must be defined on the source object in BimlFlex metadata (e.g., LastLoadDate on SalesLT.Customer).
• The BimlCatalog database Linked Service must be configured (or whichever source the custom logic targets).
• The dependency parameter provides the name of the preceding activity for chaining.
• CustomOutput.OutputPathName must be set to the last activity name so the Copy Activity can depend on it.

Implementation Steps

1. Create a new extension point targeting the source object (e.g., SalesLT.Customer on AWLT_SRC).
2. Add the BimlScript directive with the custom parameter retrieval logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfGetParameter" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = false; #>
<#
var srcObj = new ObjectViewModel(table, table.Connection.RelatedItem.IntegrationStage, "SRC");
var parameters = srcObj.Parameters;
if (!parameters.Any()) return null;
foreach(var parameter in parameters) {
  if (parameter.IsNotPersisted == "Y") continue;
#>
<Lookup Name="Lkp<#=parameter.ParameterName#>">
  <AzureSqlDatabaseSource StoredProcedure="[adf].[GetConfigVariable]"
      DatasetName="BimlCatalog">
    <StoredProcedureParameters>
      <Parameter Name="SystemName" DataType="String"><#=srcObj.SourceConnection.Name#></Parameter>
      <Parameter Name="ObjectName" DataType="String"><#=parameter.ParameterName#></Parameter>
      <Parameter Name="VariableName" DataType="String"><#=parameter.ParameterName#></Parameter>
      <Parameter Name="VariableValue" DataType="String"><#=parameter.ParameterDefault#></Parameter>
      <Parameter Name="ExecutionID" DataType="Int64">@activity('LogExecutionStart').output.firstRow.ExecutionID</Parameter>
    </StoredProcedureParameters>
  </AzureSqlDatabaseSource>
</Lookup>
<SetVariable Name="Set<#=parameter.ParameterName#>" VariableName="<#=parameter.ParameterName#>">
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="Lkp<#=parameter.ParameterName#>" />
  </Dependencies>
  <Value>@activity('Lkp<#=parameter.ParameterName#>').output.firstRow.VariableValue</Value>
</SetVariable>
<# CustomOutput.OutputPathName = $"Set{parameter.ParameterName}";
} #>

3. Build and verify the Lookup and SetVariable activities appear in the pipeline with your custom logic.

Example

Before (default parameter retrieval):

{ "activities": [
    { "name": "LkpLastLoadDate", "type": "Lookup",
      "typeProperties": { "source": { "sqlReaderStoredProcedureName": "[adf].[GetConfigVariable]" } } },
    { "name": "SetLastLoadDate", "type": "SetVariable",
      "typeProperties": { "value": "@activity('LkpLastLoadDate').output.firstRow.VariableValue" } }
  ]
}

After (custom parameter retrieval with lookback window):

{ "activities": [
    { "name": "LkpLastLoadDate", "type": "Lookup",
      "typeProperties": { "source": { "sqlReaderStoredProcedureName": "[custom].[GetLoadWindow]" } } },
    { "name": "SetLastLoadDate", "type": "SetVariable",
      "typeProperties": { "value": "@adddays(activity('LkpLastLoadDate').output.firstRow.VariableValue, -1)" } }
  ]
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName to the last SetVariable activity. Symptom: The Copy Activity runs before parameter values are set, resulting in a full load instead of incremental. Fix: Always set CustomOutput.OutputPathName to the name of the last activity in the parameter chain.
2. Mistake: Overriding AdfGetParameter without reviewing AdfSetParameter. Symptom: The parameter names used in the set logic do not match what was retrieved. Fix: Ensure both extension points use consistent parameter and variable names.
3. Mistake: Not returning null when there are no parameters. Symptom: An empty activity block is generated. Fix: Include if (!parameters.Any()) return null; at the top.
4. Mistake: Using hardcoded parameter defaults instead of dynamic values. Symptom: The first run always uses the hardcoded default instead of the stored value. Fix: Use parameter.ParameterDefault and let the BimlCatalog provide the actual stored value.

Related Extension Points

• AdfSetParameter -- persists parameter values after the main processing; the counterpart to this extension point.
• AdfPreProcess -- adds custom logic before the main processing but after parameter retrieval.
• AdfPipelineVariable -- defines variables that parameters are stored in during execution.
• AdfPreCopyScript -- the pre-copy script that may reference parameter-derived values.

Global Parameter

Configure a Global Parameter in Data Factory.

Parameters

Name	Type	Description
instance	BimlFlexModelWrapper	The metadata instance to which the Data Factory belongs. There is no 'Data Factory' object in the metadata model that can directly be accessed, and the instance object allows for full access to all metadata to define the specific behaviour of the Integration Runtime that is added using this Extension Point.
dataFactoryName	String	The Data Factory name the Extension Point is applied to. E.g. where the Integration Runtime will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfGlobalParameter" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>



<!-- The below example adds a Schedule Trigger and connects it to a Data Factory. -->

<Schedule Name="ScheduleTriggerName" Frequency="Hour" Interval="1" Start="2021-01-01" End="2025-12-31">
	<Pipelines>
		<Pipeline PipelineName="MyDataFactory">
			<Parameters>
				<Parameter Name="IsInitialLoad">false</Parameter>
			</Parameters>
		</Pipeline>
	</Pipelines>
</Schedule>

What It Does

Adds global parameters to the Azure Data Factory. Global parameters are factory-level key-value pairs that are accessible from any pipeline via pipeline().globalParameters.ParameterName. They provide a centralized way to manage configuration values that apply across all pipelines, such as environment names, shared connection strings, or feature flags. Unlike pipeline parameters, global parameters do not need to be passed between pipelines.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point emits global parameter definitions in the Data Factory ARM template at the factory level. It is evaluated once per data factory during output generation.

When to Use It

1. You need a shared environment identifier (e.g., Environment = "Production") accessible from all pipelines without passing it as a parameter.
2. You want to store a global feature flag (e.g., EnableVerboseLogging = true) that controls behavior across all pipelines.
3. You need a shared configuration value (e.g., a notification email address or webhook URL) referenced by multiple pipelines.
4. You want to centralize timeout or retry settings that apply to all pipelines. Use this instead of AdfBatchParameter or AdfPipelineParameter when the value should be shared across all pipelines without explicit parameter passing.

Prerequisites

• The Data Factory connection (adf-bimlflex-prod) must be configured in BimlFlex metadata.
• The instance and dataFactoryName parameters provide access to metadata and the factory name.
• Global parameter names must be unique at the factory level.

Implementation Steps

1. Create a new extension point targeting the data factory.
2. Add the BimlScript directive with the global parameter definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfGlobalParameter" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>

<GlobalParameter Name="Environment" DataType="String">Production</GlobalParameter>
<GlobalParameter Name="EnableVerboseLogging" DataType="Bool">false</GlobalParameter>

3. Build and verify the global parameters appear in the ARM template output.
4. Reference global parameters in pipeline activities using @pipeline().globalParameters.Environment.

Example

Before (no global parameters):

{ "name": "adf-bimlflex-prod",
  "type": "Microsoft.DataFactory/factories",
  "properties": { "globalParameters": {} }
}

After (global parameters added):

{ "name": "adf-bimlflex-prod",
  "type": "Microsoft.DataFactory/factories",
  "properties": {
    "globalParameters": {
      "Environment": { "type": "String", "value": "Production" },
      "EnableVerboseLogging": { "type": "Bool", "value": false }
    }
  }
}

Common Mistakes

1. Mistake: Overusing global parameters for values that should be pipeline-specific. Symptom: Global parameters become a dumping ground for configuration, making the factory harder to manage. Fix: Use pipeline parameters for values specific to individual pipelines.
2. Mistake: Not updating global parameter values after deployment. Symptom: The production factory uses development values. Fix: Update global parameters via the ADF portal or deployment scripts for each environment.
3. Mistake: Using a global parameter name that conflicts with a pipeline parameter name. Symptom: Pipeline expressions reference the wrong parameter scope. Fix: Use a distinct naming convention (e.g., prefix global parameters with Global_).

Related Extension Points

• AdfBatchParameter -- batch pipeline-level parameters.
• AdfPipelineParameter -- object pipeline-level parameters.
• AdfPipeline -- custom pipelines that can reference global parameters.
• AdfTrigger -- triggers that can pass global parameter values to pipelines.

Integration Runtime

Adds a bespoke Integration Runtime to an Data Factory.

Parameters

Name	Type	Description
instance	BimlFlexModelWrapper	The metadata instance to which the Data Factory belongs. There is no 'Data Factory' object in the metadata model that can directly be accessed, and the instance object allows for full access to all metadata to define the specific behaviour of the Integration Runtime that is added using this Extension Point.
dataFactoryName	String	The Data Factory name the Extension Point is applied to. E.g. where the Integration Runtime will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfIntegrationRuntime" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>



<!-- The below example is adding two bespoke Integration Runtimes to the specified Data Factory. -->

<LinkedSelfHostedIntegrationRuntime Name="MyLinkedSelfHostedIntegrationRuntime">
	<RbacAuthorization AuthorizationType="Rbac" ResourceId="/subscriptions/subscriptionid/resourcegroups/resourcegroupname/providers/Microsoft.DataFactory/factories/datafactoryname/integrationruntimes/integrationruntimename"></RbacAuthorization>
</LinkedSelfHostedIntegrationRuntime>
<SelfHostedIntegrationRuntime Name="MySelfHostedIntegrationRuntime">
</SelfHostedIntegrationRuntime>

What It Does

Adds custom Integration Runtime definitions to the Azure Data Factory. Integration Runtimes (IR) provide the compute infrastructure for ADF activities -- Azure IR for cloud-to-cloud data movement, Self-Hosted IR for accessing on-premises or private network resources, and Linked Self-Hosted IR for sharing IRs across factories. This extension point creates IR resources that Linked Services can reference via ConnectVia.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point emits Integration Runtime JSON resources at the factory level in the generated ARM template. It is evaluated once per data factory during output generation.

When to Use It

1. You need a Self-Hosted Integration Runtime for extracting data from on-premises SQL Server (e.g., AWLT_SRC running on a local server) into Azure.
2. You want a Linked Self-Hosted IR that shares an existing IR from another Data Factory using RBAC authorization.
3. You need a custom Azure IR in a specific region for data residency compliance (e.g., a managed VNet IR for secure data movement).
4. You need multiple IRs for different connectivity scenarios (e.g., one for on-premises access and another for cross-region Azure access). Use this instead of manually creating IRs in the ADF portal, because the extension point ensures the IR definition is included in ARM template deployments.

Prerequisites

• The Data Factory connection (adf-bimlflex-prod) must be configured in BimlFlex metadata.
• The instance and dataFactoryName parameters provide access to metadata and the factory name.
• For Linked Self-Hosted IR, the source Data Factory and IR must already exist and RBAC permissions must be configured.
• For Self-Hosted IR, the IR software must be installed on the on-premises machine after deployment.

Implementation Steps

1. Create a new extension point targeting the data factory.
2. Add the BimlScript directive with the IR definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfIntegrationRuntime" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>

<SelfHostedIntegrationRuntime Name="OnPremises_IR" />
<LinkedSelfHostedIntegrationRuntime Name="Shared_IR">
  <RbacAuthorization AuthorizationType="Rbac"
      ResourceId="/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-data/providers/Microsoft.DataFactory/factories/adf-shared/integrationruntimes/SharedIR" />
</LinkedSelfHostedIntegrationRuntime>

3. Build and verify the IR definitions appear in the ARM template output.
4. After deployment, install the Self-Hosted IR software on the on-premises machine and register it with the generated IR key.

Example

Before (only default AutoResolveIntegrationRuntime):

{ "resources": [] }

After (Self-Hosted IR added):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/integrationRuntimes",
      "name": "OnPremises_IR",
      "properties": { "type": "SelfHosted" }
    },
    { "type": "Microsoft.DataFactory/factories/integrationRuntimes",
      "name": "Shared_IR",
      "properties": {
        "type": "SelfHosted",
        "typeProperties": {
          "linkedInfo": {
            "authorizationType": "Rbac",
            "resourceId": "/subscriptions/.../integrationruntimes/SharedIR"
          }
        }
      }
    }
  ]
}

Common Mistakes

1. Mistake: Defining an IR but not referencing it from any Linked Service. Symptom: The IR is created but unused; data movement still uses the default Azure IR. Fix: Update Linked Services (via OverrideAdfLinkedService or metadata) to include <ConnectVia> referencing the new IR.
2. Mistake: Using an incorrect ResourceId for a Linked Self-Hosted IR. Symptom: ADF deployment fails with a "resource not found" error. Fix: Verify the full ARM resource ID of the source IR in the Azure portal.
3. Mistake: Forgetting to install and register the Self-Hosted IR software after deployment. Symptom: The IR appears in ADF but shows as "Unavailable". Fix: Download and install the IR software from the ADF portal and register it using the generated key.
4. Mistake: Not configuring firewall rules for the Self-Hosted IR machine. Symptom: The IR cannot connect to on-premises databases or Azure services. Fix: Configure outbound firewall rules per the ADF documentation.

Related Extension Points

• AdfLinkedService -- defines Linked Services that reference the IR via ConnectVia.
• OverrideAdfLinkedService -- overrides existing Linked Services to use the custom IR.
• AdfPipeline -- custom pipelines whose activities may use Linked Services referencing the IR.
• AdfDataset -- datasets that use Linked Services connected through the IR.

Linked Service

Configure additional Linked Service references to an Data Factory.

Parameters

Name	Type	Description
instance	BimlFlexModelWrapper	The metadata instance of which the Data Factory is part. There is no 'Data Factory' object in the metadata model that can directly be accessed, and the instance object allows for full access to all metadata to define the specific behaviour of the Linked Service that is added using this Extension Point.
dataFactoryName	String	The Data Factory name the Extension Point is applied to. E.g. where the Linked Service will be added.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfLinkedService" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>


<# 	CustomOutput.ObjectInherit = true; #>
<AzureBlobStorage Name="AdfLinkedService_SRC" AuthenticationMethod="AccountKey">
	<ConnectionStringKVS StoreName="AdfKeyVault" Name="AdfLinkedService-SRC-ConnectionString"/>
</AzureBlobStorage>

What It Does

Adds entirely new Linked Service definitions to the Azure Data Factory alongside the Linked Services BimlFlex generates from connection metadata. Linked Services are the ADF equivalent of connection managers -- they define how ADF connects to external data stores, compute services, or key vaults. This extension point creates additional Linked Service resources that can be referenced by custom Datasets, Copy Activities, or Script Activities in other extension points.

Where It Fires

Consumed in Code/LinkedServiceHelper.cs, with example templates in ExtensionPoints/AdfLinkedService1.biml. The extension point emits Linked Service JSON resources at the factory level in the generated ARM template. It is referenced by 8 files in the BimlFlex codebase and evaluated once per data factory during output generation.

When to Use It

1. You need a Linked Service for a data store that is not represented as a BimlFlex connection (e.g., a REST API, an SFTP server, or a third-party SaaS connector).
2. You want to add a staging Blob Storage Linked Service for use with OverrideAdfStagingSettings or OverrideAdfLogSettings.
3. You need a second Linked Service pointing to the same database but with different credentials (e.g., read-only vs. read-write).
4. You want to add a Key Vault Linked Service for managing secrets. Use this instead of OverrideAdfLinkedService when you need to create a new Linked Service rather than modifying an existing one.
5. Add a linked service for a secondary storage account used in archival. Your ADF pipelines need access to a blob storage account that isn't defined as a BimlFlex connection (e.g., a compliance archive account). Use AdfLinkedService to define the linked service with connection string or Key Vault reference so archive activities can reference it.

Prerequisites

• The Data Factory connection (adf-bimlflex-prod) must be configured in BimlFlex metadata.
• The instance and dataFactoryName parameters provide access to metadata and the factory name.
• If the Linked Service uses Key Vault references, the Key Vault Linked Service must also be defined.
• The Integration Runtime referenced by ConnectVia must exist.

Implementation Steps

1. Create a new extension point targeting the data factory.
2. Add the BimlScript directive with the Linked Service definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfLinkedService" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>

<# CustomOutput.ObjectInherit = true; #>
<AzureBlobStorage Name="AdfLinkedService_StagingBlob"
    AuthenticationMethod="AccountKey">
  <ConnectionStringKVS StoreName="AdfKeyVault"
      Name="StagingBlob-ConnectionString" />
</AzureBlobStorage>

3. Build and verify the Linked Service appears in the ARM template output under Microsoft.DataFactory/factories/linkedservices.
4. Reference the new Linked Service in other extension points (e.g., AdfDataset, OverrideAdfStagingSettings).

Example

Before (only BimlFlex-generated Linked Services):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/linkedservices", "name": "AWLT_SRC" },
    { "type": "Microsoft.DataFactory/factories/linkedservices", "name": "BFX_LND" }
  ]
}

After (custom staging Linked Service added):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/linkedservices", "name": "AWLT_SRC" },
    { "type": "Microsoft.DataFactory/factories/linkedservices", "name": "BFX_LND" },
    { "type": "Microsoft.DataFactory/factories/linkedservices", "name": "AdfLinkedService_StagingBlob",
      "properties": {
        "type": "AzureBlobStorage",
        "typeProperties": {
          "connectionString": { "type": "AzureKeyVaultSecret",
            "store": { "referenceName": "AdfKeyVault" },
            "secretName": "StagingBlob-ConnectionString" }
        }
      }
    }
  ]
}

Common Mistakes

1. Mistake: Using AdfLinkedService to modify an existing auto-generated Linked Service. Symptom: A duplicate Linked Service is created with a different name. Fix: Use OverrideAdfLinkedService to modify existing Linked Services.
2. Mistake: Creating a Linked Service name that conflicts with an auto-generated one. Symptom: The build fails or the auto-generated Linked Service is unexpectedly overwritten. Fix: Use a distinct naming convention (e.g., prefix with Custom_ or EP_).
3. Mistake: Referencing a Key Vault Linked Service that does not exist. Symptom: ADF deployment fails with a "linked service not found" error for the Key Vault. Fix: Ensure the Key Vault Linked Service is defined in BimlFlex metadata or via this extension point.
4. Mistake: Not specifying ConnectVia when a Self-Hosted Integration Runtime is needed. Symptom: The Linked Service uses the default Azure IR and cannot reach on-premises resources. Fix: Add a <ConnectVia> section referencing the correct IR.

Related Extension Points

• OverrideAdfLinkedService -- overrides an existing auto-generated Linked Service.
• AdfDataset -- defines custom Datasets that reference the new Linked Service.
• AdfIntegrationRuntime -- defines custom Integration Runtimes for the Linked Service.
• OverrideAdfStagingSettings / OverrideAdfLogSettings -- use custom Linked Services for staging and logging.

Override Copy Sink

Configure a Copy Activity sink in Data Factory. This overrides the entire sink definition for an Data Factory Copy Activity.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains the Object (e.g. table) for which the sink properties for the Activity Sink will be overridden.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfCopySink" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>



<!-- This Extension Point overrides the entire 'Sink' definition for the targeted Copy Activity in an ADF Pipeline. -->
<!-- The below example replaces the existing sink properties for the Copy Activity with an specific Dataset and Parameters. -->

<AzureSqlDatabaseSink DatasetName="SQLDB_BFX_LND">
	<Parameters>
		<Parameter Name="TargetSchemaName">@pipeline().parameters.TargetSchemaName</Parameter>
		<Parameter Name="TargetTableName">@pipeline().parameters.TargetTableName</Parameter>
	</Parameters>
</AzureSqlDatabaseSink>

What It Does

Completely replaces the sink (destination) definition of a Copy Activity in an ADF pipeline. The sink definition controls which Dataset receives the data, write behavior (insert, upsert, stored procedure), pre-copy scripts, and any sink-specific parameters. This extension point gives you full control over how data is written to the target, including the ability to change the target Dataset, write method, or table mapping.

Where It Fires

Consumed in Code/DataFactoryHelper.cs. The extension point replaces the entire sink definition within the Copy Activity for the targeted object in the generated ARM template. It fires once per Copy Activity for each targeted object.

When to Use It

1. You need to use a stored procedure sink instead of direct table insert for the landing table (e.g., to apply custom merge logic during the copy).
2. You want to redirect data to a different target Dataset than BimlFlex generates by default.
3. You need to change the write behavior (e.g., from Insert to Upsert with a specific key column).
4. You want to add table-level lock hints or bulk insert settings for performance optimization. Use this instead of AdfPreCopyScript when you need to change the entire sink definition, not just the pre-copy script.

Prerequisites

• A Copy Activity must exist in the pipeline for the targeted object.
• The target Dataset must be defined (either generated by BimlFlex or added via AdfDataset).
• The table parameter provides access to the object metadata.

Implementation Steps

1. Create a new extension point targeting the source object.
2. Add the BimlScript directive with the custom sink definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfCopySink" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<AzureSqlDatabaseSink DatasetName="SQLDB_BFX_LND"
    WriteBehavior="Upsert" UpsertKeyColumns="CustomerID">
  <Parameters>
    <Parameter Name="TargetSchemaName">@pipeline().parameters.TargetSchemaName</Parameter>
    <Parameter Name="TargetTableName">@pipeline().parameters.TargetTableName</Parameter>
  </Parameters>
</AzureSqlDatabaseSink>

3. Build and verify the Copy Activity's sink definition reflects your custom write behavior.

Example

Before (default insert sink):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "sink": {
      "type": "AzureSqlSink",
      "writeBehavior": "insert"
    }
  }
}

After (upsert sink with key column):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "sink": {
      "type": "AzureSqlSink",
      "writeBehavior": "upsert",
      "upsertSettings": { "useTempDB": true, "keys": ["CustomerID"] }
    }
  }
}

Common Mistakes

1. Mistake: Using OverrideAdfCopySink when only the pre-copy script needs changing. Symptom: The entire sink definition is replaced, potentially losing other auto-generated properties. Fix: Use AdfPreCopyScript for pre-copy script changes only.
2. Mistake: Changing the sink Dataset without updating the Dataset parameters. Symptom: The Copy Activity fails with missing parameter errors at runtime. Fix: Ensure the Dataset parameters match what the sink passes.
3. Mistake: Using Upsert write behavior without specifying key columns. Symptom: The Copy Activity fails or produces duplicate records. Fix: Always specify UpsertKeyColumns when using upsert write behavior.
4. Mistake: Not matching the sink type to the connection type. Symptom: ADF deployment fails with a type mismatch error. Fix: Use the correct sink type (e.g., AzureSqlDatabaseSink for Azure SQL, SqlServerSink for on-premises SQL).

Related Extension Points

• OverrideAdfCopySource -- overrides the source definition of the Copy Activity.
• AdfPreCopyScript -- sets only the pre-copy script on the sink (less invasive than overriding the whole sink).
• OverrideAdfDataset -- overrides the Dataset definition that the sink references.
• OverrideAdfStagingSettings -- overrides the staging settings for PolyBase/COPY staged copies.

Override Copy Source

Configure a Copy Activity source in Data Factory. This overrides the entire source definition for an Data Factory Copy Activity.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains the Object (e.g. source table) for which the source properties for the Copy Activity will be overridden.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfCopySource" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>



<!-- This Extension Point overrides the entire 'Source' definition for the targeted Copy Activity in an ADF Pipeline. -->
<!-- Any other Source properties that may be defined using more specific Extension Point for the particular Copy Activity will be omitted. -->

<!-- The below example replaces the existing source properties for the Copy Activity with an Azure SQL Server connection containing a customer query. -->

<AzureSqlDatabaseSource DatasetName="SQLDB_AWLT_SRC">
	<Query>
		SELECT TOP 100 * FROM [SalesLT].[Address] 
	</Query>
	<Parameters>
		<Parameter Name="TargetSchemaName">@pipeline().parameters.TargetSchemaName</Parameter>
		<Parameter Name="TargetTableName">@pipeline().parameters.TargetTableName</Parameter>
	</Parameters>
</AzureSqlDatabaseSource>

What It Does

Completely replaces the source definition of a Copy Activity in an ADF pipeline. The source definition controls which Dataset is read from, the query or stored procedure used for extraction, and any source-specific parameters. This extension point gives you full control over the Copy Activity's source side, letting you change the extraction query, switch to a stored procedure source, or redirect to a different Dataset entirely.

Where It Fires

Consumed in Code/DataFactoryHelper.cs. The extension point replaces the entire source definition within the Copy Activity for the targeted object in the generated ARM template. It fires once per Copy Activity for each targeted object.

When to Use It

1. You need to use a custom SQL query instead of the default table extraction for a specific source object (e.g., extracting only active customers from SalesLT.Customer with a WHERE clause).
2. You want to call a stored procedure as the source instead of a direct table read.
3. You need to change the source Dataset or add Dataset parameters not generated by default.
4. You want to add query timeout or isolation level settings to the source. Use this instead of AdfPreProcess when you need to change the extraction query itself rather than adding a pre-processing step.
5. Extract Dynamics CRM entities using FetchXML queries. Override the copy source to use a DynamicsCrmSource with a FetchXML query instead of a table reference. This allows you to extract specific entities (e.g., Contacts, Accounts) with attribute filtering that the standard table-based extraction doesn't support. Use multi-target syntax to apply the same FetchXML pattern across multiple objects (e.g., target="AWLT_SRC.SalesLT.Address;AWLT_SRC.SalesLT.Customer").

Prerequisites

• A Copy Activity must exist in the pipeline for the targeted source object.
• The source Dataset must be defined (either generated by BimlFlex or added via AdfDataset).
• The table parameter provides access to the source object metadata.

Implementation Steps

1. Create a new extension point targeting the source object.
2. Add the BimlScript directive with the custom source definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfCopySource" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<AzureSqlDatabaseSource DatasetName="SQLDB_AWLT_SRC">
  <Query>
    SELECT * FROM [SalesLT].[Customer] WHERE IsActive = 1
  </Query>
  <Parameters>
    <Parameter Name="TargetSchemaName">SalesLT</Parameter>
    <Parameter Name="TargetTableName">Customer</Parameter>
  </Parameters>
</AzureSqlDatabaseSource>

3. Build and verify the Copy Activity's source definition reflects your custom query.

Example

Before (default table extraction):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "source": {
      "type": "AzureSqlSource",
      "sqlReaderQuery": "SELECT * FROM [SalesLT].[Customer]"
    }
  }
}

After (filtered query with WHERE clause):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "source": {
      "type": "AzureSqlSource",
      "sqlReaderQuery": "SELECT * FROM [SalesLT].[Customer] WHERE IsActive = 1"
    }
  }
}

Common Mistakes

1. Mistake: Using OverrideAdfCopySource when a parameter-based filter would suffice. Symptom: The extension point hardcodes a filter that should be dynamic. Fix: Use BimlFlex parameters for dynamic filtering and reserve this extension point for structural changes.
2. Mistake: Changing the source Dataset without updating the Dataset parameters. Symptom: The Copy Activity fails with missing parameter errors. Fix: Ensure Dataset parameters match what the Copy Activity passes.
3. Mistake: Specifying a source type that does not match the connection type. Symptom: The Copy Activity fails with a type mismatch error. Fix: Use the correct source type (e.g., AzureSqlDatabaseSource for Azure SQL, SqlServerSource for on-premises SQL Server).
4. Mistake: Not testing the custom query independently. Symptom: The Copy Activity fails with a SQL error. Fix: Test the extraction query directly against the source database before adding it to the extension point.

Related Extension Points

• OverrideAdfCopySink -- overrides the sink (destination) definition of the Copy Activity.
• OverrideAdfDataset -- overrides the Dataset definition referenced by the source.
• AdfPreCopyScript -- sets the pre-copy script on the sink side.
• AdfGetParameter -- retrieves parameters that can be used in the source query.

Override Dataset

Configure a Dataset in Data Factory. This overrides the entire definition for a Dataset.

Parameters

Name	Type	Description
connection	BimlFlexModelWrapper.ConnectionsWrapper	Contains the connection for the Linked Service.
dataset	dynamic	Contains the connection for the Linked Service.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfDataset" #>
<#@ property name="connection" type="BimlFlexModelWrapper.ConnectionsWrapper" #>
<#@ property name="dataset" type="dynamic" #>



<!-- The below example replaces the targeted Dataset (via the Connection Name in BimlFlex) with the code below. -->

<#
var datasetName = dataset.Name;
var datasetType = dataset.Type;
var datasetIntegrationStage = dataset.IntegrationStage;
var datasetTable = dataset.Table;
var datasetLinkedServiceName = dataset.LinkedServiceName;
var datasetLinkedServiceType = dataset.LinkedServiceType;
var datasetDatasetPrefix = dataset.DatasetPrefix;
var datasetProject = dataset.Project;
var datasetAdfFolder = dataset.AdfFolder;
var datasetLogicalDisplayFolder = dataset.LogicalDisplayFolder;
var datasetSystemType = dataset.SystemType;
var datasetConnectionType = dataset.ConnectionType;
var datasetDestContainer = dataset.DestContainer;
#>
<SqlServer Name="<#=datasetDatasetPrefix #>_<#=datasetLinkedServiceName#>" Schema="@dataset().TargetSchemaName" Table="@dataset().TargetTableName" LinkedServiceName="<#=datasetLinkedServiceName#>" AdfFolder="<#=datasetAdfFolder#>" LogicalDisplayFolder="<#=datasetLogicalDisplayFolder#>">
	<Parameters>
		<Parameter Name="TargetSchemaName" DataType="String">TargetSchemaNamePlaceholder</Parameter>
		<Parameter Name="TargetTableName" DataType="String">TargetTableNamePlaceholder</Parameter>
	</Parameters>
</SqlServer>

What It Does

Completely replaces the Dataset definition that BimlFlex generates for a specific connection. Datasets in ADF define the structure of data -- which Linked Service to use, the table or file path, and any parameterization. This extension point gives you full control over the Dataset definition, including the type, schema, table name, parameters, and folder organization. Use it when the auto-generated Dataset does not match your specific requirements.

Where It Fires

Consumed in Code/DatasetHelper.cs. The extension point replaces the entire Dataset JSON resource for the targeted connection in the generated ARM template. It is evaluated once per connection that produces a dataset during output generation.

When to Use It

1. You need to change the Dataset type for a connection (e.g., from AzureSqlTable to SqlServerTable for a hybrid deployment using Self-Hosted IR).
2. You want to add custom parameters to the Dataset that allow dynamic table name resolution at runtime.
3. The auto-generated Dataset uses the wrong schema or table name structure for your database.
4. You need to use a parameterized file path for a Blob Storage or ADLS Gen2 Dataset. Use this instead of AdfDataset when you need to modify an existing auto-generated Dataset rather than adding a new one.

Prerequisites

• A connection must be defined in BimlFlex metadata that generates a Dataset.
• The connection and dataset parameters provide access to connection and dataset properties.
• The Linked Service referenced by the Dataset must exist.

Implementation Steps

1. Create a new extension point targeting the specific connection.
2. Add the BimlScript directive with the complete Dataset definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfDataset" #>
<#@ property name="connection" type="BimlFlexModelWrapper.ConnectionsWrapper" #>
<#@ property name="dataset" type="dynamic" #>

<#
var datasetName = dataset.Name;
var datasetLinkedServiceName = dataset.LinkedServiceName;
var datasetAdfFolder = dataset.AdfFolder;
#>
<SqlServer Name="<#=datasetName#>"
    Schema="@dataset().TargetSchemaName"
    Table="@dataset().TargetTableName"
    LinkedServiceName="<#=datasetLinkedServiceName#>"
    AdfFolder="<#=datasetAdfFolder#>">
  <Parameters>
    <Parameter Name="TargetSchemaName" DataType="String">dbo</Parameter>
    <Parameter Name="TargetTableName" DataType="String">placeholder</Parameter>
  </Parameters>
</SqlServer>

3. Build and verify the Dataset definition in the generated ARM template uses your custom definition.

Example

Before (auto-generated AzureSqlTable Dataset):

{ "name": "SQLDB_BFX_LND",
  "type": "Microsoft.DataFactory/factories/datasets",
  "properties": {
    "type": "AzureSqlTable",
    "linkedServiceName": { "referenceName": "BFX_LND" },
    "typeProperties": { "schema": "dbo", "table": "SalesLT_Customer" }
  }
}

After (parameterized SqlServer Dataset):

{ "name": "SQLDB_BFX_LND",
  "type": "Microsoft.DataFactory/factories/datasets",
  "properties": {
    "type": "SqlServerTable",
    "linkedServiceName": { "referenceName": "BFX_LND" },
    "typeProperties": {
      "schema": { "type": "Expression", "value": "@dataset().TargetSchemaName" },
      "table": { "type": "Expression", "value": "@dataset().TargetTableName" }
    },
    "parameters": {
      "TargetSchemaName": { "type": "string", "defaultValue": "dbo" },
      "TargetTableName": { "type": "string", "defaultValue": "placeholder" }
    }
  }
}

Common Mistakes

1. Mistake: Using OverrideAdfDataset when you want to add a new Dataset. Symptom: The override replaces an existing connection's Dataset instead of adding a new one. Fix: Use AdfDataset to add new Datasets.
2. Mistake: Changing the Dataset Name to a different value. Symptom: Copy Activities still reference the original Dataset name and fail with "not found". Fix: Keep the Name matching the auto-generated name using dataset.Name or dataset.DatasetPrefix.
3. Mistake: Not including required Dataset parameters that Copy Activities depend on. Symptom: Copy Activities fail with "missing parameter" errors at runtime. Fix: Include all parameters that the Copy Activity source or sink references.
4. Mistake: Using a Dataset type that does not match the Linked Service type. Symptom: ADF deployment fails with a type mismatch validation error. Fix: Ensure the Dataset type (e.g., SqlServer) matches the Linked Service type (e.g., SqlServer).

Related Extension Points

• AdfDataset -- adds new Datasets to the factory without overriding existing ones.
• OverrideAdfLinkedService -- overrides the Linked Service that the Dataset references.
• OverrideAdfCopySource / OverrideAdfCopySink -- overrides Copy Activity source/sink that reference the Dataset.
• AdfLinkedService -- adds new Linked Services that custom Datasets can reference.

Override Execute Pipeline

Configure the Override Execute Pipeline is Sub Batch for the Copy or Load pipelines.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the object to which the activity will be added.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the activity will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the batch.
OutputPathName	String	You must add CustomOutput.OutputPathName with the last task to connect it with the next Data Flow task.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfExecutePipeline" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>


<# 	CustomOutput.ObjectInherit = false; #>

<#	var sourceTableObject = new ObjectViewModel(sourceTable, sourceTable.Connection.RelatedItem.IntegrationStage, "SRC");
#>

What It Does

Overrides the ExecutePipeline activity that the batch pipeline uses to call individual object pipelines (sub-pipelines). By default, BimlFlex generates an ExecutePipeline activity for each object in a batch, passing standard parameters. This extension point lets you customize how the batch pipeline invokes each object pipeline, including which parameters are passed, whether to wait for completion, and how dependencies are structured.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point replaces the ExecutePipeline activity for the targeted object within the batch pipeline. It fires once per object within the batch during generation.

When to Use It

1. You need to pass additional custom parameters from the batch pipeline to individual object pipelines (e.g., a FilterDate or ProcessingMode parameter).
2. You want to change an object pipeline from WaitOnCompletion="true" to WaitOnCompletion="false" for parallel execution of specific objects.
3. You need to add conditional execution logic (e.g., only call the object pipeline if an IfCondition evaluates to true).
4. You want to add retry logic or error handling around the ExecutePipeline call for a specific object. Use this instead of AdfBatchPreProcess/AdfBatchPostProcess when you need to modify the pipeline invocation itself rather than adding activities around it.

Prerequisites

• A batch must be defined in BimlFlex metadata that contains the targeted object.
• The sourceTable, targetTable, and dependency parameters provide access to object metadata and the preceding activity.
• Any custom parameters passed must be defined on the object pipeline via AdfPipelineParameter.

Implementation Steps

1. Create a new extension point targeting the specific object or batch.
2. Add the BimlScript directive with the custom ExecutePipeline definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfExecutePipeline" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = false; #>
<# var srcObj = new ObjectViewModel(sourceTable, sourceTable.Connection.RelatedItem.IntegrationStage, "SRC"); #>
<# var outputPathName = "EP_" + srcObj.SourceSsisSafeName; #>
<ExecutePipeline Name="<#=outputPathName#>"
    PipelineName="<#=srcObj.SourceSsisSafeName#>"
    WaitOnCompletion="true">
  <Dependencies>
    <Dependency DependsOnActivityName="<#=dependency#>" Condition="Succeeded" />
  </Dependencies>
  <Parameters>
    <Parameter Name="IsInitialLoad">@pipeline().parameters.IsInitialLoad</Parameter>
    <Parameter Name="FilterDate">@pipeline().parameters.FilterDate</Parameter>
  </Parameters>
</ExecutePipeline>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the ExecutePipeline activity in the batch pipeline passes the additional parameters.

Example

Before (default ExecutePipeline call):

{ "name": "EXT_AWLT_SRC_SalesLT_Customer",
  "type": "ExecutePipeline",
  "typeProperties": {
    "pipeline": { "referenceName": "EXT_AWLT_SRC_SalesLT_Customer" },
    "waitOnCompletion": true,
    "parameters": { "IsInitialLoad": "@pipeline().parameters.IsInitialLoad" }
  }
}

After (additional parameter passed):

{ "name": "EP_EXT_AWLT_SRC_SalesLT_Customer",
  "type": "ExecutePipeline",
  "typeProperties": {
    "pipeline": { "referenceName": "EXT_AWLT_SRC_SalesLT_Customer" },
    "waitOnCompletion": true,
    "parameters": {
      "IsInitialLoad": "@pipeline().parameters.IsInitialLoad",
      "FilterDate": "@pipeline().parameters.FilterDate"
    }
  }
}

Common Mistakes

1. Mistake: Setting ObjectInherit = true when only one object needs the override. Symptom: All objects in the batch get the same custom ExecutePipeline logic. Fix: Set CustomOutput.ObjectInherit = false; and target the specific object.
2. Mistake: Passing a parameter that is not defined on the object pipeline. Symptom: ADF deployment validation fails with "parameter not found". Fix: Ensure matching parameters are defined via AdfPipelineParameter.
3. Mistake: Not setting CustomOutput.OutputPathName. Symptom: Downstream activities in the batch pipeline lose their dependency on this object's execution. Fix: Always set CustomOutput.OutputPathName to the activity name.
4. Mistake: Removing WaitOnCompletion without understanding the impact. Symptom: The batch pipeline does not wait for the object pipeline to finish, causing data dependency issues. Fix: Only set WaitOnCompletion="false" for objects that can safely run in parallel.

Related Extension Points

• AdfPipelineParameter -- defines parameters on the object pipeline that can be passed from the batch.
• AdfBatchParameter -- defines parameters on the batch pipeline that are available for passing to object pipelines.
• AdfBatchPreProcess / AdfBatchPostProcess -- adds activities before or after the ExecutePipeline calls in the batch.
• AdfPipeline -- creates custom pipelines that can include their own ExecutePipeline calls.

Override Linked Service

Configure a Linked Service in Data Factory. This overrides the entire definition for a Linked Service. Target the connection and add the relevant Biml for the Linked Service.

Parameters

Name	Type	Description
connection	BimlFlexModelWrapper.ConnectionsWrapper	Contains the connection for the Linked Service.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfLinkedService" #>
<#@ property name="connection" type="BimlFlexModelWrapper.ConnectionsWrapper" #>



<!-- The below example replaces the entire targeted Linked Service (Connection in BimlFlex) with the code below. -->

<SqlServer Name="<#=connection.Name#>"<# if (!string.IsNullOrWhiteSpace(connection.LS_Username)){#>Username="<#=connection.LS_Username#>"<# } #>
<# if (!string.IsNullOrWhiteSpace(connection.LS_Password)){#>Password="<#=connection.LS_Password#>"<# } #>
<# if (!string.IsNullOrWhiteSpace(connection.LS_ServicePrincipalId)){#>ServicePrincipalId="<#=connection.LS_ServicePrincipalId#>"<# } #>
<# if (!string.IsNullOrWhiteSpace(connection.LS_ServicePrincipalKey)){#>ServicePrincipalKey="<#=connection.LS_ServicePrincipalKey#>"<# } #>
<# if (!string.IsNullOrWhiteSpace(connection.LS_Tenant)){#>Tenant="<#=connection.LS_Tenant#>"<# } #>
<# if (!string.IsNullOrWhiteSpace(connection.LS_ConnectionString)){#>ConnectionString="<#=connection.LS_ConnectionString#>"<# } #>
<# if (!string.IsNullOrWhiteSpace(connection.LS_EncryptedCredential)){#>EncryptedCredential="<#=connection.LS_EncryptedCredential#>"<# } #>>
<# if (!string.IsNullOrWhiteSpace(connection.LS_PasswordKVS_StoreName) && !string.IsNullOrWhiteSpace(connection.LS_PasswordKVS_SecretName) ){#>
	<PasswordKVS <# if (!string.IsNullOrWhiteSpace(connection.LS_PasswordKVS_SecretVersion)){#>SecretVersion="<#=connection.LS_PasswordKVS_SecretVersion#>"<# } #>
		StoreName="<#=connection.LS_PasswordKVS_StoreName#>" Name="<#=connection.LS_PasswordKVS_SecretName#>">
	</PasswordKVS>
	<# } #>
	<# if (!string.IsNullOrWhiteSpace(connection.LS_ServicePrincipalKeyKVS_StoreName) && !string.IsNullOrWhiteSpace(connection.LS_ServicePrincipalKeyKVS_SecretName) ){#>
	<ServicePrincipalKeyKVS
		<# if (!string.IsNullOrWhiteSpace(connection.LS_ServicePrincipalKeyKVS_SecretVersion)){#>SecretVersion="<#=connection.LS_ServicePrincipalKeyKVS_SecretVersion#>"<# } #>
		StoreName="<#=connection.LS_ServicePrincipalKeyKVS_StoreName#>" Name="<#=connection.LS_ServicePrincipalKeyKVS_SecretName#>">
	</ServicePrincipalKeyKVS>
	<# } #>
	<# if (!string.IsNullOrWhiteSpace(connection.LS_ConnectionStringKVS_StoreName) && !string.IsNullOrWhiteSpace(connection.LS_ConnectionStringKVS_SecretName) ){#>
	<ConnectionStringKVS <# if (!string.IsNullOrWhiteSpace(connection.LS_ConnectionStringKVS_SecretVersion)){#>SecretVersion="<#=connection.LS_ConnectionStringKVS_SecretVersion#>"<# } #> StoreName="<#=connection.LS_ConnectionStringKVS_StoreName#>" Name="<#=connection.LS_ConnectionStringKVS_SecretName#>">
	</ConnectionStringKVS><# } #><# if (!string.IsNullOrWhiteSpace(connection.LS_ConnectVia_IntegrationRuntime) && connection.LS_ConnectVia_IntegrationRuntime != "AutoResolveIntegrationRuntime"){#><ConnectVia><IntegrationRuntime IntegrationRuntimeName="<#=connection.LS_ConnectVia_IntegrationRuntime#>" /></ConnectVia><# } #>
</SqlServer>

What It Does

Completely replaces the Linked Service definition that BimlFlex generates for a specific connection. While BimlFlex automatically creates Linked Services based on the connection metadata, this extension point gives you full control over every property of the Linked Service -- authentication method, Key Vault references, connection strings, Integration Runtime, and service principal credentials. This is necessary when the auto-generated Linked Service does not support your specific authentication or networking requirements.

Where It Fires

Consumed in Code/LinkedServiceHelper.cs. The extension point replaces the entire Linked Service JSON resource for the targeted connection in the generated ARM template. It is evaluated once per connection during output generation.

When to Use It

1. You need to use Managed Identity authentication for a SQL Server connection, but BimlFlex generates connection string authentication by default.
2. The auto-generated Linked Service does not include a specific property you need (e.g., AlwaysEncryptedSettings or a custom ConnectVia Integration Runtime).
3. You need to switch a Linked Service from Key Vault-based connection strings to direct connection strings for a development environment.
4. You need to override the Linked Service type entirely (e.g., changing from AzureSqlDatabase to SqlServer for a hybrid deployment). Use this instead of AdfLinkedService when you need to modify an existing auto-generated Linked Service rather than adding a new one.
5. Connect to Dynamics 365 CRM as a source in ADF. Override the default linked service to define a DynamicsCrm connection with Office365 authentication, specifying the CRM service URI (e.g., https://yourorg.crm.dynamics.com/) and credentials. Optionally reference an Azure Key Vault secret for the password instead of hard-coding it.

Prerequisites

• A connection must be defined in BimlFlex metadata (e.g., AWLT_SRC, BFX_LND, BFX_STG).
• The connection parameter provides access to all connection properties from metadata.
• The target extension point must be assigned to the specific connection to override.

Implementation Steps

1. Create a new extension point targeting the specific connection (e.g., AWLT_SRC).
2. Add the BimlScript directive with the complete Linked Service definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfLinkedService" #>
<#@ property name="connection" type="BimlFlexModelWrapper.ConnectionsWrapper" #>

<AzureSqlDatabase Name="<#=connection.Name#>"
    ConnectionString="<#=connection.LS_ConnectionString#>"
    AuthenticationType="ManagedIdentity"
    ServerName="<#=connection.LS_ConnectionString#>">
  <ConnectVia>
    <IntegrationRuntime IntegrationRuntimeName="AutoResolveIntegrationRuntime" />
  </ConnectVia>
</AzureSqlDatabase>

3. Build and verify the Linked Service definition in the generated ARM template uses your custom properties.

Example

Before (auto-generated Linked Service with connection string auth):

{ "name": "AWLT_SRC",
  "type": "Microsoft.DataFactory/factories/linkedservices",
  "properties": {
    "type": "SqlServer",
    "typeProperties": {
      "connectionString": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "AdfKeyVault" } }
    }
  }
}

After (overridden with Managed Identity auth):

{ "name": "AWLT_SRC",
  "type": "Microsoft.DataFactory/factories/linkedservices",
  "properties": {
    "type": "AzureSqlDatabase",
    "typeProperties": {
      "connectionString": "Server=awlt-server.database.windows.net;Database=AWLT;",
      "authenticationType": "ManagedIdentity"
    }
  }
}

Common Mistakes

1. Mistake: Using OverrideAdfLinkedService when you want to add a new Linked Service. Symptom: The override replaces an existing connection's Linked Service instead of adding a new one. Fix: Use AdfLinkedService to add new Linked Services.
2. Mistake: Not including all required properties for the Linked Service type. Symptom: ADF deployment fails with missing property validation errors. Fix: Include all required properties for the specific Linked Service type (refer to ADF ARM template documentation).
3. Mistake: Changing the Name attribute to a different value than the connection name. Symptom: Other pipeline activities still reference the original Linked Service name, causing "not found" errors. Fix: Keep the Name matching <#=connection.Name#>.
4. Mistake: Removing the ConnectVia section when a Self-Hosted Integration Runtime is required. Symptom: The Linked Service attempts to use the Azure-hosted IR and fails to reach on-premises resources. Fix: Include the <ConnectVia> section with the correct IR name.

Related Extension Points

• AdfLinkedService -- adds new Linked Services to the factory without overriding existing ones.
• AdfIntegrationRuntime -- defines custom Integration Runtimes referenced by the Linked Service.
• OverrideAdfDataset -- overrides datasets that reference this Linked Service.
• OverrideAdfCopySource / OverrideAdfCopySink -- overrides Copy Activity source/sink that use this Linked Service.

Override Log Settings

Configure the Enable Logging property in the settings for an Copy Activity. This overrides the definition of the Enable Logging setting, including the targeted Linked Service.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains the Object for which the Copy Activity settings for 'Enable Logging' will be replaced or set.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfLogSettings" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>



<!-- This extension point updates the Enable Logging property, which is part of the settings of a Copy Activity. -->

<LogSettings LinkedServiceName="" LogLevel="Warning" EnableReliableLogging="true" Path="logging" />

What It Does

Overrides the Enable Logging settings on a Copy Activity in an ADF pipeline. ADF Copy Activities can log detailed session information (rows read, rows written, skipped rows, throughput) to a Blob Storage location for troubleshooting and auditing. This extension point lets you control the logging Linked Service, log level, reliable logging option, and storage path, replacing the default BimlFlex-generated log settings.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point is emitted as the logSettings property inside the Copy Activity definition in the generated ARM template. It fires once per Copy Activity for each targeted object.

When to Use It

1. You need to enable verbose logging for specific Copy Activities to troubleshoot data movement issues (e.g., logging skipped rows for SalesLT.Customer).
2. You want to redirect Copy Activity logs to a centralized monitoring storage account separate from the data staging storage.
3. You need to enable reliable logging to ensure log completeness even if the copy fails partway through.
4. You want to reduce the log level to Warning for high-volume tables to minimize storage costs. Use this instead of OverrideAdfCopySink when you only need to change the logging behavior, not the entire sink definition.

Prerequisites

• A Copy Activity must exist in the pipeline.
• The logging Linked Service must be defined (either generated by BimlFlex or added via AdfLinkedService).
• The logging Blob Storage container must exist and be accessible by the ADF Integration Runtime.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the log settings override:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfLogSettings" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<LogSettings LinkedServiceName="AdfLinkedService_Monitoring"
             LogLevel="Info" EnableReliableLogging="true"
             Path="logging/copy-activity" />

3. Build and verify the logSettings property appears in the Copy Activity definition.

Example

Before (default or no logging):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": { "source": {}, "sink": {} }
}

After (verbose logging enabled):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "source": {}, "sink": {},
    "logSettings": {
      "enableCopyActivityLog": true,
      "copyActivityLogSettings": {
        "logLevel": "Info",
        "enableReliableLogging": true
      },
      "logLocationSettings": {
        "linkedServiceName": { "referenceName": "AdfLinkedService_Monitoring" },
        "path": "logging/copy-activity"
      }
    }
  }
}

Common Mistakes

1. Mistake: Specifying a Linked Service that does not exist. Symptom: ADF deployment validation fails. Fix: Ensure the logging Linked Service is defined via AdfLinkedService or BimlFlex metadata.
2. Mistake: Using Info log level on all tables in a high-volume environment. Symptom: The logging storage account fills up quickly and incurs high costs. Fix: Use Warning level for most tables and Info only for specific troubleshooting.
3. Mistake: Not setting EnableReliableLogging for critical tables. Symptom: Logs are incomplete when the Copy Activity fails partway through. Fix: Set EnableReliableLogging="true" for tables where complete audit logs are required.

Related Extension Points

• OverrideAdfStagingSettings -- overrides the staging settings for the Copy Activity.
• OverrideAdfCopySink -- overrides the entire sink definition.
• AdfLinkedService -- defines the Linked Service used for log storage.
• AdfPostCopy -- adds post-copy logic where you can act on copy results.

Override Staging Settings

Configure the Enable Staging property in the settings for an Copy Activity. This overrides the definition of the Enable Staging setting, including the targeted Linked Service.

Parameters

Name	Type	Description
table	BimlFlexModelWrapper.ObjectsWrapper	Contains the Object for which the Copy Activity settings for 'Enable Staging' will be replaced or set.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfStagingSettings" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>



<!-- This extension point updates the Enable Staging property, which is part of the settings of a Copy Activity. -->

<StagingSettings LinkedServiceName="" Path="staging" />

What It Does

Overrides the Enable Staging (PolyBase/COPY staging) settings on a Copy Activity in an ADF pipeline. When copying data into Azure Synapse Analytics or Azure SQL Database via PolyBase or the COPY command, ADF requires a staging storage location. This extension point lets you control the staging Linked Service and the Blob Storage path used for intermediate staging, replacing the default BimlFlex-generated staging settings.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point is emitted as the enableStaging and stagingSettings properties inside the Copy Activity definition in the generated ARM template. It fires once per Copy Activity for each targeted object.

When to Use It

1. You need to use a different Azure Blob Storage account for staging than the one BimlFlex configures by default.
2. You want to change the staging path to include date-partitioned folders for easier troubleshooting (e.g., staging/2025/03/26/SalesLT_Customer).
3. You need to enable staging for a Copy Activity that BimlFlex does not stage by default (e.g., Azure SQL Database to Synapse via PolyBase).
4. You want to use a different Linked Service for staging that points to ADLS Gen2 instead of Blob Storage. Use this instead of OverrideAdfCopySink when you only need to change the staging settings, not the entire sink definition.

Prerequisites

• A Copy Activity must exist in the pipeline targeting a destination that supports staged copy (e.g., Azure Synapse Analytics).
• The staging Linked Service must be defined (either generated by BimlFlex or added via AdfLinkedService).
• The staging Blob Storage container and path must exist and be accessible by the ADF Integration Runtime.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the staging settings override:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="OverrideAdfStagingSettings" #>
<#@ property name="table" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<StagingSettings LinkedServiceName="AdfLinkedService_StagingBlob" Path="staging/synapse" />

3. Build and verify the enableStaging and stagingSettings properties appear in the Copy Activity definition.

Example

Before (default staging settings):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "enableStaging": true,
    "stagingSettings": {
      "linkedServiceName": { "referenceName": "AdfLinkedService_Staging" },
      "path": "staging"
    }
  }
}

After (custom staging storage and path):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "typeProperties": {
    "enableStaging": true,
    "stagingSettings": {
      "linkedServiceName": { "referenceName": "AdfLinkedService_StagingBlob" },
      "path": "staging/synapse"
    }
  }
}

Common Mistakes

1. Mistake: Specifying a Linked Service that does not exist in the Data Factory. Symptom: ADF deployment fails with a "linked service not found" validation error. Fix: Ensure the Linked Service is defined via AdfLinkedService or BimlFlex metadata.
2. Mistake: Using ADLS Gen2 hierarchical namespace paths without the correct Linked Service type. Symptom: Staging files cannot be written, and the Copy Activity fails. Fix: Use an Azure Blob Storage or ADLS Gen2 Linked Service that matches the storage type.
3. Mistake: Forgetting to grant the ADF managed identity access to the staging container. Symptom: The Copy Activity fails with an authorization error during staging. Fix: Assign the Storage Blob Data Contributor role to the ADF managed identity on the staging container.

Related Extension Points

• OverrideAdfLogSettings -- overrides the logging settings for the Copy Activity.
• OverrideAdfCopySink -- overrides the entire sink definition, including staging.
• AdfLinkedService -- defines the Linked Service used as the staging storage.
• AdfDataset -- defines datasets referenced by the Copy Activity.

Pipeline

Adds a bespoke Pipeline to an Data Factory.

Parameters

Name	Type	Description
instance	BimlFlexModelWrapper	The metadata instance of which the Data Factory is part. There is no 'Data Factory' object in the metadata model that can directly be accessed, and the instance object allows for full access to all metadata to define the specific behaviour of the Pipeline that is added using this Extension Point.
dataFactoryName	String	The Data Factory name the Extension Point is applied to. E.g. where the Pipeline will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPipeline" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>



<!-- This extension point allows the addition of a bespoke Pipeline to the specified Data Factory. -->
<!-- The below example shows a Bimlscript Pipeline code snippet, which will be added to the Pipelines already generated by BimlFlex. -->
<!-- This example creates a 'master' batch by starting existing Batch pipelines in sequence. -->

<Pipeline Name="Main AWLT Orchestration" AdfFolder="01 Main AWLT Orchestration" LogicalDisplayFolder="01 Main AWLT Orchestration">
	<Parameters>
		<Parameter Name="IsInitialLoad" DataType="Bool">false</Parameter>
		<Parameter Name="OtherPipelineParameter" DataType="String">placeholder</Parameter>
	</Parameters>
	<Variables>
		<Variable Name="PipelineVariable" DataType="String">placeholder</Variable>
	</Variables>
	<Activities>
		<!-- Run Extract to Staging, PSA and Data Vault Pipeline -->
		<ExecutePipeline Name="01_EXT_AWLT_SRC_Batch" PipelineName="01_EXT_AWLT_SRC_Batch" WaitOnCompletion="true">
			<Parameters>
				<Parameter Name="IsInitialLoad">@pipeline().parameters.IsInitialLoad</Parameter>
			</Parameters>
		</ExecutePipeline>
		<!-- Run Delete Detection Pipeline -->
		<ExecutePipeline Name="01_EXT_AWLT_SRC_DEL_Batch" PipelineName="01_EXT_AWLT_SRC_DEL_Batch" WaitOnCompletion="true">
			<Dependencies>
				<Dependency DependsOnActivityName="01_EXT_AWLT_SRC_Batch" Condition="Succeeded"></Dependency>
			</Dependencies>
		</ExecutePipeline>
		<!-- Run Data Vault PIT and BRG Pipelines -->
		<ExecutePipeline Name="01_LOAD_BFX_DV_BRG_Batch" PipelineName="01_LOAD_BFX_DV_BRG_Batch" WaitOnCompletion="true">
			<Dependencies>
				<Dependency DependsOnActivityName="01_EXT_AWLT_SRC_DEL_Batch" Condition="Succeeded"></Dependency>
			</Dependencies>
		</ExecutePipeline>
		<ExecutePipeline Name="01_LOAD_BFX_DV_PIT_Batch" PipelineName="01_LOAD_BFX_DV_PIT_Batch" WaitOnCompletion="true">
			<Dependencies>
				<Dependency DependsOnActivityName="01_LOAD_BFX_DV_BRG_Batch" Condition="Succeeded"></Dependency>
			</Dependencies>
		</ExecutePipeline>
		<!-- Run Data Mart Pipeline -->
		<ExecutePipeline Name="01_LOAD_BFX_DM_Batch" PipelineName="01_LOAD_BFX_DM_Batch" WaitOnCompletion="true">
			<Dependencies>
				<Dependency DependsOnActivityName="01_LOAD_BFX_DV_PIT_Batch" Condition="Succeeded"></Dependency>
			</Dependencies>
		</ExecutePipeline>
	</Activities>
</Pipeline>

What It Does

Adds entirely custom pipelines to the Azure Data Factory alongside the pipelines BimlFlex generates automatically. This extension point creates complete pipeline definitions with their own activities, parameters, variables, and dependency chains. The most common use is building a master orchestration pipeline that calls generated batch pipelines in sequence, but it can also be used for utility pipelines, monitoring pipelines, or any custom workflow.

Where It Fires

Consumed in Code/DataFactoryHelper.cs and Code/DataFactoryProvider.cs. The extension point emits one or more Pipeline JSON resources at the factory level in the generated ARM template. It is evaluated once per data factory during output generation.

When to Use It

1. You need a master orchestration pipeline that runs 01_EXT_AWLT_SRC_Batch, then 02_LOAD_BFX_DV_Batch, then 03_LOAD_BFX_DM_Batch in sequence with dependency handling.
2. You want a utility pipeline that performs maintenance tasks (e.g., database index rebuilds, statistics updates) that is not part of the standard BimlFlex data flow.
3. You need a monitoring pipeline that checks the status of other pipelines and sends notifications.
4. You want to create a parameterized reusable pipeline that multiple batch pipelines can call. Use this instead of AdfBatchPreProcess or AdfBatchPostProcess when you need a standalone pipeline rather than injecting activities into an existing one.

Prerequisites

• The Data Factory connection (adf-bimlflex-prod) must be configured in BimlFlex metadata.
• The instance and dataFactoryName parameters provide access to metadata and the factory name.
• Any pipelines referenced via ExecutePipeline activities must exist (either generated by BimlFlex or defined in this extension point).
• Any Linked Services referenced by Script or Copy activities must be defined.

Implementation Steps

1. Create a new extension point targeting the data factory.
2. Add the BimlScript directive with the pipeline definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPipeline" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>

<Pipeline Name="Master_AWLT_Orchestration" AdfFolder="Orchestration">
  <Parameters>
    <Parameter Name="IsInitialLoad" DataType="Bool">false</Parameter>
  </Parameters>
  <Activities>
    <ExecutePipeline Name="Run_Extract" PipelineName="01_EXT_AWLT_SRC_Batch"
                     WaitOnCompletion="true">
      <Parameters>
        <Parameter Name="IsInitialLoad">@pipeline().parameters.IsInitialLoad</Parameter>
      </Parameters>
    </ExecutePipeline>
    <ExecutePipeline Name="Run_DV_Load" PipelineName="02_LOAD_BFX_DV_Batch"
                     WaitOnCompletion="true">
      <Dependencies>
        <Dependency DependsOnActivityName="Run_Extract" Condition="Succeeded" />
      </Dependencies>
    </ExecutePipeline>
  </Activities>
</Pipeline>

3. Build and verify the pipeline appears in the ARM template output.
4. Deploy to adf-bimlflex-prod and test the pipeline in the ADF portal.

Example

Before (only BimlFlex-generated batch pipelines exist):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/pipelines", "name": "01_EXT_AWLT_SRC_Batch" },
    { "type": "Microsoft.DataFactory/factories/pipelines", "name": "02_LOAD_BFX_DV_Batch" }
  ]
}

After (master orchestration pipeline added):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/pipelines", "name": "01_EXT_AWLT_SRC_Batch" },
    { "type": "Microsoft.DataFactory/factories/pipelines", "name": "02_LOAD_BFX_DV_Batch" },
    { "type": "Microsoft.DataFactory/factories/pipelines", "name": "Master_AWLT_Orchestration",
      "properties": {
        "activities": [
          { "name": "Run_Extract", "type": "ExecutePipeline",
            "typeProperties": { "pipeline": { "referenceName": "01_EXT_AWLT_SRC_Batch" } } },
          { "name": "Run_DV_Load", "type": "ExecutePipeline",
            "dependsOn": [{ "activity": "Run_Extract", "dependencyConditions": ["Succeeded"] }],
            "typeProperties": { "pipeline": { "referenceName": "02_LOAD_BFX_DV_Batch" } } }
        ]
      }
    }
  ]
}

Common Mistakes

1. Mistake: Referencing a pipeline name that does not match the generated name. Symptom: The ExecutePipeline activity fails because the referenced pipeline does not exist. Fix: Use the exact naming convention (e.g., 01_EXT_AWLT_SRC_Batch).
2. Mistake: Not setting WaitOnCompletion="true" on ExecutePipeline activities. Symptom: Child pipelines run in parallel instead of sequentially, causing data dependency issues. Fix: Set WaitOnCompletion="true" for sequential orchestration.
3. Mistake: Creating a circular dependency between custom and generated pipelines. Symptom: ADF deployment fails with a circular reference validation error. Fix: Ensure the custom pipeline only calls generated pipelines, not the other way around.
4. Mistake: Not organizing pipelines into ADF folders. Symptom: The ADF portal becomes cluttered with many pipelines at the root level. Fix: Use the AdfFolder attribute to organize custom pipelines.

Related Extension Points

• AdfTrigger / AdfBatchTrigger -- attach triggers to the custom pipeline for automated execution.
• AdfBatchPreProcess / AdfBatchPostProcess -- inject activities into existing batch pipelines instead of creating new ones.
• AdfLinkedService -- define Linked Services referenced by activities in the custom pipeline.
• AdfDataset -- define datasets referenced by Copy activities in the custom pipeline.

Trigger

Configure a Trigger in Data Factory.

Parameters

Name	Type	Description
instance	BimlFlexModelWrapper	The metadata instance of which the Data Factory is part. There is no 'Data Factory' object in the metadata model that can directly be accessed, and the instance object allows for full access to all metadata to define the specific behaviour of the Pipeline that is added using this Extension Point.
dataFactoryName	String	The Data Factory name the Extension Point is applied to. E.g. where the Pipeline will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfTrigger" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>


<!-- The below example adds a Schedule Trigger and connects it to a Data Factory Pipeline. -->

<Schedule Name="ScheduleTriggerName" Frequency="Hour" Interval="1" Start="2021-01-01" End="2025-12-31">
	<Pipelines>
		<Pipeline PipelineName="MyPipelineName">
			<Parameters>
				<Parameter Name="IsInitialLoad">false</Parameter>
			</Parameters>
		</Pipeline>
	</Pipelines>
</Schedule>

What It Does

Adds scheduling triggers to the Azure Data Factory at the factory level. Triggers define when and how pipelines are executed automatically, supporting schedule triggers (cron-like recurrence), tumbling window triggers (for reprocessable time windows), and event-based triggers (e.g., blob creation events). This extension point creates triggers as top-level factory resources that can be linked to any pipeline.

Where It Fires

Consumed in Code/TriggerHelper.cs, with example templates in ExtensionPoints/AdfTrigger1.biml. The extension point emits Trigger JSON resources at the factory level in the generated ARM template. It is evaluated once per data factory during output generation.

When to Use It

1. You need a schedule trigger that starts the 01_EXT_AWLT_SRC_Batch pipeline every hour on weekdays.
2. You want a tumbling window trigger to enable reprocessing of missed time intervals for the extraction batch.
3. You need an event-based trigger that fires when a file arrives in a specific Azure Blob Storage container.
4. You want to connect a single trigger to multiple pipelines (e.g., trigger both 01_EXT_AWLT_SRC_Batch and 02_LOAD_BFX_DV_Batch from the same schedule). Use this instead of AdfBatchTrigger when the trigger should be a factory-level resource not scoped to a single batch.
5. Schedule an hourly batch execution trigger. Define a Schedule trigger with Frequency="Hour" and Interval="1" that runs the batch orchestration pipeline (e.g., 0_<ProjectName>_Batch) with IsInitialLoad=false. This automates recurring incremental loads without manual pipeline execution.

Prerequisites

• The Data Factory connection (adf-bimlflex-prod) must be configured in BimlFlex metadata.
• The instance and dataFactoryName parameters provide access to metadata and the factory name.
• The pipelines referenced in the trigger must exist (either generated by BimlFlex or added via AdfPipeline).
• If the trigger passes parameters, those parameters must be defined on the target pipeline.

Implementation Steps

1. Create a new extension point targeting the data factory.
2. Add the BimlScript directive with the trigger definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfTrigger" #>
<#@ property name="instance" type="BimlFlexModelWrapper" #>
<#@ property name="dataFactoryName" type="String" #>

<Schedule Name="Hourly_Extraction" Frequency="Hour" Interval="1"
         Start="2025-01-01" End="2027-12-31">
  <Pipelines>
    <Pipeline PipelineName="01_EXT_AWLT_SRC_Batch">
      <Parameters>
        <Parameter Name="IsInitialLoad">false</Parameter>
      </Parameters>
    </Pipeline>
  </Pipelines>
</Schedule>

3. Build and verify the trigger appears in the ARM template output under Microsoft.DataFactory/factories/triggers.
4. Deploy to adf-bimlflex-prod and start the trigger in the ADF portal.

Example

Before (no triggers -- all pipelines must be started manually):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/pipelines", "name": "01_EXT_AWLT_SRC_Batch" }
  ]
}

After (schedule trigger added):

{ "resources": [
    { "type": "Microsoft.DataFactory/factories/pipelines", "name": "01_EXT_AWLT_SRC_Batch" },
    { "type": "Microsoft.DataFactory/factories/triggers", "name": "Hourly_Extraction",
      "properties": {
        "type": "ScheduleTrigger",
        "typeProperties": { "recurrence": { "frequency": "Hour", "interval": 1 } },
        "pipelines": [{ "pipelineReference": { "referenceName": "01_EXT_AWLT_SRC_Batch" } }]
      }
    }
  ]
}

Common Mistakes

1. Mistake: Using AdfTrigger when the trigger should only apply to one batch. Symptom: The trigger is a factory-level resource rather than scoped to a specific batch pipeline. Fix: Use AdfBatchTrigger for batch-scoped triggers.
2. Mistake: Setting PipelineName to a value that does not match any generated pipeline name. Symptom: The trigger deploys but references a non-existent pipeline. Fix: Verify pipeline names match the BimlFlex naming convention (e.g., 01_EXT_AWLT_SRC_Batch).
3. Mistake: Overlapping schedule triggers that fire at the same time for dependent pipelines. Symptom: Pipelines run concurrently and cause data conflicts or deadlocks. Fix: Stagger trigger schedules or use a parent orchestration pipeline with sequential execution.
4. Mistake: Forgetting to start the trigger after deployment. Symptom: The trigger exists in ADF but never fires. Fix: Start the trigger via the ADF portal or ARM deployment scripts.

Related Extension Points

• AdfBatchTrigger -- batch-scoped triggers tied to a specific batch pipeline.
• AdfPipeline -- adds custom pipelines that can be referenced by triggers.
• AdfBatchParameter / AdfPipelineParameter -- define parameters that triggers can pass to pipelines.
• AdfGlobalParameter -- factory-level parameters accessible from all pipelines and triggers.

Pipeline Parameter

Configure Data Factory Pipeline Parameters.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object for which the pipeline is created.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the pipeline will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPipelineParameter" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>


<Parameter Name="MyAdfParameter" DataType="Bool">false</Parameter>

What It Does

Adds custom parameters to the individual object-level ADF pipeline (the Execute Pipeline for a specific source object). Pipeline parameters are read-only inputs that are set when the pipeline is triggered or called by a parent pipeline. They allow you to pass configuration values such as flags, date ranges, or environment-specific settings into the object pipeline at execution time.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point emits parameter definitions into the parameters section of the individual object pipeline ARM template.

When to Use It

1. You need to pass an IsInitialLoad flag from the batch pipeline to individual object pipelines to control full-load vs. incremental behavior.
2. You want to pass environment-specific values (e.g., a storage account name or container path) that differ between development and production.
3. You need to pass a date range filter from the parent batch pipeline into object pipelines for time-bounded extraction.
4. You want to pass a custom processing mode flag that your extension point activities can reference. Use this instead of AdfBatchParameter when the parameter is needed at the individual object pipeline level.

Prerequisites

• A source-to-target mapping must exist in BimlFlex metadata for the object.
• The sourceTable and targetTable parameters provide access to object metadata.
• Parameter names must be unique within the pipeline.
• The parent batch pipeline or trigger must pass values for any parameters that do not have defaults.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the parameter definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPipelineParameter" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<Parameter Name="IsInitialLoad" DataType="Bool">false</Parameter>
<Parameter Name="FilterDate" DataType="String">1900-01-01</Parameter>

3. Build and verify the parameters appear in the pipeline's parameters section in the generated ARM template.
4. Update the batch pipeline or trigger to pass values for these parameters using OverrideAdfExecutePipeline or AdfBatchTrigger.

Example

Before (pipeline has only BimlFlex-generated parameters):

{ "name": "EXT_AWLT_SRC_SalesLT_Customer",
  "properties": { "parameters": {} }
}

After (custom parameters added):

{ "name": "EXT_AWLT_SRC_SalesLT_Customer",
  "properties": {
    "parameters": {
      "IsInitialLoad": { "type": "bool", "defaultValue": false },
      "FilterDate": { "type": "string", "defaultValue": "1900-01-01" }
    }
  }
}

Common Mistakes

1. Mistake: Adding a parameter to the object pipeline without passing it from the batch pipeline. Symptom: The parameter always uses its default value because no caller sets it. Fix: Use OverrideAdfExecutePipeline to pass the parameter from the batch pipeline.
2. Mistake: Using AdfPipelineParameter when the parameter is needed at batch level. Symptom: The parameter only exists on the object pipeline, not the batch orchestration. Fix: Use AdfBatchParameter for batch-level parameters.
3. Mistake: Defining a parameter with a name that conflicts with a BimlFlex-generated parameter. Symptom: The build fails or the BimlFlex-generated value is overwritten. Fix: Prefix custom parameter names (e.g., EP_IsInitialLoad).

Related Extension Points

• AdfBatchParameter -- adds parameters to the batch-level pipeline.
• AdfPipelineVariable -- adds variables (read-write) to the object-level pipeline.
• OverrideAdfExecutePipeline -- customizes how the batch pipeline calls the object pipeline, including which parameter values are passed.
• AdfBatchTrigger / AdfTrigger -- triggers that can pass parameter values when starting a pipeline.

Pipeline Variable

Configure Data Factory Pipeline Variables.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object for which the pipeline is created.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the pipeline will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPipelineVariable" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>


<Variable Name="MyAdfVariable" DataType="String"></Variable>

What It Does

Adds custom variables to the individual object-level ADF pipeline (the Execute Pipeline for a specific source object). Pipeline variables are scoped to the pipeline execution and can be used to hold intermediate values such as row counts, computed file paths, or conditional flags throughout the pipeline's activity chain. Unlike parameters, variables can be set and updated during pipeline execution using SetVariable activities.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point emits variable definitions into the variables section of the individual object pipeline ARM template.

When to Use It

1. You need a pipeline-scoped variable to hold intermediate results (e.g., a row count from a Lookup activity used in a conditional check later in the pipeline).
2. You want to store a computed file path or container name that is referenced by multiple activities within the same pipeline.
3. You need a flag variable to control conditional execution paths within the pipeline (e.g., SkipDVProcess set based on a data quality check).
4. You want to accumulate values across loop iterations within the pipeline. Use this instead of AdfBatchVariable when the variable is needed at the individual object pipeline level rather than the batch pipeline level.

Prerequisites

• A source-to-target mapping must exist in BimlFlex metadata for the object.
• The sourceTable and targetTable parameters provide access to object metadata.
• Variable names must be unique within the pipeline.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the variable definition:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPipelineVariable" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<Variable Name="RowCount" DataType="String">0</Variable>
<Variable Name="SkipProcessing" DataType="Bool">false</Variable>

3. Build and verify the variables appear in the pipeline's variables section in the generated ARM template.
4. Use SetVariable activities in other extension points (e.g., AdfPostCopy, AdfPreProcess) to update the variable values during execution.

Example

Before (pipeline has only BimlFlex-generated variables):

{ "name": "EXT_AWLT_SRC_SalesLT_Customer",
  "properties": { "variables": {} }
}

After (custom variables added):

{ "name": "EXT_AWLT_SRC_SalesLT_Customer",
  "properties": {
    "variables": {
      "RowCount": { "type": "String", "defaultValue": "0" },
      "SkipProcessing": { "type": "Bool", "defaultValue": false }
    }
  }
}

Common Mistakes

1. Mistake: Using AdfPipelineVariable when the variable is needed at batch level. Symptom: The variable is only available in the individual object pipeline, not the batch orchestration pipeline. Fix: Use AdfBatchVariable for batch-level variables.
2. Mistake: Defining a variable with a name that conflicts with a BimlFlex-generated variable. Symptom: The build fails with a duplicate variable name error. Fix: Prefix custom variable names (e.g., EP_RowCount) to avoid collisions.
3. Mistake: Using a DataType that does not match the value being assigned. Symptom: SetVariable activities fail at runtime with a type mismatch error. Fix: Ensure the variable DataType matches the value type (use String for ADF expressions that return strings).

Related Extension Points

• AdfBatchVariable -- adds variables to the batch-level pipeline.
• AdfPipelineParameter -- adds parameters (read-only inputs) to the object-level pipeline.
• AdfPreProcess / AdfPostProcess -- activities where SetVariable can be used to update pipeline variables.
• AdfPostCopy -- another activity location where variables can be set based on copy results.

Post Copy

Configure bespoke logic that will be applied immediately after the Copy Activity that is part of the 'MainActivity' IfCondition container, for the targeted individual process Execute Pipeline.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the Object for which the pipeline is created.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the pipeline will be added.
dependency	String	Contains the name of the previous activity to manage dependencies.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPostCopy" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- This extension point allows the addition of bespoke logic that will be applied immediately after the Copy Activity that is part of the 'MainActivity' IfCondition container, for the targeted individual process Execute Pipeline.-->
<!-- Please note that a Copy Activity is only relevant for source-to-staging type processes that receive and land data from external data sources. -->
<!-- In the Biml output, the location is specified as 'Placeholder for extensionpoint="AdfPostCopy"' -->
<!-- The below example shows 'Wait' step to be added after the main processing in the Pipeline has completed. -->

<#
// The following is an example of how BimlScript can be used for this extension point.
var outputPathName = "Wait For Ten Seconds Post Copy";
#>

<Wait Name="<#=outputPathName#>" WaitTimeInSeconds="10">
	<Dependencies>
		<Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>"></Dependency>
	</Dependencies>
</Wait>

<#
// The output path name is set here, so that BimlFlex can connect this activity to the other generated components.
CustomOutput.OutputPathName = outputPathName;#>

What It Does

Injects custom logic immediately after the Copy Activity within the MainActivity IfCondition of an individual object's Execute Pipeline. This is the first opportunity to act on freshly landed data before the staging or Data Vault processing begins. Common uses include data validation on the landing layer, triggering file archival, or logging copy statistics.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline's MainActivity IfCondition, directly after the Copy Activity and before the staging process (SQL_PROCESS_STG). The dependency parameter contains the name of the Copy Activity.

When to Use It

1. You need to verify that the Copy Activity wrote the expected number of rows before proceeding to staging (e.g., comparing @activity('Copy_SalesLT_Customer').output.rowsCopied against a threshold).
2. You want to log copy statistics (rows copied, duration) to a custom audit table immediately after the copy completes.
3. You need to trigger an external process (e.g., a Logic App or Azure Function) after data lands in BFX_LND but before staging transforms run.
4. You want to run a deduplication query on the landing table before staging picks up the data. Use this instead of AdfPostProcess when the logic must run specifically after the copy step rather than after all processing.

Prerequisites

• A Copy Activity must exist in the pipeline (source-to-staging or source-to-landing process).
• The sourceTable, targetTable, and dependency parameters provide access to object metadata and the preceding activity name.
• CustomOutput.OutputPathName must be set so that downstream activities (staging, DV) can depend on this step.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the post-copy logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPostCopy" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<#
var outputPathName = "Audit_Copy_" + sourceTable.ObjectName;
#>
<Script Name="<#=outputPathName#>" Timeout="0.00:10:00"
        LinkedServiceName="BimlCatalog">
  <Scripts>
    <Script ScriptType="Query">
      <Query>INSERT INTO [audit].[CopyLog] (ObjectName, RowsCopied, ExecutionID)
VALUES ('<#=sourceTable.Name#>', @activity('<#=dependency#>').output.rowsCopied, @activity('LogExecutionStart').output.firstRow.ExecutionID)</Query>
    </Script>
  </Scripts>
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Script>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the new activity appears between the Copy Activity and SQL_PROCESS_STG in the generated pipeline.

Example

Before (staging runs directly after copy):

{ "activities": [
    { "name": "Copy_SalesLT_Customer", "type": "Copy" },
    { "name": "SQL_PROCESS_STG", "dependsOn": [{ "activity": "Copy_SalesLT_Customer" }] }
  ]
}

After (audit step inserted between copy and staging):

{ "activities": [
    { "name": "Copy_SalesLT_Customer", "type": "Copy" },
    { "name": "Audit_Copy_Customer", "type": "Script",
      "dependsOn": [{ "activity": "Copy_SalesLT_Customer", "dependencyConditions": ["Succeeded"] }]
    },
    { "name": "SQL_PROCESS_STG", "dependsOn": [{ "activity": "Audit_Copy_Customer" }] }
  ]
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: The staging process (SQL_PROCESS_STG) depends directly on the Copy Activity, bypassing the post-copy step. Fix: Always set CustomOutput.OutputPathName = outputPathName;.
2. Mistake: Confusing AdfPostCopy with AdfPostProcess. Symptom: The logic runs at the end of all processing instead of immediately after the copy. Fix: Use AdfPostCopy for post-copy logic and AdfPostProcess for end-of-pipeline logic.
3. Mistake: Referencing @activity('Copy_SalesLT_Customer') with a hardcoded name. Symptom: The expression fails when the copy activity has a different generated name. Fix: Use @activity('<#=dependency#>') to dynamically reference the preceding activity.
4. Mistake: Adding a long-running operation in post-copy that delays staging. Symptom: The overall pipeline duration increases significantly. Fix: Keep post-copy logic lightweight or move heavy processing to AdfPostProcess.

Related Extension Points

• AdfPreCopyScript -- sets the pre-copy script on the sink; runs before the copy starts.
• AdfPostProcess -- runs as the last step in the MainActivity IfCondition after all processing.
• AdfProcessStaging -- the staging process that runs after post-copy.
• AdfPreProcess -- runs as the first step in the MainActivity IfCondition, before the copy.

Post Process

Add bespoke logic as the last step in the 'MainActivity' IfCondition before any parameters are updated, for the targeted individual Execute Pipeline.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the object to which the activity will be added.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the activity will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPostProcess" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>



<!-- The ADF Post Process extension points targets an ADF Pipeline as the last part of the 'MainActivity' IfCondition before any parameters are updated.-->
<!-- This extension point can be used to add bespoke logic using BimlScript syntax as the last step in the IfCondition. -->
<!-- The parameters (load windows) are still updated after this extension point, in case there are failures. -->
<!-- In the Biml output, the location is specified as 'Placeholder for extensionpoint="AdfPostProcess"'. -->
<!-- The below example shows 'Wait' step to be added after the main processing in the Pipeline has completed. -->

<# 
// Object inheritance is enabled by default.
// This means the target can be set either as individual object or parent objects (e.g. Batch-level), and this extension will be applied to all child objects.
CustomOutput.ObjectInherit = true;
#>

<#
// The following is an example of how BimlScript can be used for this extension point.
var outputPathName = "Wait For Ten Seconds Post Process";
#>

<Wait Name="<#=outputPathName#>" WaitTimeInSeconds="10">
	<Dependencies>
		<Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>"></Dependency>
	</Dependencies>
</Wait>
<# 

// Wrapping up
CustomOutput.OutputPathName = outputPathName;#>

What It Does

Injects custom logic as the last step in the MainActivity IfCondition of an individual object's Execute Pipeline, before any parameter values are persisted back to the BimlCatalog. This is the ideal location for post-load validation, data quality checks, or cleanup operations that should run after the main processing (copy, staging, Data Vault) but before the load window parameters are updated.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline's MainActivity IfCondition, after the last main processing activity (e.g., SQL_PROCESS_DV) and before the parameter persistence activities (EP_LOG_*). The dependency parameter references the preceding activity.

When to Use It

1. You need to run a data quality validation query after data has been loaded into staging or the Data Vault (e.g., check row counts on SalesLT.Customer in BFX_STG).
2. You want to trigger a downstream notification (e.g., call a Logic App webhook) after each object pipeline completes successfully.
3. You need to update a custom audit table with load statistics before parameters are persisted.
4. You want to run a cleanup step that removes temporary data created during processing. Use this instead of AdfBatchPostProcess when you need per-object post-processing rather than batch-level post-processing.

Prerequisites

• A source-to-target mapping must exist in BimlFlex metadata for the object.
• The sourceTable and targetTable parameters provide access to object metadata.
• The dependency parameter provides the name of the preceding activity.
• CustomOutput.OutputPathName must be set so that parameter persistence activities can depend on this step.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the post-process logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPostProcess" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<# CustomOutput.ObjectInherit = true; #>
<# var outputPathName = "DQ_Check_" + targetTable.ObjectName; #>
<Script Name="<#=outputPathName#>" Timeout="0.00:10:00"
        LinkedServiceName="<#=targetTable.Connection.RelatedItem.Name#>">
  <Scripts>
    <Script ScriptType="Query">
      <Query>SELECT COUNT(*) AS RowCount FROM <#=targetTable.Name#></Query>
    </Script>
  </Scripts>
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Script>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the new activity appears after the main processing activities in the MainActivity IfCondition.

Example

Before (parameter persistence runs directly after DV processing):

{ "activities": [
    { "name": "SQL_PROCESS_DV", "type": "Script" },
    { "name": "EP_LOG_LastLoadDate", "dependsOn": [{ "activity": "SQL_PROCESS_DV" }] }
  ]
}

After (DQ check inserted between processing and parameter persistence):

{ "activities": [
    { "name": "SQL_PROCESS_DV", "type": "Script" },
    { "name": "DQ_Check_Customer", "type": "Script",
      "dependsOn": [{ "activity": "SQL_PROCESS_DV", "dependencyConditions": ["Succeeded"] }],
      "typeProperties": { "scripts": [{ "text": "SELECT COUNT(*) AS RowCount FROM BFX_DV.dv.HUB_Customer" }] }
    },
    { "name": "EP_LOG_LastLoadDate", "dependsOn": [{ "activity": "DQ_Check_Customer" }] }
  ]
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: The parameter persistence activities (EP_LOG_*) run immediately after the main processing, bypassing the post-process step. Fix: Always set CustomOutput.OutputPathName = outputPathName;.
2. Mistake: Confusing AdfPostProcess with AdfPostCopy. Symptom: The logic runs after the Copy Activity instead of after all processing. Fix: Use AdfPostProcess for end-of-pipeline logic and AdfPostCopy for logic immediately after the copy step.
3. Mistake: Using AdfPostProcess when the logic should run at batch level. Symptom: The activity runs once per object instead of once per batch. Fix: Use AdfBatchPostProcess for batch-level post-processing.
4. Mistake: Missing the dependency reference. Symptom: The post-process activity runs in parallel with the main processing. Fix: Always include <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>">.

Related Extension Points

• AdfPreProcess -- runs as the first step in the MainActivity IfCondition; the counterpart to post-process.
• AdfPostCopy -- runs immediately after the Copy Activity, before staging and DV processing.
• AdfBatchPostProcess -- batch-level post-processing that runs after all object pipelines complete.
• AdfSetParameter -- persists parameters after post-process; customize both to control the end-of-pipeline flow.

Pre Copy Script

Configure a Pre Copy Script Attribute for the Sink property of a Copy Activity.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the (source) Object, from which the data will be copied in the Copy Activity.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the (target) Object, which will act as the sink for the Copy Activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.
OutputPathName	String	You must add CustomOutput.OutputPathName with the last task to connect it with the next Data Flow task.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPreCopyScript" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>



<!-- This Extension Point overrides the 'Pre-copy script' property in the Sink definition of the targeted Copy Activity. -->
<!-- The below example populates a Pre-copy script. -->

<!-- Assuming the Name property of the 'targetTable' variable is 'AWLT_SRC.SalesLT.Address' the output of the below code will be -->
<!-- <PreCopyScript>TRUNCATE TABLE AWLT_SRC.SalesLT.Address</PreCopyScript> -->
<!-- The <PreCopyScript> tags will be automatically added by BimlFlex. -->

TRUNCATE TABLE <#=targetTable.Name#>

What It Does

Overrides the PreCopyScript property on the Sink definition of a Copy Activity in an ADF pipeline. The pre-copy script runs on the target (sink) database immediately before data is written by the Copy Activity. This is commonly used to truncate or clean up the target table before loading new data, ensuring idempotent loads and preventing duplicate records.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point is emitted as the preCopyScript property inside the Copy Activity's sink definition in the generated ARM template. It fires once per Copy Activity for each targeted object.

When to Use It

1. You need to truncate the landing or staging table before each load (e.g., TRUNCATE TABLE BFX_LND.SalesLT.Customer).
2. You want to delete only specific partitions or date ranges from the target table before loading (e.g., DELETE FROM BFX_STG.SalesLT.Customer WHERE LoadDate < GETDATE()-7).
3. You need to run a custom cleanup script that differs from the default BimlFlex behavior.
4. You want to disable indexes or constraints before the copy for performance reasons. Use this instead of AdfPreProcess when the cleanup should be a property of the Copy Activity itself rather than a separate pipeline activity.

Prerequisites

• A Copy Activity must exist in the pipeline (source-to-staging or source-to-landing process).
• The sourceTable and targetTable parameters provide access to the source and target object metadata.
• The sink database must support the SQL commands used in the pre-copy script.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the pre-copy script:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPreCopyScript" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>

TRUNCATE TABLE <#=targetTable.Name#>

3. Build and verify the preCopyScript property appears in the Copy Activity's sink definition in the generated ARM template.

Example

Before (no pre-copy script):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "sink": { "type": "AzureSqlSink" }
}

After (with truncate pre-copy script):

{ "name": "Copy_SalesLT_Customer",
  "type": "Copy",
  "sink": {
    "type": "AzureSqlSink",
    "preCopyScript": "TRUNCATE TABLE BFX_LND.SalesLT.Customer"
  }
}

Common Mistakes

1. Mistake: Including the <PreCopyScript> XML tags in the extension point output. Symptom: The generated ARM template contains double-wrapped tags. Fix: Only output the SQL text; BimlFlex wraps it in <PreCopyScript> tags automatically.
2. Mistake: Using DDL statements that are not supported as pre-copy scripts (e.g., CREATE TABLE). Symptom: The Copy Activity fails at runtime with a SQL syntax error. Fix: Limit the script to DML operations like TRUNCATE TABLE or DELETE FROM.
3. Mistake: Truncating a table that is referenced by foreign keys. Symptom: The pre-copy script fails with a foreign key constraint error. Fix: Use DELETE FROM instead of TRUNCATE TABLE, or drop and recreate foreign keys.
4. Mistake: Not using ObjectInherit when applying at batch level. Symptom: Only one object's Copy Activity gets the pre-copy script. Fix: Set CustomOutput.ObjectInherit = true; and use dynamic table references like <#=targetTable.Name#>.

Related Extension Points

• AdfPreProcess -- adds a separate activity before the Copy Activity (not a sink property).
• AdfPostCopy -- adds logic after the Copy Activity completes.
• OverrideAdfCopySink -- overrides the entire sink definition, including the pre-copy script.
• AdfProcessStaging -- the staging process that runs after the copy lands data.

Pre Process

Add an activity as the first step in the MainActivity IfCondition of an individual execute pipeline.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the object to which the activity will be added.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the activity will be added.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPreProcess" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>



<# 
// Object inheritance is enabled by default.
// This means the target can be set either as individual object or parent objects (e.g. Batch-level), and this extension will be applied to all child objects.
CustomOutput.ObjectInherit = true;

// The ADF Pre Process extension points targets an ADF Pipeline as the first entry point in the 'MainActivity' IfCondition.
// This extension point can be used to add an Activity using BimlScript syntax as the first one in the IfCondition.
// In the Biml output, the location is specified as <!-- Placeholder for extensionpoint="AdfPreProcess" -->.

// As it is the first activity in the IfCondition, there are no incoming dependencies.
// However, outgoing dependencies need to be connected so the CustomOutput.OutputPathName needs to be set for the extension point to be integrated properly.

// This extension point comes with input parameters 'sourceTable' and 'targetTable', which can be used in the logic.
// Their properties (e.g. sourceTable.Name) can be viewed using the BimlStudio Intellisense.
#>

<#
// The following is an example of how BimlScript can be used for this extension point.
var outputPathName = "Wait for Ten Seconds Pre Process";
#>
<Wait Name="<#=outputPathName#>" WaitTimeInSeconds="10"></Wait>
<# 

// The output path name is set here, so that BimlFlex can connect this activity to the other generated components.
CustomOutput.OutputPathName = outputPathName;#>

What It Does

Injects a custom activity as the very first step inside the MainActivity IfCondition of an individual object's Execute Pipeline in Azure Data Factory. This runs before any parameter lookups, copy activities, or processing steps. Common uses include setting up temporary resources, running pre-validation checks, or logging custom metadata before the main data flow begins.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline's MainActivity IfCondition as the first activity. Because it is the first activity, there are no incoming dependencies, but the CustomOutput.OutputPathName must be set so downstream activities can depend on it.

When to Use It

1. You need to run a data quality pre-check on the source before extracting data (e.g., verifying row counts or schema changes on SalesLT.Customer in AWLT_SRC).
2. You want to truncate a staging table before the copy activity lands data, as an alternative to using AdfPreCopyScript.
3. You need to call a custom logging procedure at the start of each object pipeline.
4. You want to set a pipeline variable based on external logic before the main processing begins. Use this instead of AdfBatchPreProcess when you need per-object pre-processing rather than batch-level pre-processing.

Prerequisites

• A source-to-target mapping must exist in BimlFlex metadata for the object.
• The sourceTable and targetTable parameters provide access to object metadata.
• CustomOutput.OutputPathName must be set to connect the activity to the downstream dependency chain.
• There is no incoming dependency parameter because this is the first activity in the IfCondition.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the pre-process logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfPreProcess" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>

<# CustomOutput.ObjectInherit = true; #>
<# var outputPathName = "PreCheck_" + sourceTable.ObjectName; #>
<Script Name="<#=outputPathName#>" Timeout="0.00:10:00"
        LinkedServiceName="<#=sourceTable.Connection.RelatedItem.Name#>">
  <Scripts>
    <Script ScriptType="Query">
      <Query>SELECT COUNT(*) AS RowCount FROM <#=sourceTable.Name#></Query>
    </Script>
  </Scripts>
</Script>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the new activity appears as the first step in the MainActivity IfCondition.

Example

Before (no pre-process -- copy activity is first):

{ "activities": [
    { "name": "Copy_SalesLT_Customer", "type": "Copy", "dependsOn": [] }
  ]
}

After (pre-check runs before copy):

{ "activities": [
    { "name": "PreCheck_Customer", "type": "Script",
      "typeProperties": { "scripts": [{ "text": "SELECT COUNT(*) AS RowCount FROM SalesLT.Customer" }] }
    },
    { "name": "Copy_SalesLT_Customer", "type": "Copy",
      "dependsOn": [{ "activity": "PreCheck_Customer", "dependencyConditions": ["Succeeded"] }]
    }
  ]
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: All downstream activities (copy, staging, DV) lose their dependency on the pre-process step and execute immediately. Fix: Always set CustomOutput.OutputPathName = outputPathName;.
2. Mistake: Adding a dependency on a non-existent activity. Symptom: The pipeline fails validation because no incoming dependency exists for the first activity. Fix: Do not add <Dependencies> because this is the first activity in the IfCondition.
3. Mistake: Confusing AdfPreProcess with AdfBatchPreProcess. Symptom: The logic runs once per batch instead of once per object. Fix: Use AdfPreProcess for per-object logic and AdfBatchPreProcess for batch-level logic.
4. Mistake: Using a long-running query without an adequate timeout. Symptom: The pre-process step times out and the entire pipeline fails. Fix: Set an appropriate Timeout value for the expected query duration.

Related Extension Points

• AdfPostProcess -- runs as the last step in the MainActivity IfCondition; the counterpart to pre-process.
• AdfPreCopyScript -- sets the pre-copy script attribute on the Copy Activity sink (e.g., TRUNCATE TABLE).
• AdfBatchPreProcess -- batch-level pre-processing that runs before all object pipelines.
• AdfGetParameter -- retrieves parameters before the copy; runs after pre-process.

Process Data Vault

Add bespoke logic that will replace the default Data Vault processing script for Data Factory pipelines (SQL_PROCESS_DV).

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the object to which the activity will be added.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the activity will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfProcessDataVault" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- The ADF Process Data Vault extension point targets the Script Activity that executes the Data Vault process in individual ADF Pipeline. This is the SQL_PROCESS_DV activity by default. -->
<!-- This extension point can be used to add bespoke logic using BimlScript syntax that will replace the code that is generated by BimlFlex as default. -->
<!-- In the Biml output, the location is specified as 'Placeholder for extensionpoint="AdfProcessDataVault"'. -->
<!-- The below example contains the default script that will be generated by BimlFlex. -->

<# 
// Object inheritance is enabled by default.
// This means the target can be set either as individual object or parent objects (e.g. Batch-level), and this extension will be applied to all child objects.
CustomOutput.ObjectInherit = true;
#>

<#
// The following is an example of how BimlScript can be used for this extension point.
var outputPathName = "SQL_PROCESS_DV";
#>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00" LinkedServiceName="<#=targetTable.Connection.RelatedItem.Name#>">
    <Scripts>
        <Script ScriptType="Query">
            <Query>CALL dv.flex_<#=targetTable.Connection.RelatedItem.RecordSource#>_<#=targetTable.ObjectName#>;</Query>
            <Parameters>
				<Parameter Name="RowAuditId" DataType="Int64" Direction="Input" TreatAsNull="false">@activity('LogExecutionStart').output.firstRow.ExecutionID</Parameter>
            </Parameters>
        </Script>
    </Scripts>
    <Dependencies>
        <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
    </Dependencies>
</Script>
<# 

// Wrapping up
CustomOutput.OutputPathName = outputPathName;#>

What It Does

Replaces the default Data Vault processing Script Activity (SQL_PROCESS_DV) in individual ADF object pipelines. This activity typically executes the stored procedure that loads data from the staging area into Data Vault structures (Hubs, Links, Satellites). By overriding this extension point, you can customize the Data Vault stored procedure call, add additional transformation logic, or redirect the load to a different Data Vault implementation.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline's MainActivity IfCondition, after the staging process completes. It replaces the SQL_PROCESS_DV Script Activity.

When to Use It

1. You need to call a custom Data Vault stored procedure instead of the default BimlFlex-generated flex_* procedure.
2. You want to add a row count audit step between staging and Data Vault loading.
3. The target Data Vault database uses a different schema or naming convention for stored procedures.
4. You need to execute multiple stored procedures in sequence (e.g., load Hub first, then Satellite) within a single activity replacement. Use this instead of AdfPostProcess when you need to replace the Data Vault execution itself rather than adding steps after it.

Prerequisites

• A Data Vault target must be configured in BimlFlex metadata (e.g., the BFX_DV connection).
• The Data Vault stored procedure must exist on the target database.
• The dependency parameter provides the name of the preceding activity for chaining.
• The staging process should have completed successfully before this runs.

Implementation Steps

1. Create a new extension point targeting the source object or batch (e.g., SalesLT.Customer on AWLT_SRC).
2. Add the BimlScript directive with the custom Data Vault logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfProcessDataVault" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = true; #>
<# var outputPathName = "SQL_PROCESS_DV"; #>
<Script Name="<#=outputPathName#>" Timeout="0.01:00:00"
        LinkedServiceName="<#=targetTable.Connection.RelatedItem.Name#>">
  <Scripts>
    <Script ScriptType="Query">
      <Query>EXEC [dv].[usp_load_<#=targetTable.ObjectName#>] @RowAuditId = @activity('LogExecutionStart').output.firstRow.ExecutionID</Query>
    </Script>
  </Scripts>
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Script>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify that the SQL_PROCESS_DV activity in the generated pipeline reflects your custom procedure.

Example

Before (default BimlFlex Data Vault call):

{ "name": "SQL_PROCESS_DV",
  "type": "Script",
  "typeProperties": { "scripts": [{
    "text": "CALL dv.flex_AWLT_SRC_Customer;"
  }]}
}

After (custom stored procedure with audit parameter):

{ "name": "SQL_PROCESS_DV",
  "type": "Script",
  "typeProperties": { "scripts": [{
    "text": "EXEC [dv].[usp_load_HUB_Customer] @RowAuditId = 12345"
  }]}
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: The parameter persistence activities (EP_LOG_*) lose their dependency chain and either run prematurely or not at all. Fix: Always set CustomOutput.OutputPathName = outputPathName;.
2. Mistake: Using the staging Linked Service instead of the Data Vault Linked Service. Symptom: The Script Activity attempts to run on the staging database. Fix: Use targetTable.Connection.RelatedItem.Name for the Data Vault Linked Service.
3. Mistake: Replacing the Data Vault process without ensuring the staging process has completed. Symptom: The Data Vault procedure reads empty or stale staging tables. Fix: Verify the dependency parameter references the correct preceding activity (typically SQL_PROCESS_STG).
4. Mistake: Setting ObjectInherit = false at batch level. Symptom: Only one object gets the custom Data Vault logic. Fix: Set CustomOutput.ObjectInherit = true; when applying at batch level.

Related Extension Points

• AdfProcessStaging -- the staging process that runs before Data Vault loading; customize both when changing the data flow.
• AdfPostProcess -- runs after the Data Vault process within the MainActivity IfCondition.
• AdfSetParameter -- persists parameter values after Data Vault processing completes.
• AdfPostCopy -- runs immediately after the Copy Activity and before staging/DV processing.

Process Staging

Add bespoke logic that will replace the default Staging Area processing script for Data Factory pipelines (SQL_PROCESS_STG).

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the object to which the activity will be added.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the activity will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfProcessStaging" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- The ADF Process Staging extension point targets the Script Activity that executes the source-to-staging process in individual ADF Pipeline. This is the SQL_PROCESS_STG activity by default. -->
<!-- This extension point can be used to add bespoke logic using BimlScript syntax that will replace the code that is generated by BimlFlex as default. -->
<!-- In the Biml output, the location is specified as 'Placeholder for extensionpoint="AdfProcessStaging"'. -->
<!-- The below example contains the default script that will be generated by BimlFlex. -->

<# 
// Object inheritance is enabled by default.
// This means the target can be set either as individual object or parent objects (e.g. Batch-level), and this extension will be applied to all child objects.
CustomOutput.ObjectInherit = true;
#>

<#
// The following is an example of how BimlScript can be used for this extension point.
var outputPathName = "SQL_PROCESS_STG";
#>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00" LinkedServiceName="<#=targetTable.Connection.RelatedItem.Name#>">
    <Scripts>
        <Script ScriptType="Query">
            <Query>CALL <#=sourceTable.Connection.RelatedItem.RecordSource#>.flex_<#=sourceTable.Connection.RelatedItem.RecordSource#>_<#=sourceTable.ObjectName#>;</Query>            
            <Parameters>
				<Parameter Name="RowAuditId" DataType="Int64" Direction="Input" TreatAsNull="false">@activity('LogExecutionStart').output.firstRow.ExecutionID</Parameter>
            </Parameters>
        </Script>
    </Scripts>
    <Dependencies>
        <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
    </Dependencies>
</Script>
<# 

// Wrapping up
CustomOutput.OutputPathName = outputPathName;#>

What It Does

Replaces the default source-to-staging processing Script Activity (SQL_PROCESS_STG) in individual ADF object pipelines. This activity typically executes a stored procedure that transforms and loads data from the landing area into the staging tables. By overriding this extension point, you can customize the staging stored procedure call, add additional processing steps, or redirect staging to a different target system.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline's MainActivity IfCondition, after the Copy Activity lands data and before any Data Vault processing. It replaces the SQL_PROCESS_STG Script Activity.

When to Use It

1. You need to call a custom staging stored procedure instead of the default BimlFlex-generated flex_* procedure.
2. You want to add a data quality check or row count validation between the copy and staging steps.
3. The target staging database uses a different naming convention for stored procedures than the BimlFlex default.
4. You need to pass additional parameters to the staging procedure (e.g., a processing mode flag). Use this instead of AdfPreProcess or AdfPostProcess when you need to replace the staging execution itself, not add steps around it.

Prerequisites

• A source-to-staging process must be configured in BimlFlex metadata (source connection such as AWLT_SRC mapped to a staging connection such as BFX_STG).
• The staging stored procedure must exist on the target database.
• The dependency parameter provides the name of the preceding activity for chaining.
• The target Linked Service must be configured and accessible from the ADF Integration Runtime.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the custom staging logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfProcessStaging" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = true; #>
<# var outputPathName = "SQL_PROCESS_STG"; #>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00"
        LinkedServiceName="<#=targetTable.Connection.RelatedItem.Name#>">
  <Scripts>
    <Script ScriptType="Query">
      <Query>EXEC [stg].[usp_load_<#=sourceTable.ObjectName#>] @ExecutionID = @activity('LogExecutionStart').output.firstRow.ExecutionID</Query>
      <Parameters>
        <Parameter Name="RowAuditId" DataType="Int64" Direction="Input"
                   TreatAsNull="false">@activity('LogExecutionStart').output.firstRow.ExecutionID</Parameter>
      </Parameters>
    </Script>
  </Scripts>
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Script>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify that the SQL_PROCESS_STG activity in the generated pipeline reflects your custom procedure.

Example

Before (default BimlFlex staging call):

{ "name": "SQL_PROCESS_STG",
  "type": "Script",
  "typeProperties": { "scripts": [{
    "text": "CALL AWLT_SRC.flex_AWLT_SRC_Customer;"
  }]}
}

After (custom stored procedure with execution ID):

{ "name": "SQL_PROCESS_STG",
  "type": "Script",
  "typeProperties": { "scripts": [{
    "text": "EXEC [stg].[usp_load_Customer] @ExecutionID = 12345"
  }]}
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: The Data Vault processing step (SQL_PROCESS_DV) or parameter persistence activities lose their dependency chain. Fix: Always set CustomOutput.OutputPathName = outputPathName;.
2. Mistake: Using the wrong LinkedServiceName. Symptom: The Script Activity attempts to run on the source database instead of the staging database. Fix: Use targetTable.Connection.RelatedItem.Name for the staging Linked Service.
3. Mistake: Replacing the staging process without updating the corresponding Data Vault process. Symptom: The Data Vault load fails because it expects the default staging table structure. Fix: Review AdfProcessDataVault to ensure compatibility with your custom staging logic.
4. Mistake: Setting a timeout that is too short for large table loads. Symptom: The Script Activity times out on large objects like SalesLT.SalesOrderDetail. Fix: Increase the Timeout attribute (e.g., 0.02:00:00 for 2 hours).

Related Extension Points

• AdfProcessDataVault -- the Data Vault processing step that runs after staging; customize both when changing the data flow.
• AdfPostCopy -- runs immediately after the Copy Activity and before staging processing.
• AdfPreProcess / AdfPostProcess -- add logic before or after the main activity block.
• AdfPreCopyScript -- sets the pre-copy script on the sink, often used to truncate staging tables.

Set Parameter

Override the parameters that are saved back to metadata, and how they behave in the Data Factory Pipeline. Note that this extension point replaces all parameters.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the object to which the activity will be added.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the activity will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the batch.
OutputPathName	String	You must add CustomOutput.OutputPathName with the last task to connect it with the next Data Flow task.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfSetParameter" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- This extension point allows the manipulation of metadata parameters that are used in the data pipelines. -->
<!-- The most common use-case is to change the behaviour of the parameters that are already defined in the metadata. -->
<!-- In this case, the parameter code will be replaced by the contents on this Extension Point. -->

<!-- This extension point affects all Parameters that are referred to after the main 'Copy Data' activity in the targeted ADF Pipeline. -->
<!-- This can be used to control how parameter values are saved back into the BimlFlex repository database. -->
<!-- The below example shows how parameters (e.g. LOG_LastLoadDate) are replaced by values retrieved from metadata. -->
            
<# 	CustomOutput.ObjectInherit = true; #>

<#	var srcObj = new ObjectViewModel(sourceTable, sourceTable.Connection.RelatedItem.IntegrationStage, "SRC");
	var parameters = srcObj.Parameters;
	if (!parameters.Any()) return null; 
	foreach(var parameter in parameters) {
		if (parameter.IsNotPersisted == "Y") continue; 
		var parameterColumn = parameter.Column.RelatedItem != null ? parameter.Column.RelatedItem.Name : "";
		parameterColumn = string.IsNullOrWhiteSpace(parameterColumn)
			? parameter.Object.RelatedItem != null ? parameter.Object.RelatedItem.Name + "." + parameter.ParameterName : parameter.ParameterName
			: parameterColumn; #>
	<SqlServerStoredProcedure Name="EP_LOG_<#=parameter.ParameterName #>" LinkedServiceName="BimlCatalog" StoredProcedure="[adf].[SetConfigVariable]">
	<#	if (!string.IsNullOrEmpty(dependency)) {#>
		<Dependencies>
			<Dependency DependsOnActivityName="<#=dependency #>" />
		</Dependencies>
	<#	} #>
		<StoredProcedureParameters>
			<Parameter Name="SystemName" DataType="String" IsNull="false"><#=srcObj.SourceConnection.Name#></Parameter>
			<Parameter Name="ObjectName" DataType="String" IsNull="false"><#=parameterColumn#></Parameter>
			<Parameter Name="VariableName" DataType="String" IsNull="false"><#=parameter.ParameterName#></Parameter>
			<Parameter Name="VariableValue" DataType="String" IsNull="false">@activity('LKP_<#=parameter.ParameterToName #>').output.firstRow.<#=parameter.ParameterToName #></Parameter>
			<Parameter Name="ExecutionID" DataType="Int64" IsNull="false">@activity('LogExecutionStart').output.firstRow.ExecutionID</Parameter>
		</StoredProcedureParameters>
	</SqlServerStoredProcedure>
<#	} #>

What It Does

Overrides the default logic that persists parameter values (such as high-water mark dates) back to the BimlCatalog after a successful data load. By default, BimlFlex generates SqlServerStoredProcedure activities that call [adf].[SetConfigVariable] to save the updated parameter values. This extension point replaces all of those activities, giving you full control over how and when load window boundaries and other parameter values are written back to the metadata repository.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires within the individual object pipeline, after the main processing activities (Copy, Process Staging, Process Data Vault) complete. It targets the parameter persistence section at the end of the MainActivity IfCondition.

When to Use It

1. You need to change the stored procedure used to persist parameter values (e.g., using a custom logging procedure instead of [adf].[SetConfigVariable]).
2. You want to add conditional logic that only updates the high-water mark when all rows pass a quality check.
3. You need to persist additional custom parameters beyond what BimlFlex defines by default.
4. You want to write parameter values to a different target (e.g., an Azure SQL Database instead of the BimlCatalog). Use this instead of AdfGetParameter when you need to control the write-back, not the initial retrieval.

Prerequisites

• Parameters must be defined on the source object in BimlFlex metadata.
• The BimlCatalog database Linked Service must be configured (or whichever Linked Service the custom logic targets).
• The AdfGetParameter extension point should be reviewed to ensure consistency between parameter retrieval and persistence.
• The dependency parameter is required to chain this after prior activities.

Implementation Steps

1. Create a new extension point targeting the source object (e.g., SalesLT.Customer on AWLT_SRC).
2. Add the BimlScript directive with the custom parameter persistence logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfSetParameter" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = true; #>
<#
var srcObj = new ObjectViewModel(sourceTable, sourceTable.Connection.RelatedItem.IntegrationStage, "SRC");
var parameters = srcObj.Parameters;
if (!parameters.Any()) return null;
foreach(var parameter in parameters) {
  if (parameter.IsNotPersisted == "Y") continue;
#>
<SqlServerStoredProcedure Name="EP_LOG_<#=parameter.ParameterName#>"
    LinkedServiceName="BimlCatalog"
    StoredProcedure="[adf].[SetConfigVariable]">
  <Dependencies>
    <Dependency DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
  <StoredProcedureParameters>
    <Parameter Name="SystemName" DataType="String"><#=srcObj.SourceConnection.Name#></Parameter>
    <Parameter Name="ObjectName" DataType="String"><#=parameter.ParameterName#></Parameter>
    <Parameter Name="VariableName" DataType="String"><#=parameter.ParameterName#></Parameter>
    <Parameter Name="VariableValue" DataType="String">@activity('LKP_<#=parameter.ParameterToName#>').output.firstRow.<#=parameter.ParameterToName#></Parameter>
    <Parameter Name="ExecutionID" DataType="Int64">@activity('LogExecutionStart').output.firstRow.ExecutionID</Parameter>
  </StoredProcedureParameters>
</SqlServerStoredProcedure>
<# } #>

3. Build and verify the EP_LOG_* activities appear in the generated pipeline with your custom logic.

Example

Before (default parameter persistence):

{ "name": "EP_LOG_LastLoadDate",
  "type": "SqlServerStoredProcedure",
  "typeProperties": {
    "storedProcedureName": "[adf].[SetConfigVariable]",
    "storedProcedureParameters": {
      "VariableValue": "@activity('LKP_NextLoadDate').output.firstRow.NextLoadDate"
    }
  }
}

After (custom persistence with quality gate):

{ "name": "EP_LOG_LastLoadDate",
  "type": "SqlServerStoredProcedure",
  "dependsOn": [{ "activity": "DQ_Check", "dependencyConditions": ["Succeeded"] }],
  "typeProperties": {
    "storedProcedureName": "[custom].[SetParameterWithAudit]",
    "storedProcedureParameters": {
      "VariableValue": "@activity('LKP_NextLoadDate').output.firstRow.NextLoadDate",
      "RowCount": "@activity('Copy_SalesLT_Customer').output.rowsCopied"
    }
  }
}

Common Mistakes

1. Mistake: Overriding AdfSetParameter without reviewing AdfGetParameter. Symptom: The parameter names or variable references used in the set logic do not match what was retrieved. Fix: Ensure both extension points use consistent parameter names and activity references.
2. Mistake: Not returning null when there are no parameters. Symptom: An empty activity block is generated, causing validation errors. Fix: Include if (!parameters.Any()) return null; at the top of the loop.
3. Mistake: Missing the dependency in the <Dependencies> block. Symptom: The parameter update activity runs before the data load completes, saving incorrect values. Fix: Always include <Dependency DependsOnActivityName="<#=dependency#>" />.

Related Extension Points

• AdfGetParameter -- retrieves parameter values before the main copy; the counterpart to this extension point.
• AdfPostProcess -- runs after the main activity but before parameter persistence; can be used for validation before write-back.
• AdfPostCopy -- runs immediately after the copy activity; useful for interim processing before parameters are set.

Snowflake Create Stage

Add bespoke logic that will replace the default Create Stage script for Snowflake.

Parameters

Name	Type	Description
sourceTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the object to which the activity will be added.
targetTable	BimlFlexModelWrapper.ObjectsWrapper	Contains all information related to the target object to which the activity will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfSnowflakeCreateStage" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- The ADF Snowflake Create Stage extension point targets the 'Create Stage' Script Activity that is part of an individual ADF Pipeline.-->
<!-- This extension point can be used to add bespoke logic using BimlScript syntax that will replace the code that is generated by BimlFlex as default. -->
<!-- In the Biml output, the location is specified as 'Placeholder for extensionpoint="AdfSnowflakeCreateStage"'. -->
<!-- The below example contains the default script that will be generated by BimlFlex. -->

<# 
// Object inheritance is enabled by default.
// This means the target can be set either as individual object or parent objects (e.g. Batch-level), and this extension will be applied to all child objects.
CustomOutput.ObjectInherit = true;
#>

<#
// The following is an example of how BimlScript can be used for this extension point.
var outputPathName = "SFLDW_CREATE_STAGE";
#>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00" LinkedServiceName="<#=targetTable.Connection.RelatedItem.Name#>">
    <Scripts>
        <Script ScriptType="Query">
            <Query>@concat('CREATE OR REPLACE STAGE ""{$"<#=targetTable.Schema#>\".\"stage_<#=sourceTable.Name#>\""} URL = ''azure://', pipeline().parameters.AzureStorageAccount, '.{stageContainerDomain}/', pipeline().parameters.AzureStageContainer, '/<#=sourceTable.Name#>/'' CREDENTIALS=(AZURE_SAS_TOKEN=''', pipeline().parameters.AzureStorageSasToken, ''') FILE_FORMAT = (TYPE = ''PARQUET'');')</Query>
        </Script>
    </Scripts>
    <Dependencies>
        <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
    </Dependencies>
</Script>
<# 

// Wrapping up
CustomOutput.OutputPathName = outputPathName;#>

What It Does

Replaces the default Snowflake CREATE OR REPLACE STAGE Script Activity generated by BimlFlex in individual object pipelines. This activity creates the external stage that Snowflake uses to read Parquet files from Azure Blob Storage during the COPY INTO operation. Customizing this extension point lets you control the stage URL, credentials method, and file format options for each source object.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires once per object pipeline during generation when the target connection is Snowflake. It replaces the SFLDW_CREATE_STAGE Script Activity in the individual pipeline's MainActivity IfCondition.

When to Use It

1. You need to use a storage integration instead of a SAS token for the external stage credentials (e.g., for tighter security governance).
2. The default stage URL pattern does not match your Azure Storage account structure (e.g., ADLS Gen2 hierarchical namespace with custom container paths).
3. You need to specify a file format other than Parquet (e.g., CSV or JSON) for specific source objects.
4. You want to add additional stage options such as ENCRYPTION or MASTER_KEY. Use this instead of manually editing the ARM template output, because the extension point is preserved across regeneration.

Prerequisites

• The target connection must be a Snowflake connection type.
• Azure Storage parameters (AzureStorageAccount, AzureStageContainer, AzureStorageSasToken) must be configured in the pipeline.
• The Snowflake Linked Service must have permissions to execute CREATE OR REPLACE STAGE.
• The source object must be configured for file-based extraction to Blob Storage.

Implementation Steps

1. Create a new extension point targeting the source object or batch.
2. Add the BimlScript directive with the custom stage creation logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfSnowflakeCreateStage" #>
<#@ property name="sourceTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="targetTable" type="BimlFlexModelWrapper.ObjectsWrapper" #>
<#@ property name="dependency" type="String" #>

<# CustomOutput.ObjectInherit = true; #>
<# var outputPathName = "SFLDW_CREATE_STAGE"; #>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00"
        LinkedServiceName="<#=targetTable.Connection.RelatedItem.Name#>">
  <Scripts>
    <Script ScriptType="Query">
      <Query>@concat('CREATE OR REPLACE STAGE "', '<#=targetTable.Schema#>', '"."stage_<#=sourceTable.Name#>" URL = ''azure://', pipeline().parameters.AzureStorageAccount, '.blob.core.windows.net/', pipeline().parameters.AzureStageContainer, '/<#=sourceTable.Name#>/'' STORAGE_INTEGRATION = my_azure_integration FILE_FORMAT = (TYPE = ''PARQUET'');')</Query>
    </Script>
  </Scripts>
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
</Script>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify the SFLDW_CREATE_STAGE activity in the generated ARM template.

Example

Before (default with SAS token):

{ "name": "SFLDW_CREATE_STAGE",
  "typeProperties": { "scripts": [{
    "text": "CREATE OR REPLACE STAGE \"stg\".\"stage_SalesLT_Customer\" URL = 'azure://mystorage.blob.core.windows.net/stage/SalesLT_Customer/' CREDENTIALS=(AZURE_SAS_TOKEN='...') FILE_FORMAT = (TYPE = 'PARQUET');"
  }]}
}

After (using storage integration):

{ "name": "SFLDW_CREATE_STAGE",
  "typeProperties": { "scripts": [{
    "text": "CREATE OR REPLACE STAGE \"stg\".\"stage_SalesLT_Customer\" URL = 'azure://mystorage.blob.core.windows.net/stage/SalesLT_Customer/' STORAGE_INTEGRATION = my_azure_integration FILE_FORMAT = (TYPE = 'PARQUET');"
  }]}
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: The COPY INTO activity that depends on the stage creation step has a broken dependency chain. Fix: Always set CustomOutput.OutputPathName = outputPathName;.
2. Mistake: Incorrect quoting of Snowflake identifiers in the @concat expression. Symptom: Snowflake raises a syntax error during stage creation. Fix: Use doubled single quotes ('') for literal quotes inside the ADF expression, and ensure schema/stage names are properly escaped.
3. Mistake: Using a storage integration that does not have access to the target container. Symptom: The CREATE STAGE succeeds but the COPY INTO fails with an access denied error. Fix: Verify the storage integration's external ID and allowed locations in Snowflake.
4. Mistake: Setting ObjectInherit = false when applied at batch level. Symptom: Only one object pipeline gets the custom stage logic. Fix: Set CustomOutput.ObjectInherit = true; when targeting at batch level.

Related Extension Points

• AdfSnowflakeScaleUp / AdfSnowflakeScaleDown -- warehouse sizing that runs before and after the load.
• AdfProcessStaging -- the staging process script that runs after data is copied and staged.
• ExternalFileFormat -- defines file formats for PolyBase external tables (Synapse/SQL DW, not Snowflake).

Snowflake Scale Down

Add bespoke logic that will replace the default Scale Down script for Snowflake which is generated at Batch level.

Parameters

Name	Type	Description
batch	BimlFlexModelWrapper.BatchesWrapper	Contains all information about the Data Factory (Batch) Pipeline to which the Extension Point content will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfSnowflakeScaleDown" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- The ADF Snowflake Scale Down extension point targets the 'Scale Down' Script Activity that is part of an ADF Batch Pipeline.-->
<!-- This extension point can be used to add bespoke logic using BimlScript syntax that will replace the code that is generated by BimlFlex as default. -->
<!-- In the Biml output, the locaUption is specified as 'Placeholder for extensionpoint="AdfSnowflakeScaleDown"'. -->
<!-- The below example contains the default script that will be generated by BimlFlex. -->

<#
// The following is an example of how BimlScript can be used for this extension point.
var outputPathName = "SFLSQL - SCALE DOWN";

var scaleDown = batch.GetConfigurationValue("SnowflakeScaleDown") == "Y";
var autoSuspend = batch.GetConfigurationValue("SnowflakeAutoSuspend") == "Y";
                
var snowflakeScaleDown = scaleDown
    ? "'ALTER WAREHOUSE ', pipeline().parameters.SnowflakeWarehouse, ' SET WAREHOUSE_SIZE = ', pipeline().parameters.SnowflakeScaleDownSize, ';'"
    : "";

var snowflakeAutoSuspend = autoSuspend ? "'ALTER WAREHOUSE ', pipeline().parameters.SnowflakeWarehouse, ' SUSPEND;'" : "";

snowflakeScaleDown = string.IsNullOrWhiteSpace(snowflakeAutoSuspend) ? snowflakeScaleDown + ", " + snowflakeAutoSuspend : snowflakeScaleDown;
#>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00" LinkedServiceName=">
    <Dependencies>
        <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
    </Dependencies>
    <Scripts>
        <Script ScriptType="Query">
            <Query>@concat(<#=snowflakeScaleDown#>)</Query>
        </Script>
    </Scripts>
</Script>
<# 

// Wrapping up
CustomOutput.OutputPathName = outputPathName;#>

What It Does

Replaces the default Snowflake warehouse scale-down Script Activity that BimlFlex generates at the end of a batch pipeline. This activity reduces the Snowflake virtual warehouse size and optionally suspends it after batch processing completes, directly controlling Snowflake credit consumption. The scale-down logic runs as one of the final steps in the batch pipeline, ensuring compute resources are released promptly.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires once per batch pipeline during generation when the target connection is Snowflake. It replaces the SFLSQL - SCALE DOWN Script Activity in the batch pipeline.

When to Use It

1. You want to suspend the warehouse immediately after the batch completes rather than just scaling down, to minimize idle credit usage.
2. You need to conditionally skip scale-down when another batch is expected to start shortly after (e.g., 02_LOAD_BFX_DV_Batch follows 01_EXT_AWLT_SRC_Batch).
3. The default scale-down size or auto-suspend behavior does not match your Snowflake cost governance policy.
4. You need to add logging or notification logic alongside the scale-down operation. Use this instead of modifying the generated ARM template, because the extension point preserves changes across regeneration.

Prerequisites

• The target connection must be a Snowflake connection type.
• Snowflake warehouse parameters (SnowflakeWarehouse, SnowflakeScaleDownSize) must be configured in the batch settings.
• The configuration values SnowflakeScaleDown and SnowflakeAutoSuspend control default behavior.
• The Snowflake Linked Service must have permissions to execute ALTER WAREHOUSE and SUSPEND commands.

Implementation Steps

1. Create a new extension point targeting the batch (e.g., 01_EXT_AWLT_SRC_Batch).
2. Add the BimlScript directive with the scale-down logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfSnowflakeScaleDown" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>
<#@ property name="dependency" type="String" #>

<# var outputPathName = "SFLSQL - SCALE DOWN"; #>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00" LinkedServiceName="BFX_STG">
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
  <Scripts>
    <Script ScriptType="Query">
      <Query>@concat('ALTER WAREHOUSE ', pipeline().parameters.SnowflakeWarehouse, ' SET WAREHOUSE_SIZE = ', pipeline().parameters.SnowflakeScaleDownSize, '; ALTER WAREHOUSE ', pipeline().parameters.SnowflakeWarehouse, ' SUSPEND;')</Query>
    </Script>
  </Scripts>
</Script>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify that the SFLSQL - SCALE DOWN activity in the generated ARM template reflects your custom logic.

Example

Before (default -- scale down only, no suspend):

{ "name": "SFLSQL - SCALE DOWN",
  "type": "Script",
  "typeProperties": { "scripts": [{
    "text": "ALTER WAREHOUSE ETL_WH SET WAREHOUSE_SIZE = XSMALL;"
  }]}
}

After (scale down and suspend):

{ "name": "SFLSQL - SCALE DOWN",
  "type": "Script",
  "typeProperties": { "scripts": [{
    "text": "ALTER WAREHOUSE ETL_WH SET WAREHOUSE_SIZE = XSMALL; ALTER WAREHOUSE ETL_WH SUSPEND;"
  }]}
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: The logging and cleanup activities after scale-down lose their dependency chain. Fix: Always set CustomOutput.OutputPathName = outputPathName;.
2. Mistake: Suspending the warehouse when downstream batches still need it. Symptom: The next batch fails with a "warehouse suspended" error during its scale-up retry window. Fix: Only suspend if this is the last batch in the execution chain, or allow auto-suspend via Snowflake's built-in idle timeout.
3. Mistake: Using a Condition="Completed" dependency instead of Condition="Succeeded". Symptom: The warehouse is scaled down even when the batch fails, potentially masking errors. Fix: Use Condition="Succeeded" or handle failure scenarios explicitly.

Related Extension Points

• AdfSnowflakeScaleUp -- scales the warehouse up at the start of the batch; always pair with scale-down.
• AdfSnowflakeCreateStage -- creates the Snowflake external stage for data loading.
• AdfBatchPostProcess -- adds post-batch logic that runs before or alongside scale-down.

Snowflake Scale Up

Add bespoke logic that will replace the default Scale Up script for Snowflake which is generated at Batch level.

Parameters

Name	Type	Description
batch	BimlFlexModelWrapper.BatchesWrapper	Contains all information about the Data Factory (Batch) Pipeline to which the Extension Point content will be added.
dependency	String	Contains the dependency name for the previous (incoming) activity.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfSnowflakeScaleUp" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>
<#@ property name="dependency" type="String" #>



<!-- The ADF Snowflake Scale Up extension point targets the 'Scale Up' Script Activity that is part of an ADF Batch Pipeline.-->
<!-- This extension point can be used to add bespoke logic using BimlScript syntax that will replace the code that is generated by BimlFlex as default. -->
<!-- In the Biml output, the location is specified as 'Placeholder for extensionpoint="AdfSnowflakeScaleUp"'. -->
<!-- The below example contains the default script that will be generated by BimlFlex. -->

<#
// The following is an example of how BimlScript can be used for this extension point.
var outputPathName = "SFLSQL - SCALE UP";
#>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00" LinkedServiceName=">
    <Dependencies>
        <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
    </Dependencies>
    <Scripts>
        <Script ScriptType="Query">
            <Query>@concat('ALTER WAREHOUSE ', pipeline().parameters.SnowflakeWarehouse, ' RESUME IF SUSPENDED; ALTER WAREHOUSE ', pipeline().parameters.SnowflakeWarehouse, ' SET WAREHOUSE_SIZE = ', pipeline().parameters.SnowflakeScaleUpSize, ';')</Query>
        </Script>
    </Scripts>
</Script>
<# 

// Wrapping up
CustomOutput.OutputPathName = outputPathName;#>

What It Does

Replaces the default Snowflake warehouse scale-up Script Activity that BimlFlex generates at the beginning of a batch pipeline. This ensures the Snowflake virtual warehouse is resumed (if suspended) and resized to the desired compute size before any data processing begins. Controlling warehouse sizing at batch start lets you right-size compute for the workload while managing Snowflake credit consumption.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point fires once per batch pipeline during generation when the target connection is Snowflake. It replaces the SFLSQL - SCALE UP Script Activity in the batch pipeline.

When to Use It

1. You need a larger warehouse size for specific heavy-load batches (e.g., 02_LOAD_BFX_DV_Batch requires X-Large while extraction batches use Medium).
2. You want to add conditional logic to skip scale-up during off-peak hours or for initial loads.
3. You need to resume a specific named warehouse that differs from the pipeline parameter default.
4. You want to add a USE ROLE statement before the ALTER WAREHOUSE command. Use this instead of manually modifying the generated ARM template, because changes are preserved across regeneration.

Prerequisites

• The target connection must be a Snowflake connection type.
• Snowflake warehouse parameters (SnowflakeWarehouse, SnowflakeScaleUpSize) must be configured in the batch settings.
• The Snowflake Linked Service must have permissions to execute ALTER WAREHOUSE and RESUME commands.

Implementation Steps

1. Create a new extension point targeting the batch (e.g., 01_EXT_AWLT_SRC_Batch).
2. Add the BimlScript directive with the scale-up logic:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfSnowflakeScaleUp" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>
<#@ property name="dependency" type="String" #>

<# var outputPathName = "SFLSQL - SCALE UP"; #>
<Script Name="<#=outputPathName#>" Timeout="0.00:30:00" LinkedServiceName="BFX_STG">
  <Dependencies>
    <Dependency Condition="Succeeded" DependsOnActivityName="<#=dependency#>" />
  </Dependencies>
  <Scripts>
    <Script ScriptType="Query">
      <Query>@concat('ALTER WAREHOUSE ', pipeline().parameters.SnowflakeWarehouse, ' RESUME IF SUSPENDED; ALTER WAREHOUSE ', pipeline().parameters.SnowflakeWarehouse, ' SET WAREHOUSE_SIZE = ', pipeline().parameters.SnowflakeScaleUpSize, ';')</Query>
    </Script>
  </Scripts>
</Script>
<# CustomOutput.OutputPathName = outputPathName; #>

3. Build and verify that the SFLSQL - SCALE UP activity in the generated ARM template contains your custom logic.

Example

Before (default generated scale-up):

{ "name": "SFLSQL - SCALE UP",
  "type": "Script",
  "typeProperties": { "scripts": [{
    "text": "ALTER WAREHOUSE ETL_WH RESUME IF SUSPENDED; ALTER WAREHOUSE ETL_WH SET WAREHOUSE_SIZE = MEDIUM;"
  }]}
}

After (custom scale-up with role switch):

{ "name": "SFLSQL - SCALE UP",
  "type": "Script",
  "typeProperties": { "scripts": [{
    "text": "USE ROLE SYSADMIN; ALTER WAREHOUSE ETL_WH RESUME IF SUSPENDED; ALTER WAREHOUSE ETL_WH SET WAREHOUSE_SIZE = XLARGE;"
  }]}
}

Common Mistakes

1. Mistake: Not setting CustomOutput.OutputPathName. Symptom: Downstream activities lose their dependency chain and execute in parallel or fail. Fix: Always set CustomOutput.OutputPathName = outputPathName; at the end.
2. Mistake: Hardcoding the warehouse name instead of using pipeline parameters. Symptom: The scale-up targets the wrong warehouse when deployed to a different environment. Fix: Use pipeline().parameters.SnowflakeWarehouse for environment portability.
3. Mistake: Setting an aggressive timeout (e.g., 30 seconds) on the Script Activity. Symptom: The ALTER WAREHOUSE RESUME times out if the warehouse takes time to provision. Fix: Use a timeout of at least 0.00:05:00.
4. Mistake: Forgetting to pair with AdfSnowflakeScaleDown. Symptom: The warehouse stays at the larger size indefinitely, consuming excess Snowflake credits. Fix: Always implement both scale-up and scale-down together.

Related Extension Points

• AdfSnowflakeScaleDown -- scales the warehouse back down after processing completes.
• AdfSnowflakeCreateStage -- creates the external stage used for data loading into Snowflake.
• AdfBatchPreProcess -- alternative location for pre-batch logic that is not Snowflake-specific.
• AdfBatchPostProcess -- post-batch logic, often used alongside scale-down.

Batch Trigger

Configure a Batch Trigger in Data Factory. This can be linked to a Batch Pipeline process.

Parameters

Name	Type	Description
batch	BimlFlexModelWrapper.BatchesWrapper	Contains all information related to the Batch to which the Trigger will be added.

Outputs

Name	Type	Description
ObjectInherit	Boolean	If CustomOutput.ObjectInherit = true then the Extension Point will be applied to all the Objects associated with the Batch.

Template

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchTrigger" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>



<!-- The below example adds a Schedule Trigger and connects it to a Batch Pipeline in Data Factory. -->

<Schedule Name="ScheduleTriggerName" Frequency="Hour" Interval="1" Start="2021-01-01" End="2025-12-31">
	<Pipelines>
		<Pipeline PipelineName="01_<#=batch.Name #>_Batch">
			<Parameters>
				<Parameter Name="IsInitialLoad">false</Parameter>
			</Parameters>
		</Pipeline>
	</Pipelines>
</Schedule>

What It Does

Attaches a scheduling trigger directly to a specific batch pipeline in Azure Data Factory. Unlike the factory-level AdfTrigger extension point, this extension point scopes the trigger to a single batch, giving you precise control over when each batch orchestration fires. The trigger definition is emitted alongside the batch pipeline ARM template so that deployments include the schedule automatically.

Where It Fires

Registered in bundle.common.manifest and consumed via ADF generation code in Code/DataFactoryHelper.cs. The extension point is evaluated once per batch during output generation for the targeted data factory. It emits a Trigger JSON resource linked to the batch pipeline by name.

When to Use It

1. You need the 01_EXT_AWLT_SRC_Batch pipeline to run on a recurring hourly schedule without a separate factory-wide trigger.
2. Different batches require different schedules -- for example, 01_EXT_AWLT_SRC_Batch runs every 15 minutes while 03_LOAD_BFX_DM_Batch runs daily at 02:00.
3. You want a tumbling window trigger for a specific batch to support reprocessing of missed intervals.
4. You need an event-based trigger (e.g., blob arrival) tied to a single batch rather than the entire factory. Use this instead of AdfTrigger when the trigger should only apply to one batch pipeline rather than being a factory-level resource.

Prerequisites

• A batch must be defined in BimlFlex metadata (e.g., 01_EXT_AWLT_SRC_Batch).
• The Data Factory connection (adf-bimlflex-prod) must be configured.
• If the trigger passes parameters such as IsInitialLoad, the batch pipeline must declare those parameters via AdfBatchParameter.

Implementation Steps

1. Open the BimlFlex application and navigate to the Extension Points editor.
2. Create a new extension point and set the target to the desired batch (e.g., 01_EXT_AWLT_SRC_Batch).
3. Add the BimlScript directive and define the trigger:

<#@ extension bundle="BimlFlex.bimlb" extensionpoint="AdfBatchTrigger" #>
<#@ property name="batch" type="BimlFlexModelWrapper.BatchesWrapper" #>

<Schedule Name="Hourly_<#=batch.Name#>" Frequency="Hour" Interval="1"
         Start="2025-01-01" End="2027-12-31">
  <Pipelines>
    <Pipeline PipelineName="01_<#=batch.Name#>_Batch">
      <Parameters>
        <Parameter Name="IsInitialLoad">false</Parameter>
      </Parameters>
    </Pipeline>
  </Pipelines>
</Schedule>

4. Build the project in BimlStudio and verify the trigger appears in the ARM template output under Microsoft.DataFactory/factories/triggers.
5. Deploy to adf-bimlflex-prod and confirm the trigger is visible and started in the ADF portal.

Example

Before (no trigger -- batch must be started manually or by a parent pipeline):

{
  "name": "01_EXT_AWLT_SRC_Batch",
  "type": "Microsoft.DataFactory/factories/pipelines",
  "properties": { "activities": [ ... ] }
}

After (trigger attached to the batch pipeline):

{
  "name": "Hourly_01_EXT_AWLT_SRC_Batch",
  "type": "Microsoft.DataFactory/factories/triggers",
  "properties": {
    "type": "ScheduleTrigger",
    "typeProperties": {
      "recurrence": { "frequency": "Hour", "interval": 1 }
    },
    "pipelines": [{
      "pipelineReference": { "referenceName": "01_EXT_AWLT_SRC_Batch" },
      "parameters": { "IsInitialLoad": false }
    }]
  }
}

Common Mistakes

1. Mistake: Using AdfTrigger when only one batch needs the schedule. Symptom: The trigger appears as a factory-level resource disconnected from the batch. Fix: Use AdfBatchTrigger and target the specific batch.
2. Mistake: Setting PipelineName to a value that does not match the generated batch pipeline name. Symptom: Deployment succeeds but the trigger references a non-existent pipeline, causing runtime failures. Fix: Use the <#=batch.Name#> property with the correct naming convention (e.g., 01_<#=batch.Name#>_Batch).
3. Mistake: Overlapping trigger schedules across batches that share dependencies. Symptom: Concurrent batch runs cause deadlocks or duplicate data loads. Fix: Stagger trigger start times or use WaitOnCompletion in a parent orchestration pipeline.
4. Mistake: Forgetting to declare pipeline parameters referenced in the trigger. Symptom: ADF deployment validation fails with "parameter not found" errors. Fix: Add the parameters using AdfBatchParameter before referencing them in the trigger.

Related Extension Points

• AdfTrigger -- factory-level triggers that are not scoped to a single batch.
• AdfBatchParameter -- defines parameters on the batch pipeline that the trigger can pass values to.
• AdfBatchVariable -- defines variables on the batch pipeline for use during execution.
• AdfBatchPreProcess / AdfBatchPostProcess -- add activities before or after the batch orchestration logic.