Upgrading to BimlFlex 2027

This article covers the changes in BimlFlex 2027 that can affect a project built with an earlier version of BimlFlex. Each section starts with who is affected and what to do, followed by the detail. For everything else that is new, see the BimlFlex 2027 release notes. For the general steps of an upgrade, see Upgrade Process.

We recommend testing the upgrade in a non-production environment first, and backing up your databases and projects before you upgrade.

Which sections apply to you

To list the places in your metadata that these sections describe, you can run the optional upgrade check script before you upgrade.

Section	Read it if
Scoped overrides apply in more places	Your metadata has setting or configuration overrides on a batch, setting overrides on a parent project, or extension point targets on a connection or a project.
Retired connectors	A connection uses the Couchbase, Drill, Oracle (Legacy) or Salesforce Marketing Cloud connector type, or an extension point has hand written Biml for a connector that the Biml language no longer has.
Snowflake connection fields	You have connections whose Linked Service Type is Snowflake, and you upgrade from a BimlFlex 2026 R1 build earlier than 26.1.399 or from an older release.
SSIS casts to DT_NUMERIC	You build SSIS packages and an expression casts a value to DT_NUMERIC.
SSIS package and project identifiers	You keep generated SSIS output in source control.
Delete detection and batch extension points	You use delete detection with Azure Data Factory and have extension points that target a load batch.
Watermark comparison in incremental loads	You have Azure Data Factory or Microsoft Fabric incremental loads with a watermark parameter.
Google dataset names	You deploy Google BigQuery V2 or Google Ads connections to Azure Data Factory.

Upgrade check script

The BimlFlex 2027 upgrade check script is an optional T-SQL script that lists the places in your metadata that the sections of this article describe. The script only reads. It changes nothing in the database, and a login that has only the db_datareader role can run it.

Run the script before you upgrade. Open it in SQL Server Management Studio, connect to your BimlFlex 2026 R1 metadata database and run it. To run it with sqlcmd instead, use the command in the comment at the top of the script. The script returns one list, which you can copy into a spreadsheet with Copy with Headers.

Each row of the list has the columns Topic, Kind, Certainty, EntityType, EntityName, Found, Consider and ArticleSection. EntityName is the name that the BimlFlex app shows. Found says what the script found, Consider says what to consider doing, and ArticleSection links to the section of this article that explains the change.

Column	Value	Meaning
Kind	Action required	BimlFlex 2027, or a pipeline it generates, fails for this metadata until you change it.
Kind	Review	BimlFlex 2027 can generate different output for this metadata. Confirm that the result is what you want.
Kind	Information	There is nothing to change before you upgrade. The row tells you what to expect after the upgrade.
Certainty	Finding	The metadata decides it, and the row is affected.
Certainty	Pointer	The metadata can only point at a place to look. You decide whether the row is affected.

The script checks these sections:

• Scoped overrides apply in more places: setting and configuration overrides on a batch, setting overrides on a parent project, and extension point targets on a connection or a project. The rows for setting overrides on a parent project are pointers, because the objects of a child project already used the override in BimlFlex 2026 R1, and only a setting that BimlFlex reads for the project itself changes. The BimlFlex 2027 app can mark more rows with warning ATT_21005007 than the script lists, because the script leaves out an override with a blank value, an override on metadata that is deleted or excluded or that the build does not load, an override whose key contains a configuration key, an override on a batch for a setting that BimlFlex 2027 does not read for a batch, an override of a Snowflake connection setting, such as SnowflakeAccount or SnowflakeWarehouse, on a batch that no project that builds SSIS packages for a Snowflake target uses, and a setting override on a parent project when every child project has its own value.
• Retired connectors: connections of the Salesforce Marketing Cloud, Couchbase, Drill and Oracle (Legacy) types, which BimlFlex 2026 R1 could build, and extension points whose Biml contains an element that the Biml language no longer has.
• SSIS casts to DT_NUMERIC: text that contains DT_NUMERIC in projects that build SSIS packages. These rows are pointers, because the script cannot tell whether a cast gives a different result.
• Delete detection and batch extension points: extension points that target a batch whose objects use delete detection with Azure Data Factory. These rows are pointers.
• Watermark comparison in incremental loads: the watermark parameters of type DateTime, Decimal, Double or Single that the generated pipeline compares, and the defaults of those parameters that SQL Server cannot read as a timestamp or a number. A row about a default is a pointer, because the pipeline uses the default only on its first run, before a watermark value is stored.
• Google dataset names: Google BigQuery V2 and Google Ads connections in Azure Data Factory and Databricks (ADF) projects, and extension points that contain an earlier dataset name.

The script cannot check these:

• Snowflake connection fields: the database upgrade fills these fields, so check the two cases in that section by hand.
• SSIS package and project identifiers: there is nothing to check, because the change needs no action.
• The values that the pipelines compare at run time, in Watermark comparison in incremental loads: they are not in the metadata database, so the script lists the watermark parameters and checks only their defaults.
• BimlStudio and BimlExpress projects: their Biml files are not in the metadata database.

The script reads the extension points that are stored in the metadata database. It does not read extension point files that you keep in your BimlStudio project, so check those files by hand against the sections of this article.

An empty list is not proof that nothing changes for the topics that the script cannot check.

Scoped setting and extension point overrides apply in more places

This section applies to you if your metadata has setting or configuration overrides attached to a batch, setting overrides on a project that other projects use as their parent, or extension points targeted only at a connection or only at a project. Before your first build with BimlFlex 2027, review these overrides and confirm that each one is intended.

BimlFlex 2026 R1 skipped some reads of the kinds of override listed below at build time. BimlFlex 2027 performs those reads, so your first build after upgrading applies each of these overrides wherever BimlFlex 2026 R1 skipped it, and this can change generated output.

• Setting and configuration overrides on a batch, for the settings that the build reads for a batch. These are SettingValue rows and {ConfigurationKey}_ConfigurationValue or {ConfigurationKey}ConfigurationValue rows attached to a batch. BimlFlex 2026 R1 did not read these overrides at all. The settings include AppendBatchName, so the names of generated packages can change. They also include the Snowflake scale and auto suspend settings. BimlFlex reads these for a batch of a project with a Snowflake target, both for SSIS packages and for Azure Data Factory batch pipelines, so a batch override can now change how such a batch scales or suspends the warehouse. For a batch of a project that builds SSIS packages for a Snowflake target, they also include the Snowflake connection settings, such as account, warehouse and database, so a batch override can now repoint the Snowflake connection of those packages, unless your build uses the SSIS project deployment model.
• Extension points targeted at a connection or a project. These are ExtensionPointTarget rows attached only to a connection or only to a project. BimlFlex 2026 R1 read these targets for the objects of the connection or project, but not for the connection or project itself. BimlFlex 2027 also reads them for the extension points of the connection or project itself, and the custom Biml of those extension points is added to the generated output. This happens only for an extension point that has no file naming the connection or project itself. Where such a file exists, BimlFlex uses that file as it did in BimlFlex 2026 R1 and the targets add nothing.
• Settings on a parent project. BimlFlex 2026 R1 read a SettingValue override entered on a parent project for that project, but its child projects did not inherit it. In BimlFlex 2027, child projects inherit the override. A blank value on a child project does not stop it inheriting the override, so a child project must carry the value you want.

Review your overrides

1. Before you build with BimlFlex 2027, run the audit query below against your BimlFlex database.
2. Confirm that each override the query lists is intended, and keep it if it is. If it is not, remove it. Removing an override removes it everywhere, including where BimlFlex 2026 R1 already applied it.
3. For an extension point target on a connection or a project, check which extension point files name the target.
4. For a setting on a parent project, set the value you want on each child project that should keep its earlier output.

After you upgrade, the BimlFlex 2027 app also marks these overrides with warning ATT_21005007: "This release applies this override in more places than BimlFlex 2026 R1 did, so generated output can differ from your last build before the upgrade. Review it before you build."

For more about targeting extension points at a scope, see Extension Points.

Audit query

The query lists the rows that warning ATT_21005007 marks. It selects each row by its attribute type and by the batch, project or connection that the row names for that type. Its key comparisons are case-sensitive to match how the build compares keys.

For an override on a batch, the query lists only the settings that the build reads for a batch, such as AppendBatchName and the Snowflake connection settings. A batch override of any other setting still has no effect after the upgrade.

The upgrade check script can list fewer rows than the query, for the reasons given in that section.

DECLARE @zero UNIQUEIDENTIFIER = '00000000-0000-0000-0000-000000000000';

-- The setting keys that the BimlFlex 2027 build reads for a batch
DECLARE @BatchKeys TABLE ([KeyName] NVARCHAR(256) COLLATE Latin1_General_BIN2 NOT NULL PRIMARY KEY);

INSERT INTO @BatchKeys ([KeyName])
VALUES (N'AppendBatchName'), (N'AppendLoadFromPsaName'), (N'DatabricksUtilsDriver'), (N'DeleteDetectionAppendName'),
       (N'DvAppendBridge'), (N'DvAppendPit'), (N'ExtensionPointTarget'), (N'SnowflakeAccount'),
       (N'SnowflakeAutoSuspend'), (N'SnowflakeDatabase'), (N'SnowflakePassword'), (N'SnowflakeRegion'),
       (N'SnowflakeScaleDown'), (N'SnowflakeScaleDownSize'), (N'SnowflakeScaleUp'), (N'SnowflakeScaleUpSize'),
       (N'SnowflakeSchema'), (N'SnowflakeUser'), (N'SnowflakeWarehouse');

SELECT ca.[UID],
       ca.[AttributeKey],
       ca.[AttributeValue],
       ca.[AttributeProperty],
       ca.[BatchUID],
       ca.[ProjectUID],
       ca.[ConnectionUID],
       [Shape] = CASE
         WHEN atyp.[Code] = N'BAT'
              AND ca.[AttributeKey] COLLATE Latin1_General_BIN2 = N'SettingValue'
           THEN 'Batch-scoped setting override'
         WHEN atyp.[Code] = N'BAT'
           THEN 'Batch-scoped configuration override'
         WHEN ca.[AttributeKey] COLLATE Latin1_General_BIN2 = N'SettingValue'
           THEN 'Parent-project setting override (children will inherit)'
         WHEN atyp.[Code] = N'CON'
           THEN 'Connection-scoped extension point'
         ELSE 'Project-scoped extension point'
       END
FROM [app].[CustomAttributes] ca
INNER JOIN [meta].[AttributeType] atyp
    ON atyp.[Id] = ca.[AttributeTypeId]
CROSS APPLY (SELECT DATALENGTH(ca.[AttributeKey]) / 2 AS [KeyLen]) kl
-- The setting key of a '{Key}_ConfigurationValue' or '{Key}ConfigurationValue' override
CROSS APPLY (SELECT CASE
                      WHEN kl.[KeyLen] <= 18 THEN N''
                      WHEN ca.[AttributeKey] COLLATE Latin1_General_BIN2 LIKE N'%[_]ConfigurationValue'
                        THEN LEFT(ca.[AttributeKey], kl.[KeyLen] - 19)
                      ELSE LEFT(ca.[AttributeKey], kl.[KeyLen] - 18)
                    END AS [TargetKey]) tk
WHERE ISNULL(ca.[IsDeleted], 0) = 0
  AND (
    -- Batch row: setting override of a setting that the build reads for a batch
    (atyp.[Code] = N'BAT'
      AND ca.[BatchUID] IS NOT NULL AND ca.[BatchUID] <> @zero
      AND ca.[AttributeKey] COLLATE Latin1_General_BIN2 = N'SettingValue'
      AND ca.[AttributeValue] COLLATE Latin1_General_BIN2 IN (SELECT [KeyName] FROM @BatchKeys))
    OR
    -- Batch row: configuration override of a setting that the build reads for a batch,
    -- both key conventions: '{Key}_ConfigurationValue' and '{Key}ConfigurationValue'
    (atyp.[Code] = N'BAT'
      AND ca.[BatchUID] IS NOT NULL AND ca.[BatchUID] <> @zero
      AND kl.[KeyLen] > 18
      AND ca.[AttributeKey] COLLATE Latin1_General_BIN2 LIKE N'%ConfigurationValue'
      AND tk.[TargetKey] <> N''
      AND tk.[TargetKey] COLLATE Latin1_General_BIN2 IN (SELECT [KeyName] FROM @BatchKeys))
    OR
    -- Project row: setting override on a project that another project names as its parent
    (atyp.[Code] = N'PRJ'
      AND ca.[ProjectUID] IS NOT NULL AND ca.[ProjectUID] <> @zero
      AND ca.[AttributeKey] COLLATE Latin1_General_BIN2 = N'SettingValue'
      AND EXISTS (SELECT 1
                  FROM [app].[Projects] p
                  WHERE ISNULL(p.[IsDeleted], 0) = 0
                    AND p.[ParentProjectUID] = ca.[ProjectUID]
                    AND p.[UID] <> ca.[ProjectUID]))
    OR
    -- Connection row: extension point target
    (atyp.[Code] = N'CON'
      AND ca.[ConnectionUID] IS NOT NULL AND ca.[ConnectionUID] <> @zero
      AND ca.[AttributeKey] COLLATE Latin1_General_BIN2 = N'ExtensionPointTarget')
    OR
    -- Project row: extension point target
    (atyp.[Code] = N'PRJ'
      AND ca.[ProjectUID] IS NOT NULL AND ca.[ProjectUID] <> @zero
      AND ca.[AttributeKey] COLLATE Latin1_General_BIN2 = N'ExtensionPointTarget')
  );

Retired connectors

This section applies to you if a connection in your metadata uses one of the connector types in the table below. Before you build with BimlFlex 2027, migrate each affected connection to a supported connector or remove it from your metadata. For an Oracle (Legacy) connection, migrate the connection to the Oracle connector (2.0).

Microsoft has removed the connector types in the table from the Azure Data Factory service. The Azure Data Factory service retired Couchbase, Drill and Salesforce Marketing Cloud on 31 December 2024, and they can no longer be deployed. Oracle (Legacy) is version 1.0 of the Oracle connector. The Azure Data Factory service removed that version on 31 March 2026 and still supports the Oracle connector in version 2.0.

BimlFlex 2027 does not offer these connector types for new connections and no longer generates them for Azure Data Factory. A connection in existing metadata that still uses one of them produces a validation error that identifies the retired connector, and the build keeps reporting that error until the connection is migrated or removed.

Connector type	Retired by Azure Data Factory	Validation error	What to do
Couchbase	31 December 2024	CON_21005013	Migrate to a supported connector or remove
Drill	31 December 2024	CON_21005013	Migrate to a supported connector or remove
Salesforce Marketing Cloud	31 December 2024	CON_21005013	Migrate to a supported connector or remove
Oracle (Legacy), version 1.0	31 March 2026	CON_21005014	Migrate to the Oracle connector (2.0)

Hand written Biml in an extension point that uses an element removed from the Biml language for a retired connector or an earlier connector version gets a Biml schema error. For the lists, see Retired Azure Data Factory connectors in Upgrading to BimlStudio 2027.

Snowflake connection fields

This section applies to you if you have connections whose Linked Service Type is Snowflake, and you upgrade from a BimlFlex 2026 R1 build earlier than 26.1.399 or from an older release. If you are already on BimlFlex 2026 R1 build 26.1.399 or later, your connections have been through this change. After the upgrade, open each Snowflake connection whose connection string is an Azure Key Vault secret, or whose connection string has no account, warehouse or database, and enter the missing values before you build.

A Snowflake connection keeps its account, warehouse, database and role in separate fields of the connection: Account, Warehouse, Database Name and Role Name. BimlFlex builds the Snowflake linked service from these fields.

When you upgrade the BimlFlex database, the upgrade reads the account, region, warehouse, db and role values from the connection string of each Snowflake connection and enters them in the fields that are empty. The account and region values are combined into Account. A field that already has a value is not changed.

Two cases leave a required field empty, and the build fails until you enter the value by hand:

• The connection string of the connection is stored as an Azure Key Vault secret. The upgrade cannot read it, so it fills none of the fields.
• The connection string has no account, warehouse or db value. The matching field stays empty.

To fix a connection, open it in the connection editor of the BimlFlex app and enter Account, Warehouse and Database Name. Role Name is optional.

Until you do, the build reports an error for each empty field, in the form Required property 'Account' was not supplied for '...'., with Database or Warehouse in place of Account.

SSIS casts to DT_NUMERIC

This section applies to you if you build SSIS packages with BimlFlex and any SSIS variable or property expression casts a value to DT_NUMERIC. Review those expressions before you build with BimlFlex 2027.

The Biml compiler that builds BimlFlex SSIS output now evaluates two forms of DT_NUMERIC cast differently from earlier versions:

• When the value has more whole digits than the declared precision minus the scale allows, the cast reports a truncation, and the truncation stops the build.
• The cast truncates the value to the declared scale without rounding. For example, (DT_NUMERIC,10,2)12.345 evaluates to 12.34, where earlier versions give 12.3.

SSIS package and project identifiers

This section applies to you if you keep generated SSIS output in source control. You do not need to change anything, and a project that is already deployed keeps running as it is.

On the first build after the upgrade, some identifiers in your generated SSIS packages and projects take new values. These identifiers kept the same values from build to build in BimlFlex 2026 R1, and they keep their new values from build to build in BimlFlex 2027, so your source control shows this change once. What the packages do does not change.

The identifiers that take a new value are the ones BimlFlex works out from your metadata:

• some of the containers, tasks and data flows in your load packages and batch packages, and some of the precedence constraints that connect them
• the package ID of each batch package that runs your load packages
• the blob connections and the satellite lookup cache connections
• the columns of the satellite lookup cache connections
• the variables that hold parameter values in your load packages, such as LastLoadDate
• the script project that end dates satellite rows
• the project ID of each SSIS project

Identifiers that keep the values they had in BimlFlex 2026 R1 include the package ID and version GUID of each load package, the main containers and the extract data flow of each source to staging package, the connection managers of your connections in your load packages and SSIS projects, and the columns of your file formats.

Delete detection and batch extension points

This section applies to you if you use delete detection with Azure Data Factory and have extension points that target a load batch. If you want an extension point to apply to the delete detection batch pipeline, target that pipeline by its own name.

In BimlFlex 2027, a generated Data Factory delete detection batch pipeline no longer takes on the variables, parameters, pre and post process activities, and AdfSnowflakeScaleUp activities of an extension point that targets the load batch, either by the batch name or through an ExtensionPointTarget attribute. Generated SSIS packages are unchanged.

For more information, see Delete Detection and Extension Points.

Watermark comparison in incremental loads

This section applies to you if you have Azure Data Factory or Microsoft Fabric incremental load pipelines with a watermark parameter. If a watermark parameter is of type DateTime, Decimal, Double or Single, check before you regenerate those pipelines that its default is a valid value, as described below.

Generated incremental load pipelines for Azure Data Factory decide whether to run the copy by comparing the To watermark with the From watermark. In BimlFlex 2027, for a source without change tracking, the declared data type of the watermark parameter decides how the two values are compared. A date or decimal watermark is now ordered by its value rather than by its text. Pipelines generated for Microsoft Fabric use the same comparison.

Watermark parameter type	Compared as
Byte, SByte, Int16, Int32, Int64	Integers, through int()
Decimal, Double, Single	Floating point numbers, through float()
DateTime	Ticks, through ticks()
String, Char, any other type, or no declared type	Text

Existing pipelines

A deployed pipeline keeps its existing watermark expression, so the new comparison only takes effect when you regenerate the pipeline. A watermark parameter of an integer type, String, or no declared type generates the same expression as before.

Watermark defaults

For a DateTime, Decimal, Double or Single watermark, the generated expression passes both watermark values through float() or ticks() before it compares them. These functions only accept a string that holds a valid floating point number or a timestamp, so every value in the From and To watermark variables must be a value that the function accepts.

• A DateTime watermark whose parameter default is blank or 0 starts its first run from 1900-01-01, so that default works as a timestamp.
• Any other DateTime default must be a timestamp.
• A Decimal, Double or Single watermark whose parameter default is blank starts its first run from 0, so that default works as a number.
• Any other Decimal, Double or Single default must be a valid number, because the pipeline uses the default as it is entered.

Correct any parameter default that does not meet these rules before you regenerate the pipeline. A decimal beyond the precision of a double is compared as a double.

For more about parameters that define a high water mark, see Load Parameters.

Google dataset names

This section applies to you if you deploy Google BigQuery V2 or Google Ads (Google AdWords) connections to Azure Data Factory. After your next deploy, the datasets with the earlier names stay in your factory until you delete them by hand.

The Azure Data Factory datasets that BimlFlex generates for these connections have new names.

Connection	Earlier dataset name	BimlFlex 2027 dataset name
Google BigQuery V2	GBQV2_<connection>	GBQ2_<connection>
Google Ads	GWADS_<connection>	GADS_<connection>

On the next deploy to an existing factory, the new dataset is added next to the old one. The old dataset is no longer used, and it stays in the factory until you delete it by hand.

• A copy of the sample extension point GoogleBigQueryV2Dataset.biml or GoogleAdWordsDataset.biml names its dataset from the connection type, so it produces the new dataset name without a change.
• Metadata, the connection editor and metadata import are unchanged.

For connection setup, see Configuring an ADF Linked Service Connection for Google BigQuery V2 and Configuring an ADF Linked Service Connection for Google AdWords.

Getting help

For installation or upgrade assistance, please email [email protected] with detailed information about your environment and any issues you encounter.