Account
Skip to main content

Deploying Generated Output Through Fabric Git Integration

BimlFlex writes the generated notebooks and pipelines for a Microsoft Fabric project into a folder tree, the Fabric root, under the FabricOutputPath setting. When that tree is pushed to a git repository connected to a Fabric workspace and the workspace is updated from git, Fabric git integration creates the items in the workspace.

This page describes what the generated output must carry from metadata for that update to succeed, and what git integration does with the output when it imports it. The build, push, sync and verify steps are described in the Deployment Workflow section of the Fabric Pipeline Configuration Guide.

Bind Connection Ids Before the Build​

Every pipeline activity that reads from or writes to a relational connection, such as a SQL Server source, carries a Fabric connection id in its externalReferences.connection value. BimlFlex takes this id from the connection's External Reference field in metadata. The External Reference section of the configuration overview describes where the id is found in the Fabric portal and that it is entered for each connection.

The External Reference field holds a different id depending on the kind of connection:

Connection kindValue of External Reference
Fabric Lakehouse or Fabric Warehouse connectionThe id of the Lakehouse or Warehouse item in the workspace
Relational connection reached through a Fabric connectionThe id of the Fabric connection for that system

Bind every relational connection's External Reference before a build that is meant for git integration. The binding contract is:

  • A blank External Reference on a Fabric Lakehouse, Warehouse or SQL Database connection, or on a connection that a Fabric project uses as its landing, stage, persistent stage or target connection, is a validation error and the build stops. A blank External Reference on the source connection of a Fabric project, or on the BimlCatalog connection, is the same validation error. The build reports the error with the code CON_21005008, the name of the connection and the statement that a connection configured for Fabric must use External Reference. Enter the id in the External Reference field of the connection that the message names, then build again.
  • Git integration checks every connection id in the pushed pipeline files against the connections the user who performs the workspace update can reach. An id that names no connection fails the connection check, and the update fails with a message that says the user does not have access to the connection used in the pipeline.
  • The update does not undo the items it has already created when a pipeline fails the connection check. Notebooks carry no connection id, so a workspace whose update fails at the connection check can hold every notebook and no pipeline.
  • A Fabric root whose relational connections all carry a bound External Reference imports completely.

Give the user who performs the update access to the connection each bound id names. Git integration reports an id that names no connection with a message about the user's access to the connection, so when the connection check fails, confirm both the id and the access of the user who performs the update.

External Reference holds one value per connection, so one build carries one set of connection ids. A workspace whose connections have different ids takes a build with those ids bound.

A Fabric root that a build has written with unbound connections can be bound for the push with the bind-connections command of the Fabric deployment tool. The command writes a bound copy of the Fabric root and leaves the Fabric root as the build wrote it. The bind-connections command section of the Fabric Deployment Tool page describes the command.

What Git Integration Does With the Generated Output​

When the workspace is updated from git, the items in the Fabric root are resolved as follows.

Notebook activities. A notebook activity in a generated pipeline refers to the notebook by its logical id, the logicalId value in the notebook's .platform file described in the .platform File Format section, and carries the all zeros workspace id. On import, git integration stores the workspace item id of the notebook and the id of the workspace in their place. When Fabric commits the workspace back to the repository, it writes the logical id and all zeros form again, so the generated form is the form the repository holds.

Pipeline references. An Execute Pipeline activity whose reference carries the child pipeline's logical id stores the child pipeline's workspace item id after the update, with the reference type kept as a pipeline reference.

Folders. The nested folders of the Fabric root become folders in the workspace.

Removal. Removing items from the repository and updating the workspace deletes those items from the workspace. The workspace folders are kept.

Before You Push​

  1. Set the External Reference of every relational connection to the id of the Fabric connection it maps to.
  2. Confirm that the user who will update the workspace from git has access to each of those connections.
  3. Build the project, then push the Fabric root to the repository connected to the workspace.
  4. Update the workspace from git and verify that the notebooks and pipelines are present.