Deploying Generated Output With the Fabric Deployment Tool

The Fabric deployment tool, bimlflex-fabric-deploy, is a command line tool that deploys the Fabric output of a BimlFlex build to one Fabric workspace through the Fabric REST API. It reads the Fabric root, the folder under the FabricOutputPath setting that holds the .Notebook and .DataPipeline item folders and the manifests, and creates or updates the items of the workspace.

The tool deploys the notebooks first, then the data pipelines, children before parents. Each item is keyed by its display name: it is created when the workspace holds no item of that name and updated when it holds one. A notebook reference by display name receives the item id and the workspace id in the definition the tool sends, never in the files of the Fabric root.

Choosing a Deployment Path

BimlFlex output reaches a Fabric workspace by one of two paths:

• The Fabric deployment tool deploys the Fabric root to the workspace through the Fabric REST API. It needs no git repository.
• Fabric git integration imports the Fabric root from a git repository connected to the workspace. The Fabric Git Integration page describes that path.

The deploy command skips a data pipeline with a connection site that carries the all zeros id, together with every parent pipeline that calls it, and its siblings still deploy. Bind the External Reference of every relational connection before the build, as described in Bind Connection Ids Before the Build. A deploy with --connection-credentials <file> also adopts or creates each connection the build describes with a provisioning payload and no connection id, other than a Lakehouse, Warehouse or KQL database row, and writes its id at the all zeros connection sites of that connection in the definition it sends, as described in Connections.

Installing the Tool

The tool is installed with the BimlFlex app. Both BimlFlex app installers place bimlflex-fabric-deploy.exe, its dll and its runtime files in the app folder beside BimlFlex.exe.

The same command line is available as a dotnet tool. The package id is Varigence.BimlFlex.FabricDeploy, and the package is published to NuGet.org at each stable release. The dotnet tool form needs the .NET 10 runtime.

dotnet tool install --global Varigence.BimlFlex.FabricDeploy

The --tool-path <folder> form of dotnet tool install places the tool in a folder of your choosing instead of the global tool location. dotnet tool list prints the version of the installed package.

Permissions of the Deploying Identity

The identity that deploys holds the contributor role on the workspace.

• A person who signs in with interactive or device-code consents to the two delegated permissions of the Fabric API that the tool asks for, Workspace.Read.All and Item.ReadWrite.All.
• A service principal calls the Fabric API only when the tenant setting that allows service principals to use Fabric APIs is on for it. A Fabric administrator sets it.
• A run that provisions connections lists and creates them with the Connection.ReadWrite.All permission of the Fabric API. A person's sign in does not ask for it, so a person provisions connections with a token of their own under --credential token, or a service principal provisions them.

The Deploy Command

The deploy command takes the Fabric root, the id of the target workspace and a credential mode:

bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --credential service-principal --tenant-id <id> --client-id <id> --client-secret-stdin
bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --credential service-principal --tenant-id <id> --client-id <id> --client-secret-env <NAME>
bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --credential token --token-stdin
bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --credential interactive
bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --credential device-code --tenant-id <id>
bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --credential token --token-stdin --prune
bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --credential token --token-stdin --connection-credentials <file>
bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --dry-run

Option	Meaning
--fabric-root <folder>	The Fabric folder of a build output. Required.
--workspace-id <id>	The id of the target workspace. Required. It is an argument and is never read from the output.
--credential <mode>	service-principal, token, interactive or device-code. A dry run needs none.
--tenant-id <id>	The tenant of the sign in. Required for service-principal. Optional for interactive and device-code, where the default is organizations, the work and school accounts of any tenant.
--client-id <id>	The client id of the service principal. interactive and device-code take none.
--client-secret-stdin	Read the client secret from the first line of standard input.
--client-secret-env <NAME>	Read the client secret from the environment variable of this name. The argument is the name, never the value.
--token-stdin	Read a token the caller acquires from the first line of standard input.
--fabric-api <address>	The address of the Fabric API. The default is https://api.fabric.microsoft.com/. The service principal scope is this address plus /.default, and the delegated scopes of a person are this address plus each permission name.
--time-bound <minutes>	The time bound of the run in minutes, measured from the start of the deploy, before the sign in and the first call. The default is 60. The run ends with exit code 4 when a call or a wait passes it.
--report <path>	The report file, written as JSON at the end of the run. Never under the Fabric root.
--dry-run	Build the plan from the files, print it and make no call.
--verbose	Write the call log to standard error.
--referenced-notebooks-only	Deploy only the notebooks a pipeline activity references.
--prune	After the deploy, remove the workspace items the deploy record proves this tool created and the current build no longer emits, each one reported before the first delete call. Absent, nothing is deleted.
--connection-credentials <file>	The connection credentials file, read before any call. The deploy adopts or creates each connection the build describes with a provisioning payload and no connection id, other than a Lakehouse, Warehouse or KQL database row, as described in Connections. Never under the Fabric root.
--deploy-record <path>	The deploy record, the JSON file of the item ids this tool creates or updates in the target workspace. The default is deploy-record.<workspace id>.json in the current directory. A relative path is resolved against the current directory, and the path is never under the Fabric root.

Credential Modes

Mode	Who signs in	Where the secret or the token comes from
service-principal	An application, with --tenant-id and --client-id.	The client secret from the first line of standard input under --client-secret-stdin, or from the environment variable that --client-secret-env <NAME> names.
token	Whoever acquired the token, with a tool of their own.	The token from the first line of standard input under --token-stdin.
interactive	A person, through the system browser. The sign in page is the Varigence application, the same registration the SQL sign in of BimlFlex uses, so the tool takes no client id. --tenant-id is optional.	Nothing is given to the tool. The browser returns the token to the tool, which holds it in memory for the run and refreshes it through the same sign in as it nears expiry.
device-code	A person, from a device that has a browser, for a host that has none. The tool prints an address and a code on standard error, waits for the sign in and then proceeds as interactive.	Nothing is given to the tool.

interactive and device-code sign a person in. service-principal signs an application in, and token sends a token acquired with another tool, so neither of those two needs a person at a browser.

Secrets

The tool applies these rules to every secret and token:

• A secret is never an argument value. --client-secret <value> and --token <value> are refused with the admitted forms named, since the process list and the shell history carry every argument.
• A client secret is read from the first line of standard input with --client-secret-stdin, or from a named environment variable with --client-secret-env <NAME>. A token is read from the first line of standard input with --token-stdin. --client-secret-stdin and --token-stdin take no value.
• An argument that belongs to no option is counted and never printed back, and neither is the value of --client-secret-env.
• The one file the tool reads a secret from is the connection credentials file, and it reads the credentials of the connections there, never the sign in secret or the token.
• --client-id with interactive or device-code is refused, since the registration is fixed in the tool.
• A person signs in once per run. The tool keeps no token cache on disk, so the next run signs in again.
• interactive on a host that cannot open a browser stops with a message that names device-code.

The Dry Run

--dry-run builds the plan from the files and prints it: the counts, one line per notebook, one line per data pipeline in the order the deploy sends them with the children it calls and any skip, the unresolved references and the warnings as JSON rows, one wouldProvision connection row for each connection a live run adopts or creates, and any call graph cycle. The dry run makes no call, needs no credential and writes no report file. It reads the connection credentials file when --connection-credentials is given, so a defect in the file stops the dry run as it stops a live run.

bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --dry-run

The Deploy Record and the Prune

A deploy deletes nothing unless --prune is given.

The tool keeps a deploy record, the JSON file of the item ids it creates or updates in the target workspace. The record is written from the first item a live run creates or updates, so the proof of what the tool creates accrues from the first deploy that creates or updates an item. The default record path is deploy-record.<workspace id>.json in the current directory. A run from another directory without --deploy-record keeps a record of its own, so run the tool from the same folder each time, or name one record path with --deploy-record.

With --prune, after the deploy the tool removes the workspace items the deploy record proves it created and the current build no longer emits. Each item is reported before the first delete call. The prune never removes:

• an item made by hand in Fabric
• an item this tool finds and updates but does not create
• a renamed item
• an item a data pipeline of the build still names
• anything while an item of the run is skipped, unresolved or failed

A dry run with --prune reads the record, prints what the prune would remove and what it would leave, and writes nothing. A failed removal returns exit code 3, and a prune row that removes nothing changes no exit code.

The Report and Exit Codes

Each report row is written to standard output as one JSON line as it forms, and the device code message goes to standard error, so standard output stays machine readable. The report file named by --report carries the rows, a summary and the exit code. Neither the rows nor the report file carries a token, a secret, an Authorization value or the content of standard input.

Each row carries its kind and its action:

• A stop row, kind stop with action stopped, carries an errorCode and a reason. A file the plan reads and cannot read, a file of the Fabric root or the deploy record under --prune, writes the stop row runFailed before any call, with a reason that names the exception type.
• A warning row, kind warning with action reported, is information only and leaves the exit code as it is. The listItemIncomplete warning row names an item that List Items returns with no id or no displayName, with the item type and whichever of the two values the item has. The deploy keys a workspace item by both values, so it leaves that item out: the item is not updated, not reported as an orphan and not removed. The workspaceItem warning row names a Lakehouse, Warehouse or KQL database row of the connections manifest, reported once: the item needs no Fabric connection, the emitted pipelines carry it inline, and a connection created for it is referenced by no emitted pipeline. The deploy creates no connection from such a row, and the row's remediation is reported alone, with no connection type and no parameters to enter.
• A connection row, kind connection, carries the action created, adopted, manualEntry, failed or wouldProvision, as described in Connections.

Code	Meaning
0	Every item is created or updated.
1	The run ends with skipped or unresolved items.
2	The run stops before any call on a defect in the arguments, the files or the credential input: a bad or missing argument, an argument that belongs to no option, a secret given as an argument value, a connection credentials file that is missing, under the Fabric root, unreadable or holds a defect, a missing Fabric root, a manifest version, reference kind or row the tool does not read, two local items with one display name, an item folder with a missing or empty part, a file of the root that does not parse, a credential mode with a missing input, a service principal the identity client refuses, a browser sign in on a host that cannot open a browser.
3	An item fails, a connection list or create fails, the run stops on a failure (a sign in that fails, which stops the run before its first call, a file the plan reads that cannot be read, which stops the run before its first call with the runFailed stop row, a list call that fails, a referenced display name that maps to two workspace items, an error the tool does not expect), the report file cannot be written, or the plan carries a call graph cycle. A report file that cannot be written raises 0 and 1 to 3 and leaves 2, 3 and 4 as they are. A dry run over a cycle returns 3 and makes no call. A dry run over a file the plan cannot read returns 3 with the same stop row and writes no report file. A listItemIncomplete warning row never raises the code.
4	The run is cancelled with Ctrl+C, during a sign in or between two calls, or passes the time bound.

The last line on standard error names the exit code and its meaning, for every run but a help request, which prints the help and returns 0. An error the tool does not expect prints one line with its type and its message and returns 3.

Connections

A build writes connections-manifest.json at the Fabric root, one row per linked service with its status, the connectionId of a bound row, the remediation text with the steps to create the connection by hand, and a provisioning payload for each family the build knows: the connection type, the creation method and the parameters of the connection, with no secret.

With --connection-credentials <file>, the deploy provisions each connection whose row carries a payload and no connectionId, other than a Lakehouse, Warehouse or KQL database row, which is a workspace item and takes the workspaceItem warning row, before the notebooks:

1. It lists the connections the deploying identity can see, once.
2. A connection whose display name equals the row name is adopted: the deploy takes its id and creates nothing. Two connections with that display name fail the row, since the deploy never picks one.
3. Otherwise the deploy creates the connection from the payload, with the display name of the row, the privacy level Organizational and the credentials object of the file, and takes the id Fabric returns.
4. The id is written at every all zeros connection site of that connection that connection-references-manifest.json lists, in the definition that is sent and never in the files, so a pipeline that waits on the connection deploys in the same run.

bimlflex-fabric-deploy deploy --fabric-root <folder> --workspace-id <id> --credential token --token-stdin --connection-credentials <file>

A bound row is never listed or created. The deploy record holds no connection, and the prune never removes one, since a connection is not a workspace item.

The Connection Credentials File

The connection credentials file is a JSON object whose members name a connection as connections-manifest.json names it and give the credentials object the Fabric create connection call takes. The tool sends each credentials object as the file gives it. A file for two connections has this shape:

{
  "Source": { "credentialType": "Basic", "username": "<user>", "password": "<password>" },
  "Landing": { "credentialType": "ServicePrincipal", "tenantId": "<tenant id>", "servicePrincipalClientId": "<client id>", "servicePrincipalSecret": "<secret>" }
}

• Keep the file outside the Fabric root and outside source control. A path under the Fabric root is refused.
• A root that is not an object, a connection named twice, a value that is not an object, a value that names a member twice and a value with no credentialType string stop the run before any call with exit code 2 and the connection named. A file that does not parse stops the run with the line and the byte position alone.
• No value of the file reaches a row, a log line, a message or the report file. Every error code and message Fabric returns to the connection list or the connection create passes through a redaction of every value of the file before it enters a row. The connection and operation ids Fabric returns enter the rows as given.

Connection Rows

Each connection whose row carries a payload and no connectionId, other than a Lakehouse, Warehouse or KQL database row, gets a row of the kind connection, and the report summary counts the connections apart from the items. The action of the row is one of these:

• created or adopted, with the id of the connection.
• manualEntry, for a connection the deploy does not create. The row is one JSON line on standard output. Its message carries the remediation text of the manifest row, or the tool's own steps when the row carries none, and the connection type, the creation method and each parameter of the payload as a name and a value. The pipelines that use the connection stay unbound and skip, the rest of the deploy continues, and the row itself changes no exit code.
• failed, with exit code 3.
• wouldProvision, in a dry run, for each connection a live run adopts or creates.

A connection takes the manualEntry row in two cases:

• The credentials file gives no entry for it. Without --connection-credentials no connection has an entry, so every connection with a payload and no connectionId, other than a Lakehouse, Warehouse or KQL database row, takes that row and the run makes no connection call.
• Fabric refuses its create with a client error, a status of the 4xx class other than 429, whose errorCode or message names the connection type of the payload as a whole word. The text splits on every character that is not a letter or a digit, and one word equals the type with case ignored.

Every other failed create, such as a client error that does not name the type as a word, a 5xx answer or an operation that ends Failed, and a connections list that fails, give a row with the action failed and exit code 3.

The bind-connections Command

A workspace that updates from git checks every externalReferences.connection of a data pipeline against the connections the signing identity can use, and a build whose relational connections carry no External Reference in metadata writes the all zeros id there. The bind-connections command writes a bound copy of the Fabric root for the push to git:

• Every connection site that connection-references-manifest.json lists with the all zeros id takes the id the binding file gives its connection.
• Every other file is copied byte for byte. A site the build already binds keeps its id, and a notebook step keeps its workspaceId.
• The Fabric root is never written.
• The command reads no credential and makes no call.

bimlflex-fabric-deploy bind-connections --fabric-root <folder> --binding <file> --output <folder> --report <path>

Option	Meaning
--fabric-root <folder>	The Fabric folder of a build output. Required. It is read and never written.
--binding <file>	The binding file. Required. A JSON object whose members name a connection as connection-references-manifest.json names it and give its Fabric connection id. Every value is a connection id and not the all zeros id.
--output <folder>	The folder the bound copy is written to. Required. It is absent or empty, and it is never the Fabric root or a folder under it, so no file of an earlier copy reaches the push beside the new one.
--report <path>	The report file, written as JSON at the end of the run with one row per connection site, the id the copy carries there, a summary and the exit code. Never under the Fabric root or the output folder.
--allow-unbound	Write the copy while a site stays unbound. Each unbound site keeps the all zeros id and is reported, and the exit code stays 1.

A binding file for a single connection has this shape:

{ "BimlCatalog": "<connection id>" }

A rewritten pipeline-content.json is written as the deploy command serializes a rewritten data pipeline, and only its bound connection values differ from the input. Each report row is written to standard output as one JSON line, and each unbound site is named on standard error. Push the output folder, not the Fabric root, to the repository connected to the workspace.

Code	Meaning
0	Every connection site the manifest lists with the all zeros id is bound and the copy is written.
1	A site stays unbound, since the binding file names no id for its connection. The sites are named, and no copy is written unless --allow-unbound is given.
2	The run stops on a defect in the arguments or the files: a bad or missing argument, an argument that belongs to no option, a missing Fabric root or binding file, an output folder that is the Fabric root, under it or not empty, a report path under the Fabric root or the output folder, a missing connection-references-manifest.json, a manifest version or row the tool does not read, a site the pipeline document lacks, a binding file that is not an object of connection ids, a file of the root that does not parse, a copy or a report file that cannot be written, an error the tool does not expect.

Related Resources

• Microsoft Fabric Configuration Overview
• Fabric Pipeline Configuration Guide
• Fabric Git Integration
• Microsoft Fabric REST API documentation