Skip to main content

Release Notes

Varigence is excited to release the 2026 version of the BimlFlex platform for data solution automation!

note
  • This is a major release and we recommend testing in a non-production environment before upgrading your production environment.
  • Please make sure databases and projects are backed up before upgrading.
  • Please email [email protected] with any installation or upgrade issues.
Important for Upgrade

  1. When upgrading, it is recommended to upgrade the BimlCatalog database across all your environments to ensure compatibility.

  2. We have consolidated the TableObject and TableConfigurations into a single object. If you have any ExtensionPoints that reference TableObject or TableConfigurations. you will need to follow these steps. Upgrade Migrations

  3. Snapshot XML to JSON Migration: This upgrade includes a one-time conversion of all solution snapshots from XML to JSON format. This process adds approximately 30 seconds per snapshot depending on the performance of your database server. If you have a large number of snapshots, this migration may significantly extend the upgrade duration. It is recommended to archive or delete any snapshots that are no longer required before upgrading to reduce the migration time.

Installation​

BimlFlex 2026 is installed and upgraded through a single consolidated installer.

Build 26.1.324.0, release date: 14 Apr 2026

New Features​

Microsoft Fabric Lakehouse Support​

Introducing support for Microsoft Fabric Lakehouse, extending metadata-driven automation to the Fabric ecosystem. This provides capabilities similar to the Databricks and Snowflake integrations, enabling customers to design and generate Lakehouse solutions directly within the platform.

  • Metadata Import Support: Schema and object metadata can be imported directly from Fabric Lakehouse to drive automation patterns. This reduces manual mapping and accelerates the creation of Data Vault and Data Mart solutions based on existing Lakehouse structures.

  • Data Vault Templates: New templates generate hubs, links, satellites, and PIT/Bridge tables within Fabric Lakehouse. These accelerate Data Vault implementation on Fabric and ensure consistent, metadata-driven automation across the platform.

  • Data Mart Templates: Specialized templates for Fabric Lakehouse automate the creation of dimensional models and reporting structures. This provides optimized Data Mart solutions that align with Fabric’s analytics and performance capabilities.

  • Delete Detection Templates: Templates for identifying and processing deletions in source systems are now available within Fabric Lakehouse. These simplify data management, ensure data integrity, and reduce manual intervention in handling deletes.

Microsoft Fabric Warehouse Support​

Extended support for metadata-driven automation to include Microsoft Fabric Warehouse, providing full support for structured, SQL-based analytics. This expands the existing Fabric coverage, enabling data warehouse patterns and automation alongside Lakehouse, Databricks, and Snowflake.

  • Metadata Import Support: Warehouse objects such as schemas, tables, and views can be imported directly from Microsoft Fabric Warehouse. This accelerates onboarding by eliminating manual mapping and allows existing structures to drive automation patterns.

  • Data Vault Templates: Templates have been introduced for implementing Data Vault modeling within Fabric Warehouse, including hubs, links, satellites, and PIT/Bridge tables. This ensures consistent automation across both Warehouse and other supported targets.

  • Data Mart Templates: Fabric Warehouse-specific templates support the generation of dimensional models, including star schemas and reporting tables. These templates are optimized for Fabric’s T-SQL–based query engine to deliver strong performance for analytics workloads.

  • Delete Detection Templates: Support for delete detection has been added to Fabric Warehouse, enabling automated handling of record removals in source data. This ensures warehouse tables remain synchronized and accurate without requiring manual intervention.

Microsoft Fabric Database Support​

Support for Microsoft Fabric Database is introduced in preview. This functionality is limited, as many of the features required are still in preview on the Microsoft Fabric platform or have not yet been added.

note

Fabric Database support should be considered experimental at this stage. It is recommended for evaluation and testing purposes only, not for production workloads.

Databricks Enhancements​

Introducing major new capabilities for Databricks, focused on performance optimization and expanded development options.

  • Pushdown Processing: A new option enables processing logic to be pushed directly into Databricks, eliminating the need for Azure Data Factory notebook activities. Instead, pipelines use the Azure Data Factory Databricks Job Activity, which significantly reduces cluster spin-up overhead and improves runtime performance.

    With pushdown enabled, all transformations are executed within Databricks workflows or jobs, rather than being orchestrated externally. This approach lowers runtime costs, simplifies orchestration, and makes better use of ephemeral job clusters. Early benchmarks demonstrated pipelines completing in under 30 minutes using job clusters, compared to over two hours on larger dedicated clusters — representing up to a 75% reduction in runtime cost. Note that BimlFlex requires the MS SQL ODBC driver, which means clusters must be configured with an init script (bfx_init_odbc.sh) generated during the Databricks Asset Bundle build. Serverless compute is not currently supported due to this requirement.

    Pushdown processing supports staging, Data Vault, and Data Mart layers, with generated Databricks workflows handling dependencies, delta detection, and restart logic for resilient and incremental loads. All generated artifacts (pipelines, jobs, and notebooks) remain fully native to Databricks and Azure Data Factory, ensuring no proprietary runtime or execution engine is required.

  • SQL Scripting: A new configuration option enables native SQL-based scripting for Databricks workloads. Metadata-driven templates now support generating staging, Data Vault, and Data Mart patterns directly in SQL, providing greater readability and easier debugging. This option allows teams to align development practices with SQL-centric approaches while still leveraging Databricks’ scalability and performance.

  • JDBC/pytds Connectivity: A new connectivity option allows Databricks notebooks to connect back to SQL Server using JDBC and pytds, enabling direct metadata and configuration lookups without requiring ODBC drivers. This also introduces support for serverless compute clusters, removing the previous dependency on init scripts for ODBC driver installation.

  • Pushdown Restart Support: Pushdown processing now supports restart scenarios through a new NextLoadStatus=’D’ mechanism. When a load is marked for restart, the framework re-executes from the appropriate checkpoint, improving resilience for long-running incremental pipelines.

  • Centralized Logging: Logging across Databricks notebook generation has been centralized and standardized, providing consistent log output and easier troubleshooting across staging, Data Vault, and Data Mart layers.

Dynamics 365 Integration​

Native support for Microsoft Dynamics 365 as a data source with flexible connectivity options:

  • Linked Service Support: Added support for the Dynamics linked service, with options for both Dynamics CRM and Dynamics 365.
  • Import Metadata: Excluded references to the systemuser entity. These references generated excessive and non-informative links that added noise without providing modeling value.
  • Many-to-Many: Many-to-Many relationship logic now correctly uses the IntersectEntity, resolving issues with incorrect results and rogue records.

REST API Metadata Import​

A new import workflow enables metadata to be imported directly from REST API sources. From the connection form, users can sample a REST API endpoint, parse the JSON response, review detected columns, and import them as BimlFlex objects — all without leaving the application.

  • JSON Response Parsing: The JSON response from a REST API is automatically parsed to detect array paths and scalar fields. The parser generates ADF-compatible bracket-notation expressions (e.g., $['data'][0]['name']) that map directly to Azure Data Factory REST source column mappings.
  • Column Review and Import: A column review grid displays detected fields with their inferred data types, ordinal positions, and source expressions. Users can review, edit, and select columns before importing them into the metadata model.
  • Collection Reference Selection: When the API response contains multiple nested arrays, a dropdown allows users to select which collection to import. Changing the collection re-parses the response and refreshes the column grid automatically.
  • Authentication Support: Six REST authentication types are supported in the linked service configuration: Anonymous, Basic, OAuth2 Client Credential, Service Principal, System-Assigned Managed Identity, and User-Assigned Managed Identity. Authentication fields are shown or hidden dynamically based on the selected type.

Linked Service Modernization​

Updated linked service connectors for Snowflake and Oracle to align with Azure Data Factory's current V2 connector standards.

  • Snowflake V2 Linked Service: The Snowflake connector has been upgraded to V2, replacing the legacy connection-string based configuration. New properties include Host, Version, and UseUtcTimestamps, providing more explicit control over connection behavior and timezone handling. The V2 connector is now registered across Copy, Delete, and GetMetadata activities.
  • Oracle V2 Linked Service: The Oracle connector has been migrated to V2 with a direct-property UI replacing the previous connection-string based configuration. This provides a clearer configuration experience with explicit fields for authentication type and connection parameters.
  • ADF Linked Service Enhancements: All ADF linked services now support additional connection properties, authentication headers, and Azure Cloud Type selection. These enhancements enable connectivity to sovereign cloud environments and APIs that require custom headers or extended connection parameters.

Project Screen Visualization​

A new visualization has been added to the Project screen to provide a clearer view of data movement across integration layers. The diagram illustrates how data flows between landing, staging, Data Vault, and Data Mart layers, helping users better understand dependencies, track lineage, and validate architecture design. This enhancement improves transparency and makes it easier to communicate solution structure to both technical and business stakeholders.

Project Screen

Metadata Merging​

A new metadata merge capability allows project metadata to be merged between versions or customers directly in the BimlFlex database. This supports parallel development workflows, environment promotion, and team consolidation — without requiring manual re-entry or export/import cycles.

  • Preview and Execute Workflow: A two-step process lets you preview all changes and conflicts before committing. The preview is read-only and shows exactly what will be added, unchanged, or in conflict.
  • Conflict Resolution: When the same entity exists in both source and target with different values, conflicts are surfaced with column-level detail. Each conflict is resolved as either take-incoming (accept the source) or take-target (keep the existing value).
  • Cross-Customer Support: Metadata can be merged across different customers within the same database, enabling consolidation from development to production customers or across isolated teams.
  • Transactional Safety: The entire merge runs within a single transaction — if any step fails, all changes are rolled back automatically.

For the full workflow and syntax reference, see Merging Metadata.

Enhanced Business Modeling​

Streamlined workflows for business attribute management and naming conventions:

  • Automated Business Naming: One-click application of business naming conventions across objects and columns.
  • Advanced Cloning: Enhanced metadata cloning with business name preservation and selective object handling.
  • Improved Grid Management: Better column reordering capabilities and enhanced visual presentation of data grids.
  • Save All Functionality: Comprehensive save operations across multiple form tabs with improved tracking of changes.

Enhancements and New Settings​

Platform Support Enhancements​

  • Connection String Builder Improvements: Enhanced connection configuration with Microsoft Fabric support and improved visibility for connection parameters.

User Interface and Experience​

  • Unified Notification System: Comprehensive notification service with progress status reporting and detailed feedback dialogs.
  • Consistent Dialog Framework: Improved dialog interactions with better handling of unsaved changes and enhanced navigation flows.
  • Enhanced Validation: New validators including Databricks Views validation to prevent configuration errors and improve data quality.
  • Documentation Viewer Enhancements: Improved in-product documentation access with better macro editing capabilities and enhanced code editing experience.

Data Processing and Integration​

  • Advanced Batch Processing: Improved error handling for batch operations with automatic cleanup of orphaned processes.
  • Enhanced Execution Monitoring: Better visibility into data processing workflows with improved status tracking and troubleshooting capabilities.
  • Configuration Management: More reliable configuration variable persistence and deployment consistency improvements.
  • FTP/SFTP DisableChunking: A new DisableChunking property has been added to FTP and SFTP connections, allowing chunked transfer encoding to be disabled for destinations that do not support it.
  • Time Travel Support: New database schema and stored procedures to support time travel queries, enabling point-in-time analysis of metadata state. Execute and preview procedures have been consolidated for a simplified experience.
  • Snapshot JSON Migration: Solution snapshots have been migrated from SQL-based storage to JSON format, improving portability and simplifying snapshot management. A new PowerShell export script is included for snapshot capture.

Bug Fixes​

Critical System Fixes​

  • User Settings Management: Fixed cascade delete issues to prevent orphaned records when managing user configurations.
  • Project Switching Reliability: Resolved settings loss problems when changing between projects, ensuring configuration persistence.
  • Business Attribute Saving: Fixed multiple issues ensuring business attributes save correctly and maintain data integrity.
  • Archive Operations: Resolved JavaScript runtime errors during entity archiving operations improving system stability.
  • BimlCatalog Auto-Archive Deadlock: Resolved transactional deadlocks introduced in the initial 2026 build where adf.LogExecutionEnd synchronously invoked bfx.ArchiveAll at the end of every successful ADF batch. On populated catalogs the inline archive issued unbounded DELETE statements against the live history tables and deadlocked against concurrent orchestration calls. Archive maintenance is now decoupled from orchestration, every archive procedure deletes in bounded 5,000-row batches, and the inserts into the archive schema are idempotent so partial runs resume cleanly. See Upgrade Considerations → BimlCatalog Archive Maintenance below for the one-time setup step.

Grid and Data Management​

  • Column Grid Display: Fixed visual presentation issues and improved usability of data grids across multiple components.
  • Metadata Import Filtering: Resolved filtering issues in import operations ensuring more reliable metadata import processes.
  • Grid Width and Layout: Fixed display formatting issues providing better visual presentation and user experience.

Platform-Specific Improvements​

  • Databricks Processing: Fixed temporary views configuration and batch processing issues ensuring proper lifecycle management.
  • Batch Objects Refresh: Resolved grid refresh issues ensuring newly created objects appear immediately after save operations.
  • Execution Status Tracking: Fixed status reporting accuracy improving monitoring and troubleshooting capabilities.

Performance Improvements​

Cloud Platform Optimization​

  • Databricks Integration: Enhanced orchestration capabilities for improved performance and cost efficiency in cloud data warehouse operations.
  • Batch Processing: Optimized large-scale operation handling with better memory management and process efficiency.
  • Query Generation: Improved SQL generation for complex scenarios reducing processing overhead.

Application Performance​

  • Change Tracking Updates: Updated application framework providing improved performance, faster load times, and better memory management.
  • Save Operations: Enhanced save functionality with improved tracking across multiple form tabs and more responsive indicators.
  • Code Generation: Optimized metadata operations including faster cloning and duplication processes.

Databricks Fixes and Enhancements​

  • PushdownExtraction Query Fix
    Resolved an issue affecting PushdownExtraction by enhancing the generated source query. An additional EXECUTE IMMEDIATE statement has been introduced to ensure the query is executed correctly in Databricks and handles dynamic SQL scenarios as expected.

  • Dynamic SQL for View Creation
    Updated the view creation logic to use dynamic SQL. This improves compatibility with Databricks SQL execution requirements and ensures views are created reliably across different runtime contexts.

  • Satellite Full Load Script Header Fix
    Added the missing header to the Satellite full load script. This ensures consistency with other generated scripts and prevents execution or validation issues caused by incomplete script metadata.

  • Table Deployment Dependency Ordering
    Enhanced the table deployment script to leverage SolveOrder and SortOrder. This guarantees that tables are created in the correct sequence based on their dependencies, reducing deployment failures caused by out-of-order object creation.

  • Deferred Creation of SourceCreateSql Objects
    Modified the deployment process to separate SourceCreateSql objects from other database objects. These objects are now created last, after all dependent objects have been successfully deployed, ensuring a more reliable and predictable deployment process. Note the deployment is done by executing the child deploy script usely named starting with a 05_*.py. 

Recent Fixes​

  • Databricks Display Folder Support: Notebook activities now respect DatabricksUseDisplayFolder, correctly resolving relative paths for Data Vault notebook generation.
  • Databricks DM/DWH Notebook Paths: Fixed notebook path resolution and parameter variable access for Data Mart and Data Warehouse patterns.
  • DIM Boolean Handling: Fixed incorrect boolean handling in dimension processing.
  • Web Activity Headers: Fixed a null reference when validating headers on Web Activity nodes.
  • REST Request Method: Corrected the HTTP method used in REST source requests.
  • Date Format Standardization: Normalized date format strings to lowercase yyyy-MM-dd across all templates for consistent behavior.

Upgrade Considerations​

Prerequisites​

  • BimlFlex 2026 requires .NET Framework 4.8
  • SQL Server 2016+ for BimlCatalog deployment

Migration Steps​

  1. Backup Current Environment: Ensure complete backup of existing BimlFlex databases and configurations
  2. Update BimlCatalog: Mandatory upgrade across all environments for compatibility

BimlCatalog Archive Maintenance​

On SQL Server, bfx.ArchiveAll is no longer invoked automatically from adf.LogExecutionEnd. The AutoArchive key in admin.Configurations is deprecated and has no effect; rows previously set to '1' are inert and can be left in place. Archive maintenance is now driven by an external schedule.

A SQL Server Agent job template is shipped with the BimlCatalog source at BimlCatalog/Scripts/bimlcatalog-archive-weekly-job.sql. The DACPAC cannot create SQL Agent jobs (they live in msdb), so this script is a one-time customer action after upgrading:

  1. Open the script and replace the <JOB_OWNER_LOGIN> and <BIMLCATALOG_DB> placeholders with the SQL login that should own the job and the name of your BimlCatalog database.
  2. Run the script against the SQL Server instance that hosts BimlCatalog. The caller must be a member of SQLAgentOperatorRole or sysadmin in msdb.
  3. The script creates a job named BimlCatalog - Weekly Archive that runs EXEC [bfx].[ArchiveAll] weekly on Sunday at 02:00 server-local time. Edit @freq_interval (day-of-week bitmask) and @active_start_time (HHMMSS) before running if you want a different cadence. The script is idempotent — re-running it drops and re-creates the job. The job step sets DEADLOCK_PRIORITY LOW and LOCK_TIMEOUT 60000 so the archive yields to live orchestration on lock contention rather than blocking it.

For Azure SQL Database (no SQL Server Agent), schedule the same EXEC [bfx].[ArchiveAll] call from Elastic Jobs, an Azure Automation runbook, an Azure SQL Managed Instance Agent job, or a scheduled Azure Data Factory pipeline.

:::note PostgreSQL port

The PostgreSQL port of BimlCatalog at PostgreSQL/bimlcatalog/ still honors the AutoArchive setting and inline-invokes bfx."ArchiveAll" from bfx.log_execution_end. PostgreSQL customers experiencing the same deadlock symptoms should set AutoArchive to '0' in admin.Configurations and schedule bfx."ArchiveAll" externally. A parallel PostgreSQL-side fix is planned for a future release.

:::

Support​

For questions or issues related to BimlFlex 2026:

For installation or upgrade assistance, please email [email protected] with detailed information about your environment and any issues encountered.