Skip to main content

Metadata Types

BimlFlex has a number of internal static types that are used to classify metadata. They can also be referred to as an enumeration type (or 'enum'), and represent the constant values that can be used in the App.

Attribute Types

Attribute Types are applied using the Attribute Editor and govern the scope for which the Attribute is applied. Selecting the attribute type will then show available options for the selected type.

The following options are available:

Attribute TypeCodeDescription
BatchBATA Batch in the BimlFlex App.
ColumnCOLA Column in the BimlFlex App.
ConnectionCONA Connection in the BimlFlex App.
CustomerCUSA Customer in the BimlFlex App.
ObjectOBJAn Object in the BimlFlex App.
ProjectPRJA Project in the BimlFlex App.

Change Types

Change Types are applied using the Column Editor and drive the behavior and purpose of the Column for the design and code generation process. The available change types are dependent on the way the Connection is used in a BimlFlex Project. Depending on the connection's Integration Stage some options may, or may not be, available.

The following options are available:

Change TypeCodeDescriptionApplies To
UpdateCHGAssigns the column to track the latest value assigned to the column. Also used as default for attribute columns. In Dimensional Modeling, this corresponds to a 'Type 1' column.All integration stages, any column.
KeyKEYIndicates that the column is treated as a key value in downstream processing. It is part of the object's Primary Key definition for all intents and purposes.All integration stages, any column.
IgnoreIGNAssigns the column to be ignored from all processing. This means that the column will not appear in the object and any data pipelines associated with it.All integration stages, any column.
Track HistoryHSTAssigns the column to track historical changes for the column. In Dimensional Modeling, this corresponds to a 'Type 2' column.Columns for an objects belonging to a connection with a Data Mart integration stage.
Change Data CaptureCDCIndicates this column is used for historization in Data Marts.Columns for an objects belonging to a connection with a Data Mart integration stage.
Hash DistributionHDKAssigns the column to be used for hash distribution when using Synapse objects.All integration stages, any column, but only when used for Synapse connections.
TransientTRSAssigns the column to be used in relevant data flow tasks, but not to persist the information. Used when a column value is needed for an expression but the value of the column itself is not needed to be persisted.Any object used in a project with the SQL Server Integration Services (SSIS) integration template without pushdown processing enabled.
Exclude DVEXCProcesses the column into the Staging Area and Persistent Staging Area (if configured), but does not process/accelerate the column into the Data Vault.Columns for an objects belonging to a connection with a Source System integration stage, any object.
Link ReferenceREFDefines what columns will be used to define Surrogate Keys for the associated Link. These values will be pre-calculated (pre-hashed, if hashing is used) and saved in the Staging Area, but not in the Persistent Staging Area (if configured). Generated by BimlFlex during acceleration to map source columns to individual Link Surrogate Keys.Columns for an objects belonging to a connection with a Source System integration stage, any object.
Link DegenerateDGAAssigns the column as part of the Unit of Work for a Link. The column will be included in the hashing of the Link's Surrogate Key, and will be added to the Data Vault Link object. This is also known as a dependent child key.Columns for an objects belonging to a connection with a Source System integration stage, any object.
Link-Satellite ReferenceLSRDefines what columns will be used to define the Surrogate Keys for the target Link Satellite. These values will be pre-calculated (pre-hashed, if hashing is used) and saved in the Staging Area, but not in the Persistent Staging Area (if configured). Generated by BimlFlex during acceleration to map source columns to individual Link Surrogate Keys.Columns for an objects belonging to a connection with a Source System integration stage, any object.
Stub HubSTHDesignates an object to be created as its own Hub using the Data Vault Accelerator, with a Link to the main Hub that will be created from the source object. The column on which Stub Hub is applied will be used as Business Key / Surrogate Key definition.Columns for an objects belonging to a connection with Source System integration stage, any object.
Driving KeyLDKA Driving Key is a special column that is used in Data Vault to drive custom historization in Link Satellites. This occurs when changes in Link key values must trigger a change record in the Link Satellite.Columns for an objects belonging to a connection with a Data Vault integration stage.
Effective DateEFDAssigns the column as an effective date marker for controlling when records become active in historical tracking scenarios. Used in Data Vault and Dimensional modeling for point-in-time queries.Columns for objects belonging to a connection with a Data Vault or Data Mart integration stage.
Effective Date Low MarkerEFDLAssigns the column as the start/from date in a date range for bi-temporal or effective-dated records. Represents when a record version becomes valid.Columns for objects belonging to a connection with a Data Vault or Data Mart integration stage.
Effective Date High MarkerEFDHAssigns the column as the end/to date in a date range for bi-temporal or effective-dated records. Represents when a record version expires or is superseded.Columns for objects belonging to a connection with a Data Vault or Data Mart integration stage.
Hub Natural KeyHNKIndicates that the column contains a natural key value used to identify a Hub entity in Data Vault. Used when referencing Hub records by their business key rather than surrogate key.Columns for objects belonging to a connection with a Source System or Data Vault integration stage.
Hub ReferenceHRFDefines what columns will be used to reference a Hub's Surrogate Key. These values are pre-calculated during staging and used to establish relationships to Hub entities in Data Vault.Columns for objects belonging to a connection with a Source System integration stage.
Link Natural KeyLNKIndicates that the column contains a natural key value used to identify a Link relationship in Data Vault. Used when the source provides natural key references rather than surrogate keys.Columns for objects belonging to a connection with a Source System or Data Vault integration stage.
Multi Active RowMAKAssigns the column to be a multi-active value for the associated object. Used to determine the grain of a Multi-Active Satellite. This column must also be part of the Primary Key, together with the Integration Key.Columns for an objects belonging to a connection with Source System integration stage, any object.
Multi Active SetMASAssigns the column to define a set for multi-activeness. Used when a group of rows are active and maintained as a unit for the Satellite. All rows in the set share the same Multi Active Set value and are versioned together.Columns for objects belonging to a connection with Source System integration stage, any object.
ReplicateRPLAssigns the column to be replicated across all distributions in Azure Synapse Analytics. Used for smaller lookup/dimension tables to improve join performance by avoiding data movement.Columns for objects belonging to a connection configured for Azure Synapse.

Configuration Attribute Types

Configuration Attribute Types are the allowed values to configure the attribution of a Configuration to a specific area of the data solution architecture. This is defined using the Configuration Editor and governs the location in the data pipeline for which the Configuration is applied.

Each configuration property will allow a list of values to be selected, which impacts how the configuration is applied to the in-scope objects.

Attribute TypeCodeDescription
IgnoreIGNSet the Configuration Attribute to Ignore to make sure the configuration does not apply to the assigned area.
DerivedDERDerive the configuration using the value specified in the appropriate Expression column. Which expression column will be used depends on the Integration Template used.
SourceSRCUse the value as is available from the source object. This column needs to exist in the source and have the same name as specified by the Configuration Value column for the configuration in the source object.
DefaultDEFApply the value specified in the Configuration Default field of the Configuration as a default value / constraint for the Objects the configuration has been defined for. Note that this only applies to SQL Server, Managed Instances and Azure SQL because these database technologies support having default constraints with expression (example: GETDATE()).
TargetTGTDefined by the Target.
HashHSHApply hashing logic when loading the value.

Connection Types

Connection Types are configured using the Connection Editor to define what kind of data source or target is connected to.

Connection TypeCodeDescriptionApplies To
ADONETADONETAn ADO.NET connection is used to connect to the data.Any Connection.
Azure Blob StorageAZBA connection to an Azure Blob Storage Account.Connections with a Source System, Landing Area, or Staging Area integration stage.
Azure Data Lake StoreAZDLSA connection to Azure Data Lake Storage Account.Any Connection except if defined with integration stage Data Warehouse.
Custom ComponentCUSTOMA connection to a custom component data source (COZYROC). This is limited to connections that are used in Projects that have a SQL Server Integration Services (SSIS) integration template.Connections with a Source System integration stage.
FILEFILEA connection to a file.Connections with a Source System integration stage.
FTPFTPAn FTP connection is used to connect to the data source.Connections with a Source System integration stage.
Microsoft MDSMDSA connection to Microsoft's Master Data Services (MDS).Connections with a Master Data Services integration stage.
ODBCODBCAn ODBC connection is used to connect to the data.Any Connections except ones with a Landing Area integration stage. Allowable System Types will be limited depending on the integration stage applied.
OLEDBOLEDBAn Object Linking and Embedding (OLE DB) connection is used to connect to the data.Any Connection. For connections with a Source System there will be additional System Types available.
ORACLEORAA connection to a Teradata data source.Connections with a Source System integration stage.
RESTRESTA connection to a Rest API endpoint.Connections with a Source System integration stage.
Script SourceSCSRCA connection to a script task data source. This is limited to connections that are used in Projects that have a SQL Server Integration Services (SSIS) integration template.Connections with a Source System integration stage.
SFTPSFTPAn secure FTP (SFTP) connection is used to connect to the data.Connections with a Source System integration stage.
TERADATATDA connection to a Teradata data source.Connections with a Source System integration stage.
Azure File ServerAZFSA connection to Azure File Storage for SMB-based file access in the cloud. Supports file shares that can be mounted as network drives.Connections with a Source System or Landing Area integration stage.
Azure Key VaultAZKVA connection to Azure Key Vault for centralized secrets management. Used to securely store and retrieve connection strings, passwords, and API keys.Connections with an Azure Key Vault integration stage.
DatabricksDBRA connection to a Databricks workspace for unified analytics and data engineering. Supports notebooks, SQL warehouses, and Unity Catalog.Connections with any integration stage when using Databricks as a compute or storage platform.
Microsoft FabricFBRA connection to a Microsoft Fabric workspace. Supports Fabric Warehouses, Lakehouses, and SQL Databases within the Fabric ecosystem.Connections with any integration stage when using Microsoft Fabric as the data platform.

Data Types

Data Types are the allowed values that can be used to specify Columns in BimlFlex.

Data TypeDescription
AnsiStringANSI string.
AnsiStringFixedLengthFixed length ANSI string.
BinaryBinary data column.
BooleanBoolean, Bit or True/False.
ByteUnsigned tiny integer.
CurrencyMoney or currency.
DateDate only.
DateTimeDate and time.
DateTime2Variable precision date and time. Increased precision over DateTime.
DateTimeOffsetData and time with time zone offset.
DecimalFixed precision and scale numbers.
DoubleFloat or Double.
GuidUnique identifier or GUID.
Int16Small integer.
Int32Integer.
Int64Big integer.
ObjectObject. Used for complex data types.
SByteTiny integer.
SingleReal or Single.
StringUnicode String.
StringFixedLengthFixed Length Unicode String.
TimeTime only.
UInt16Unsigned small integer.
UInt32Unsigned integer.
UInt64Unsigned big integer.
VarNumericVariable numeric
XmlXML formatted data.
HierarchyIdSQL Server HierarchyId data type.
GeometrySQL Server geometry data type.
GeographySQL Server geography data type.
SmallMoneySmall money.
VariantVariant. Non explicit data type.
note

For information on how these map to SQL Server, Synapse and Snowflake data types see associated link below.
BimlFlex Documentation: BimlFlex Data Type Conversions.

Integration Stages

The Integration Stage is a Connection property that defines and drives how a connection is used in the architecture. The integration stage is an important classification for a connection, because it is used by BimlFlex to direct what code is generated, and what further options are available - including Connection Types and corresponding System Types and even down to the Change Types at Column level.

The following integration stages are available:

Integration StageCodeDescriptionApplies To
Business Data VaultBDVThis layer represents the data following the application of the soft business rules that may be required.Any connection. Selecting this integration stage will limit the available connection types and system types to BimlFlex supported target platforms.
Data MartDMA subset of data stored within the data warehouse, for the needs of a specific team, section or department within the enterprise.Any connection. Selecting this integration stage will limit the available connection types and system types to BimlFlex supported target platforms. The allowed change types at column level will also be scope to support Dimensional Modeling.
Data WarehouseDWHThe connection type dedicated for the BimlCatalog. A connection with the Data Warehouse integration stage can not be used in project definitions.BimlCatalog connections only.
File ExportEXPA file or series of files that contain a subset of the data in a Data Warehouse.Connections used in projects that have been configured with an Integration Template of SQL Server Integration Services (SSIS) only.
Landing AreaLNDA Landing Area connection is only applicable to Cloud Enabled Connections, to identify the location where data needs to be 'landed' before being loaded into the cloud database.Any connection. Selecting this integration stage will limit the available connection types and system types to BimlFlex supported target platforms.
Master Data ServicesMDSMicrosoft's solution for handling "master" data within an enterprise using SQL Server.Connections used in projects that have been configured with an Integration Template of SQL Server Integration Services (SSIS) only.
Persistent Staging AreaPSAA staging area where history is kept and changed data capture (CDC) is managed for all source attributes.Any connection. Selecting this integration stage will limit the available connection types and system types to BimlFlex supported target platforms.
Data VaultDVA uniquely linked set of normalized tables that support one or more functional areas of business and provides historical tracking.Any connection. Selecting this integration stage will limit the available connection types and system types to BimlFlex supported target platforms.
Source SystemSRCA data source that is being extracted, with the intent to be processed into the BimlFlex data integration environment.Any connection.
Staging AreaSTGA temporary storage area between the source systems and a data warehouse.Any connection. Selecting this integration stage will limit the available connection types and system types to BimlFlex supported target platforms.
Target Staging AreaTSAOnly used in SSIS DataWarehouse and Data Mart templates.Connections used in projects that have been configured with an Integration Template of SQL Server Integration Services (SSIS) only.
Persistent Landing AreaPLNDA Landing Area connection where data is persisted and retained for historical tracking. Unlike standard Landing Areas which are transient, Persistent Landing Areas maintain a full history of landed files before processing into the staging layer.Any connection. Selecting this integration stage will limit the available connection types and system types to BimlFlex supported landing platforms.
ComputeCMPA connection type designated for compute resources such as Databricks clusters. Used to define processing engines that execute data transformations rather than storing data.Connections configured for Databricks compute.
Azure Key VaultAKVA connection to Azure Key Vault for managing secrets, connection strings, and sensitive configuration values. Provides centralized and secure credential storage for the data solution.Azure Key Vault connections only.

Integration Templates

An Integration Template in BimlFlex drives which code generation patterns are applied to the design metadata. In other words, the Integration Template defines how the output will be generated. An integration template is set for a Project, and all data pipelines defined in this project will be generated for the selected template.

BimlFlex supports SQL Server Integration Services (SSIS) and Azure Data Factory (ADF) Execute Pipelines as runtime integration engines, and depending on specific Connection properties also generates SQL-based Stored Procedures.

The following integration templates are supported in BimlFlex:

Integration TemplateCodeDescription
SQL Server Integration Services (SSIS)SSISDeliver the data pipelines for the project in SQL Server Integration Services (SSIS). If Pushdown Processing is enabled, SSIS will orchestrate the data pipelines, but the processing will mostly occur on the data platform using Stored Procedures. If pushdown processing is disabled, most the processing will be done in SSIS Data Flows.
Azure Data Factory (ADF)ADFDeliver the data pipelines for the project using Execute Pipelines. This will generate Execute Pipelines that use a Copy Activity to ingest the data from the source, and Stored Procedures to process the data further.
SSIS: File ExtractS2FILUse SSIS to extract data to file.
SSIS: Zip File ExtractS2ZIPUse SSIS to extract data to a zip file.
Databricks (ADF)DBRDeliver the data pipelines for the project using Azure Data Factory for orchestration with Databricks notebooks for data processing. ADF pipelines trigger Databricks jobs that execute the transformation logic using Spark.
Data Factory (Fabric)FDFDeliver the data pipelines for the project using Microsoft Fabric Data Factory pipelines. This generates Fabric Data Pipelines that orchestrate data movement and processing within the Fabric ecosystem, leveraging Fabric compute and storage.

Object Types

Object Types are configured using the Object Editor and drive the behavior and purpose of the Object for the design and code generation process. The available object types are dependent on the way the Connection is used in a BimlFlex Project. Depending on the connection's Integration Stage and System Type some options may, or may not be, available.

Change TypeCodeDescriptionAllowed for Integration StageAllowed for System Type
BridgeBRGDefines the object as a Data Vault Bridge entity.Business Data Vault, Data VaultAll
DimensionDIMDefines the object as a Dimension Table in a Dimensional Model / Data Mart.Data MartAll
Exclude in AcceleratorEXCFlags an object to not be included in the Data Vault Accelerator.Source SystemAll
FactFCTDefines the object as a Fact Table in a Dimensional Model / Data Mart.Data MartAll
Flat FileFILDetermines that the object is a flat file.File Export, Source SystemAvro Format, Binary Format, Delimited Format, JSON Format, ORC Format, Parquet Format, Ragged Right Format , XML Format
Hierarchy LinkHALDefines the object as a Data Vault Hierarchy Link entity. Hierarchy Links model recursive or parent-child relationships within a single Hub, such as organizational structures, bill-of-materials, or reporting hierarchies.Business Data Vault, Data VaultAll
HubHUBDefines the object as a Data Vault Hub entity.Business Data Vault, Data VaultAll
IgnoreIGNSetting an object to Ignore will remove it from any processing and/or visualisation in BimlFlex. No Staging Area or Persistent Staging Area objects will be derived.AllAll
JsonJSONDetermines that the object is a flat file in JSON format. Only applies when used for a REST API connection.Source SystemRest Service
LinkLNKDefines the object as a Data Vault Link entity.Business Data Vault, Data VaultAll
Link SatelliteLSATDefines the object as a Data Vault Link-Satellite entity.Business Data Vault, Data VaultAll
Point In TimePITDefines the object as a Data Vault Point-In-Time (PIT) entity.Business Data Vault, Data VaultAll
ReferenceREFDetermines that this object will be generated as a stand-alone historized table in a Data Vault context.Business Data Vault, Data VaultAll
Reference SatelliteRSATDetermines that this object will be generated as a stand-alone (unconnected) Satellite in a Data Vault context.Business Data Vault, aw Data VaultAll
SatelliteSATDefines the object as a Data Vault Satellite entity.Business Data Vault, Data VaultAll
Stored ProcedureSPFor source objects only, determines the the object is a stored procedure whose output must be ingested into the data solution by BimlFlex.Source SystemAzure SQL Database, Azure Synapse, Microsoft SQL Server
Staged ObjectSTQDefines the object as an object that is derived from other existing objects. This will be treated as a data source, but no Staging Area and/or Persistent Staging Area components will be generated. For more details, please refer to the Staged Object Concept.Source SystemAzure SQL Database, Azure Synapse, Microsoft SQL Server, Snowflake Data Warehouse
TableTBLThis is the default object type in BimlFlex. A Table property for the object means that no specific behaviour is applied by the BimlFlex code generation processes.Business Data Vault, Landing Area, Master Data Services, Persistent Staging Area, Data Vault, Source System, Staging AreaAzure Synapse, COZYROC Excel File, COZYROC Salesforce, DB2, Dynamics CRM, Microsoft SQL Server, MySql, Oracle, Oracle RDB, PostgreSQL, Salesforce, Salesforce Marketing Cloud, Salesforce Service Cloud, Snowflake Data Warehouse, Sybase, Teradata
CDC AllTCADetermines that the object is configured for Change Data Capture, querying the full history of available changes. BimlFlex will generate specific code to query the object.Source SystemMicrosoft SQL Server
CDC LastTCLDetermines that the object is configured for Change Data Capture, but only retrieves the last value every time data is loaded. BimlFlex will generate specific code to query the object.Source SystemMicrosoft SQL Server
Change TrackingTCTDetermines that the object is configured for Change Tracking. BimlFlex will generate specific code to query the object.Source SystemMicrosoft SQL Server
Same As LinkSALDefines the object as a Data Vault Same-As Link entity. Same-As Links relate equivalent business keys from different sources, establishing that records from multiple systems represent the same real-world entity.Business Data Vault, Data VaultAll
ViewTBVDefines the object as a database view. Used when the source object is a view rather than a physical table. BimlFlex will query the view during extraction but not generate DDL for the source.Source System, Data Vault, Business Data VaultAzure SQL Database, Azure Synapse, Microsoft SQL Server, Snowflake Data Warehouse

System Types

System Types are configured using the Connection Editor to define what kind of data source or target is connected to as part of the Connection Type configuration.

System TypeCodeDescriptionSupported by Connection Type
Avro FormatAVROAvro is a binary, compressed, file format. Only supported for Azure Data Factory.Azure Blob Storage, Azure Data Lake Store, FILE, FTP, SFTP
Azure SynapseSQLDWAzure Synapse Analytics database (Formerly Azure SQL Data Warehouse).ADONET, OLEDB
Binary FormatBINBinary storage file. Only supported for Azure Data Factory.Azure Blob Storage, Azure Data Lake Store, FILE, FTP, SFTP
COZYROC Excel FileCREXCELExcel File using COZYROC SSIS Components.Custom Component
COZYROC SalesforceCRSFDCSalesforce API using COZYROC SSIS Components.Custom Component
DB2DB2IBM DB2 database.ADONET, ODBC, OLEDB
Delta LakeDELTADelta Lake Storage Layer.Azure Data Lake Store
Dynamics CRMDYNCRMAPI that can connect to Microsoft Dynamics data sources.Rest API
Delimited FormatFF_DELFlat File with Delimiters.Azure Blob Storage, Azure Data Lake Store, FILE, FTP, SFTP
Ragged Right FormatFF_RAGFlat File with Ragged Right formatting.FILE
JSON FormatJSONJSON file format. Only supported for Azure Data Factory.Azure Blob Storage, Azure Data Lake Store, FILE, FTP, SFTP
Microsoft SQL ServerMSSQLMicrosoft SQL Server database. Can be configured for on-premise, also accessible via a Self-Hosted Integration Runtime. Also can be configured for cloud computing as a Managed Instance or Azure SQL database. This is done in the Linked Service configuration for the connection.ADONET, Microsoft MDS, OLEDB
MySqlMYSQLMySQL database.ADONET, ODBC, OLEDB
OracleORAOracle database.ADONET, ODBC, OLEDB, ORACLE
Oracle RDBORARDBOracle RDB Database. Discontinued product line from Oracle. Separate from there current database product.ADONET, ODBC, OLEDB, ORACLE
ORC FormatORCORC file format. Only supported for Azure Data Factory.Azure Blob Storage, Azure Data Lake Store, FILE, FTP, SFTP
Parquet FormatPARQParquet file format. Only supported for Azure Data Factory.Azure Blob Storage, Azure Data Lake Store, FILE, FTP, SFTP
PostgreSQLPGRPostgreSQL database.OLEDB
Rest ServiceRESTConfigurable REST API connector.Rest API
SalesforceSFDCAPI to connect to the generic Salesforce tenant.Rest API
Salesforce Marketing CloudSFDCMCAPI to connect to the Salesforce Marketing Cloud.Rest AP
Salesforce Service CloudSFDCSCAPI to connect to the Salesforce Service Cloud.Rest API
Snowflake Data WarehouseSFLDWSnowflake database.ODBC
SybaseSYBASESybase database.ADONET, ODBC, OLEDB
TeradataTDTeradata database.ADONET, ODBC, OLEDB, TERADATA
XML FormatXMLXML file format.Azure Blob Storage, Azure Data Lake Store, FILE, FTP, SFTP
Azure SQL DatabaseSQLDBAzure SQL Database - a fully managed PaaS database service in Azure. Supports both serverless and provisioned compute tiers. Configured for cloud connectivity via Azure-hosted resources.ADONET, OLEDB
Azure Key VaultAZKVAzure Key Vault for secrets and connection string management. Provides secure, centralized storage for credentials used by BimlFlex connections and integrations.Azure Key Vault
Custom TypeCUSTOMA custom system type for specialized integrations not covered by standard system types. Used with Custom Component connections for third-party or proprietary data sources.Custom Component
DatabricksDBRDatabricks unified analytics platform for data engineering and data science. Supports Delta Lake storage, SQL warehouses, and notebook-based transformations.Databricks
Dynamics Business CentralDYNBCMicrosoft Dynamics 365 Business Central ERP system. Connects via REST API for financial, sales, and operational data extraction.Rest API
Fabric WarehouseFBRDWMicrosoft Fabric Synapse Data Warehouse. A fully managed, cloud-scale SQL analytics warehouse within the Fabric ecosystem. Supports T-SQL and integrates with Fabric's unified data platform.Microsoft Fabric
Fabric LakehouseFBRLHMicrosoft Fabric Lakehouse combining data lake storage with warehouse capabilities. Uses Delta Lake format and supports both SQL and Spark-based analytics.Microsoft Fabric
Fabric SQL DatabaseFBRSQLMicrosoft Fabric SQL Database - a transactional SQL database within Fabric. Suitable for operational workloads that integrate with the broader Fabric ecosystem.Microsoft Fabric