60 docs tagged with "metadata"

Drive SSIS processing packages, including stored procedure execution and SSAS database processing, from a tiny metadata model that scales to many tasks.

Drive root level OleDb connections and per package connection references from a metadata table so generated SSIS packages all share the same connection managers.

Define BI assets such as tables in one Biml file, then iterate them in a second tiered file to generate truncate-and-load packages from RootNode metadata.

Drive an SSIS staging load from SAP using a metadata repository, the Theobald XtractIS source, and BimlScript that reads the SAP DD03L data dictionary.

Use the DataFactory, LinkedServices, Datasets, Pipelines, and Triggers tags in BimlStudio to generate Azure Data Factory v2 JSON from source metadata.

Drive a Biml package generator from header and detail metadata tables that describe a base SELECT and a sequence of Lookup transforms for dimensional denormalization.

Use BimlScript to generate one staging package per delimited flat file by reading file metadata from a SQL table.

Attach extra metadata to Biml objects with annotations and object tags, then read that metadata in higher tier files.

How a metadata-driven Biml project replaces hand-built SSIS templates and keeps every package aligned with a single ETL framework.

Combine source metadata alerts, a Biml automation framework, and a simple SSIS testing harness to keep a data warehouse in sync with source systems still in development.

Drive SSIS package generation from a small metadata schema covering packages, connections, and connection types, then loop over the metadata to emit one package per row.

Add a custom C# extension method to the Biml object model so a BimlScript template can convert one node type into another with a single call.

Convert helper methods in a Biml C# code file into extension methods so they can be called directly on Biml objects.

Build SSIS Change Data Capture initial load and incremental load packages in Biml using the SQL Server CDC components as custom tasks.

Define ETL as relational metadata, then generate connection Biml from a stored procedure that returns connection name and connection string rows.

Extend a metadata-driven extraction package by emitting per-row containers with truncate, dataflow, and explicit source-to-target column mappings.

Generate one SSIS package per source table with BimlScript, then extend the script to apply per-table WHERE clauses from a metadata table.

Generate CREATE TABLE statements for an entire staging environment by reading INFORMATION_SCHEMA from the source database and emitting an ExecuteSQL task per table.

Define tables in Biml, then use BimlScript to generate an SSIS package that runs the CREATE TABLE statements against the target database.

How Biml extends an auto-generated Data Vault to handle business keys, satellite splits by fast-changing attributes, and per-attribute manual overrides driven from metadata.

Use ImportDB to materialize a SQL Server metadata schema as a DataSet inside BimlExpress, then drive connections, tables, and load packages from it.

Read a SQL Server meta schema with ImportDB, then emit a matching MetadataModel and MetadataInstance using the Biml Metadata feature in BimlStudio.

Use a custom metadata table to control which packages BimlScript generates and which optional transformations they include.

Use BimlScript to loop over every table in a source database, generate matching flat file formats and connections, and build a single package that exports each table to a CSV file in parallel.

Generate SSIS packages that split a large source table into multiple flat files using OFFSET FETCH and a Biml script that calls another Biml script.

Move Sage Sotamas 90 ProvideX data into SQL Server by generating destination tables from the vendor XSD, then producing one SSIS package per table with Biml.

Generate a Biml file that extracts every base table in a SQL Server database to its own raw file by querying INFORMATION_SCHEMA with FOR XML PATH.

Validation BimlScript that walks 'RootNode.Tables' and reports any table missing the project's required audit/metadata columns, with optional reporting of passing tables, so the check can run automatically before each build.

Reads a SQL Server schema's tables and column types, then emits one package per table with a matching FlatFileFormat and FlatFileConnection whose typed columns mirror the destination table, so flat-file imports get data-type validation at the connection manager.

Use BimlScript to generate Azure Data Factory v1 input and output dataset JSON files for hundreds of source tables driven from a metadata spreadsheet.

Use BimlScript and a small C# loop to emit one Data Profiling Task per table in a source database, producing per-table profile XML files in a single SSIS package.

Use BimlScript to read a list of source tables from system metadata and emit one extract and load package per table from a single template.

Build SELECT statements from source metadata using the Biml GetColumnList method, including aliases, predicates, and delimiter overrides.

Use BimlScript to generate T-SQL scripts from source metadata, including SELECT statements and CREATE VIEW definitions across hundreds of tables.

Tag every Biml managed table with an extended property and let the generation template emit a drop step for any tagged table no longer in the metadata.

Use GetQuerySchema to import column metadata from a SELECT query, view, table valued function, or stored procedure.

Use GetDatabaseSchema to pull schema, table, and view metadata from a source database into Biml so generated packages and SQL stay in step with the source.

Call GetDatabaseSchema on an ExcelConnection to materialize each worksheet as a Biml table and generate the matching staging DDL.

Preserve existing indexes on staging tables when generating drop and create DDL by importing the live target indexes back into the Biml table definition.

Complete LINQ reference for filtering, transforming, and querying Biml metadata

Generate Biml flat file formats, connections, and matching staging tables from bcp generated XML format files using a small set of extension methods.

Use a topology sort extension method to walk foreign key dependencies and generate SSIS packages that recreate and load tables in the right order.

Extend the topology aware load pattern so independent branches run in parallel while still respecting foreign key dependencies.

Reduce mixed XML and C# complexity in Biml files by splitting into includes, using CallBimlScript, and pushing string assembly into T-SQL stored procedures.

Architecture overview for a metadata driven SSIS solution that uses Biml and BimlScript to generate table load packages and a master package from a source to target mapping table.

Drive Biml table generation from a small metadata table in the target database, attach the source SELECT as an annotation, and emit the create plus load packages.

Extend the metadata layer with multiple source connections and per table column lists, then carry the source connection and column choice through annotations into the load package.

Build SSIS staging packages from a flat file using Biml, then convert hardcoded values into a metadata driven generator that reads file format and column details from a database.

Generate Derived Column and Lookup transformations inside an SSIS data flow from rows in a transformation metadata model.

Use BimlScript and ImportDB to reverse engineer table metadata out of an existing staging database into a metadata model that drives future Biml generation.

Use an ODBC text driver to expose a folder of flat files as tables, then read schema metadata to drive ETL generation.

Compare the Biml definition of a table against the live database before recreating it so unchanged objects keep their data and indexes.

Build SSIS packages that ProcessUpdate every dimension in an SSAS database, either as a single processing task or as one task per dimension inside a parallel container.

Use AMO inside BimlScript to find the most recently processed partitions in each SSAS measure group and emit a Partition Processing task that targets only those partitions.

Generate SSIS packages that process every dimension, the most recent fact partitions, and cube indexes in an SSAS database from a single Biml script.

Use LINQ OrderBy, Skip, and Take inside BimlScript to chunk a large set of source tables into evenly sized groups, each loaded by its own SSIS package.

Generate hundreds of pipelines from a single template using loops, conditionals, and metadata queries

Tag generated SSIS packages with Biml annotations and read those tags in later scripts to drive package grouping and master orchestration.

Use BimlScript to write one .sql file per table in RootNode.Tables containing the DROP and CREATE DDL for that table.

Drive multi-region SSIS deployment from a small set of metadata tables so the same source loads the same tables into per-region target databases.