Batch Editor
BimlFlex Batches group and help to define a data logistics workload. They are used by Projects to set execution grouping. A project is always associated to a batch, and a batch can be reference by more than one project. In a new BimlFlex solution, batches are often the second component to be defined. This is so that projects can reference them when they are created.
Batches are generated into data logistics 'batch' processes that, in turn, start multiple individual processes. Which processes are included in the batch depends on the project(s) that are associated with it. In typical BimlFlex solutions, a batch is configured to process source data into the staging layer and Data Vault (if configured). A separate batch usually processes the data for deliver using Data Marts.
The degree of parallelism for executing the individual processes can be managed through the batch configuration.
Batches are created and modified using the Batch Editor in the BimlFlex Application.
Overview
The following sections describe the User Interface elements of the Batch Editor and how they are used to author and manage BimlFlex Batches.
Note
Detailed descriptions of all Batch Editor fields and options are available in the Reference Documentation.
Batch Tab
The Batch Tab is the first tab in the Batch Editor, and it is selected by default. The batch tab focuses on general batch information and configuration. This tab is used to define and create the batch itself.
Action Buttons
| Icon | Action | Description |
|---|---|---|
 |
Save | This will save the current set of staged changes. The Save button is will only enable if the Batch has changes staged and there are no major validation issues with the current Batch properties. |
 |
Discard | This will Discard any unsaved changes and revert to last saved form. |
 |
Archive | This will hard delete the selected Batch. This will result in the physical removal of the selected record from the metadata database. The data will no longer be accessible by the BimlFlex Applicationand will require a Database Administrator to restore, if possible. Clicking Archive will create an Archive Batch Dialog. |
 |
Duplicate | This will create a duplicate of the selected Batch. A Duplicate Batch Dialog will appear asking for a name for the new Batch. A new Batch will be created using all of the selected Batch's current properties. |
 |
Deleted | This will soft delete the currently selected Batch. This will remove the Batch and all associated entities from processing and validation. For information on how to add excluded or deleted items back to the scope, please read our tips and tricks section. |
Note
A Batch itself can not be excluded, only deleted. If certain parts of the data solution should be excluded, then this should be applied at Project level in the Project Editor.
Additional Dialogs
Archive Batch Dialog
Confirmation warning against the dangers of archiving. You are required to confirm by both the check box and the Ok button.
Warning
Archiving is a permanent removal of the selected entity from its associated table in the BimlFlex Database. The best practice is to first use the Deleted flag (soft delete) as an indication that the connection may need to be removed.
Allowed Values
Precedence Constraints
| Value | Description |
|---|---|
| Success | Automatically create precedence constraints so that tasks will run only run one after the other if the prior tasks are successful. |
| Completion | Automatically create precedence constraints so that tasks will run one after the other after they are completed regardless of whether they succeed or fail. |
Objects Tab
The Objects Tab provides quick access to all Objects included in the Batch.
Action Buttons
The following options are available:
| Icon | Action | Description |
|---|---|---|
 |
Add | Add will create a new Object. Clicking Add will open an Add Object Dialog. |
|
Save | Save will save the currently set of staged changes. The Save button is only enabled if any Object has changes staged and when there are no validation issues. |
|
Discard | This will Discard any unsaved changes and revert to last saved details. |
When checking a single Object in the overview two additional options will become visible. When selecting two or more objects only the Archive option will be visible.
| Icon | Action | Description |
|---|---|---|
|
Archive | Archive will remove the Object from the active metadata repository, and move it to the metadata archive. Clicking Archive will create an Archive Object Dialog. |
 |
Edit | Edit will open an Edit Object Dialog to edit the currently selected Object. After edits are complete, be sure to click the Save button to confirm the changes. |
Additional Dialogs
Add Object Dialog
Creates a new Object. Once entry is finished ensure that the Save button is clicked.
Archive Object Dialog
Archiving means that metadata will be physically deleted. A confirmation box warning against the dangers of archiving. You are required to confirm by both the check box and the Ok button.
Warning
Archiving is a permanent removal of the selected entity from its associated table in the BimlFlex Database. The best practice is to first use the Deleted flag (soft delete) as an indication that the connection may need to be removed.
Objects Tab Views
The Objects Tab Views provide different ways to review the properties of Objects associated with the selected entity. The general purpose and overview of each view will be outlined below.
Note
See the Objects Editor for further details on creating or editing an Object along with details on individual property values.
Overview
The Overview is available to give quick access to the more physical properties of the Objects relating to the selected entity. Additional properties are available for bulk management such as controlling exclusions, disabling persistence, or flagging one or more Object(s) for deletion.
Various navigational transitions are available on view form and outlined below.
The following details are shown in this view:
| Component | Type | Description |
|---|---|---|
| Project | Value | The Project that the Object belongs to. See Projects Documentation. |
| Object | Value | The name of the Objects as associated with the current Connection. See the Object Editor documentation for more details. |
| Object Type | Value | The type of Object that is being extracted or created. See Valid Object Types. |
| Excluded | Checkbox (Tri-state) | Setting to determine whether the selected entity will be built out with the rest of the solution when the BimlFlex solution is being compiled. The state of being "partially checked" is based on the selection of its child elements (below). |
| Exclude Build | Checkbox (Y/N) | Toggle to determine whether the Object is excluded from the build when the BimlFlex solution is compiled. |
| Not Persistent | Checkbox (Y/N) | Toggle to determine whether the Object is excluded from Persistent Staging Area (PSA). When enables, the Object will not be stored in the PSA. |
| Deleted | Checkbox (Y/N) | Toggle to determine whether the Object is soft-deleted from the build when the BimlFlex solution is being compiled. |
Navigational Transitions
From the Objects Tab, it is possible to navigate directly to the following areas of BimlFlex:
| Item | Action |
|---|---|
| Project Value | Navigate to the Project Editor by clicking <img class="icon-inline" src="../../static/svg/navigate.svg style="width: 18px"/> |
| Object Value | Navigate to the Object Editor, selecting the clicked Object |
Note
The Object navigational transition is the only one provided on the additional views. As such, no further callouts will be made to this. Functionality on the below forms however is consistent: click the Object name to navigate to the selected Object in the Object Editor.
Model Overrides
The Model Overrides view allows for quick management of metadata configured through the Data Vault Accelerator, Business Modeling or the Schema Diagram. This view is intended as an easy way to review, reference, and maintain previously entered metadata.
The following details are shown in this view:
| Component | Type | Description |
|---|---|---|
| Object | Navigational Value | The name of the Objects as associated with the current Connection. See the Object Editor documentation for more details. |
| Accelerator Type | Value | The type of entity the target Object will be defined as for the Data Vault Accelerator. This information will determine to what Data Vault entity type this object will be mapped. See Data Vault Accelerator Documentation for more details. |
| Business Entity | Value | Displays the Business Entity that is associated with this Object. Use Business Modeling to define the Business Entities. See Business Modeling Documentation. |
| Business Name | Value | Displays the Business Name that is defined for this Object. This value will be used to drive the naming of the target object when using the Accelerator. For a source object with Accelerator Type Hub, the Business Name will be used for the Hub name, as well as the default Satellite name. |
| Business Subject | Value | Displays the Business Subject to which this Objects is added. A Business Subject is an additional way to classify Business Entities. A Business Subject may contain multiple Business Entities. Each Business Entity can be associated with a single Business Subject. See Business Modeling Documentation. |
Important
Although it is possible to manage high level metadata from this screen, it is recommended to you use the Data Vault Accelerator for implementing changes in the design. The Data Vault Accelerator is specifically designed to simplify and improve the experience of modeling metadata via a graphical designer as opposed to the table entry that this view provides.
Query Overrides
The Query Overrides view exposes any SQL overrides that may be defined for the Object. Any SQL that is commonly used to perform most ETL/ELT tasks are dynamically generated by the BimlFlex framework. In many cases providing SQL query overrides is not required. However, when necessary this view can be used to review these overrides across all objects.
The following details are shown in this view:
| Component | Type | Description |
|---|---|---|
| Object | Navigational Value | The name of the Objects as associated with the current Connection. See the Objects Editor documentation for more details. |
| From SQL | Field | Override the FROM clause of the generated SQL statement that is used to query the Object. |
| Join SQL | Field | If the source select requires a JOIN, this is where it can be entered. Note: This field is limited to 1,000 characters. |
| Where SQL | Field | Utilize if the source select requires a WHERE clause. Note: Defining a Parameter on the Object will be appended to the statement. |
| Group By SQL | Field | Utilize if the source select requires a DISTINCT or TOP N clause. Note: This field is limited to 1000 characters. |
| Override SQL | Field | Overrides the complete SQL statement used to query the Object. This must be a complete SQL statement. Adding an override here disregards any other SQL overrides, such as FROM SQL or WHERE SQL. |
See the Objects Editor documentation for additional information on the above fields and more.
Important
This is designed for developers/architects who have a strong understanding of SQL. Intellisense and code auto completing and validation are not provided in the BimlFlex Applicationbecause a direct connection to the data is not supported for security reasons.
Attributes Tab
The Attributes Tab provides a view of any Configurations or Setting overrides that have been applied to the selected Batch.
Note
This is exclusive to the Batch level. Additional overrides may be present on any grains higher or lower than the Batch.
Action Buttons
All Attributes assigned to the current entity show here. New items can be entered via the Add action button. Existing items can be edited by clicking the Value field to select the Attribute and using the Edit action button. Various navigational transitions are available on this form and outlined below.
The following options are available:
| Icon | Action | Description |
|---|---|---|
|
Add | Add will create a new Attribute and assign it to the current entity. When adding an attribute via this button, Attribute Type and the respective linked fields, will be pre-filled on the created Add Attribute Dialog. |
|
Save | Save will save the currently set of staged changes. The Save button is only enabled if any Attribute has changes staged and there are no major validation issues with the current list of Attribute properties. |
|
Discard | This will Discard any unsaved changes and revert to last saved form. |
When checking a single Attribute in the overview two additional options will become visible. When selecting two or more attributes only the Archive option will be visible.
| Icon | Action | Description |
|---|---|---|
|
Archive | Archive will hard delete the selected Attribute. This will result in the physical removal of the selected record from the BimlFlex database. The data will no longer be accessible by the BimlFlex Applicationand will require a Database Administrator to restore. Clicking Archive will create an Archive Attribute Dialog. |
|
Edit | Edit will open an Edit Attribute Dialog to edit the currently selected Attribute. After edits are complete, be sure to click the Save button to confirm the changes. |
Additional Dialogs
Add Attribute Dialog
Creates an Attribute and associates it with the current entity. Once entry is finished ensure that the Save button is clicked.
Edit Attribute Dialog
Edit the selected Attribute. Once entry is finished ensure that the Save button is clicked.
Archive Attribute Dialog
Confirmation warning against the dangers of archiving. You are required to confirm by both the check box and the Ok button.
Warning
Archiving is a permanent removal of the selected entity from its associated table in the BimlFlex Database. The best practice is to first use the Deleted flag (soft delete) as an indication that the connection may need to be removed.
Attribute Tab View
The Attribute Tab provides a different way to review the properties of Attributes associated with the selected entity.
Overview
The following details are shown in this view:
| Component | Type | Description |
|---|---|---|
| Object | Value | The name of the Objects as associated with the current Connection. See the Object Editor documentation for more details. |
| Column | Navigational Value | Define the scope of the Attribute. See the Attribute Editor for more details. |
| Attribute | Navigational Value | Define the scope of the BimlFlex process that the Attribute is applied to. Attribute Type is required and must be a valid Attribute Type. |
| Value | Value | The value to be returned when the custom attribute is accessed by the framework. See the Attribute Editor for more details. |
| Property | Field | Define the value for a custom attribute if it is not specified in the Attribute Value field. See the Attribute Editor for more details. |
Navigational Transitions
From the Attribute Tab, it is possible to navigate directly to the following areas of BimlFlex:
| Item | Action |
|---|---|
| Object Value | Navigate to the Object Editor, by clicking <img class="icon-inline" src="../../static/svg/navigate.svg style="width: 18px"/> |
| Column Value | Navigate to Column Editor, selecting clicked Column |
| Attribute Value | Navigate to Attribute Editor, selecting clicked Attribute |