Skip to content
Contact us Go to Customer Portal
  • There are no suggestions because the search field is empty.

Fabric Lakehouse - Stage

Applies to
Platform: Fabric Lakehouse
Architecture: Stage

Fabric Lakehouse - Stage generates a Stage layer for Microsoft Fabric, as notebooks on Delta tables. Select the Generator Configuration Fabric Lakehouse Stage when you add a project whose Stage layer reads lakehouse tables directly. With the project property Target Platform set to Azure Synapse, it generates for Azure Synapse Analytics instead.

For files use Fabric Lakehouse - Stage Files, for other databases Fabric Lakehouse - Stage JDBC.

Scope and output

  • Platform: with Target Platform Fabric, the default, each layer is a schema of a Microsoft Fabric lakehouse. With Azure Synapse, each layer is a database in Azure Synapse Analytics, which the deployment creates with CREATE DATABASE IF NOT EXISTS ... LOCATION.
  • Layers: Source and Stage. Dataflow Modeling shows Stage in the Bronze column. See Understand the Layering concept.
  • Output: notebooks in the folders Jupyter, Helpers and LoadControl, the notebook <project>_deploy and, with Target Platform Fabric, deploy.ps1; described in Fabric Lakehouse - Artifacts.

Prerequisites

  • Target Platform Fabric: a Fabric workspace with a lakehouse with schemas. The deployment creates a schema per layer.
  • Target Platform Fabric: a service principal as Member of the workspace, and PowerShell 7 for deploy.ps1.
  • Target Platform Azure Synapse: an Azure Synapse workspace and a storage location per layer.

The setup is described in Fabric Lakehouse - Target environment and Azure Synapse - Target environment, the placeholders in Fabric Lakehouse - Placeholder values.

Sources and linked projects

The source system has the type Spark. Stage source views read each source table as <workspace>.<lakehouse>.<schema>.<table> with placeholders of the source system. Discover Parquet or Delta files on OneLake creates the source model objects; Source Systems per Generator Configurations lists the discovery types.

You can export this project as a linked project. The configuration cannot import linked projects.

Supported model objects

Capabilities and limitations

  • Generated objects per Stage: the table stg_st_<model object name>, the views _source and _result, and a loader notebook.
  • Load: the loader replaces the rows of the table with INSERT OVERWRITE from the source view.
  • Loaders: one notebook per loader in the folder Jupyter, for example STG_ST_bdc_customer_Loader, with the load timestamp as the parameter bg_loadtimestamp.
  • Tables: Delta tables. File Format offers only delta, and Optimization Method offers Partitioned By and Cluster By.
  • Keys: Stage tables have no hash columns and no key constraints. Indexes: index management is not available.
  • Deployment: with the default Deployment Options, each deployment replaces the tables and the load control log tables with empty ones, see Redeploy and keep data. With Target Platform Fabric, it also creates a schema per layer.
  • Loading: LoadControl Options offers Multi-Threading Notebook and Airflow.

Project properties

The configuration offers no Hash Algorithm and no End of Universe. It offers these project properties, with their defaults:

Hash Casing, Hash Concatenation Separator, Hash Null Placeholder, Hash Trim and Persist Validity have no effect on Stage objects.

Data types

The Default Terms get these data types:

  • Load timestamp, Timestamp: BG_LoadTimestamp
  • Text, String: BG_SourceSystem, BG_DataflowName, BG_DataflowSetName, BG_IncrementLoadStatus
  • Number, Long: BG_ID

Platform differences

The table compares the configuration with Microsoft SQL Server - Stage of the same release:

ItemFabric LakehouseMicrosoft SQL Server
Timestamp Default TermsTimestampdatetimeoffset
Text Default TermsStringnvarchar
Persist Validity defaultonoff
Deployment Options defaultdrop tables, create schemasdrop tables
Index managementnot availableavailable

Both platforms offer the LoadControl Option Airflow. The native load control is Multi-Threading Notebook on Fabric Lakehouse.

Example

A Stage bdc_customer reads the source table bdc_customer. It generates this table and source view. The excerpt shows the example lakehouse my_lakehouse with the schema stage for the layer and bfm for the source tables, in place of the placeholders. An empty workspace name is removed with its dot:

CREATE OR REPLACE TABLE `my_lakehouse`.`stage`.`stg_st_bdc_customer` (
     `bg_loadtimestamp` Timestamp NOT NULL
    ,`bg_sourcesystem` String
    ,`bdc_shop_id` String NOT NULL
    ,`bdc_customer_id` String NOT NULL
    ,`first_name` String
    ,`last_name` String
    ,`date_of_birth` Date
    ,`location` String
)
USING delta

CREATE OR REPLACE VIEW `my_lakehouse`.`stage`.`stg_st_bdc_customer_source`
AS
SELECT
     CAST(NULL AS Timestamp) AS `bg_loadtimestamp`
    ,'source' AS `bg_sourcesystem`
    ,`s1`.`bdc_shop_id` AS `bdc_shop_id`
    ,`s1`.`bdc_customer_id` AS `bdc_customer_id`
    ,`s1`.`first_name` AS `first_name`
    ,`s1`.`last_name` AS `last_name`
    ,`s1`.`date_of_birth` AS `date_of_birth`
    ,`s1`.`location` AS `location`
FROM `my_lakehouse`.`bfm`.`bdc_customer` AS `s1`

The loader STG_ST_bdc_customer_Loader fills the table from this view and sets bg_loadtimestamp to its parameter. source is the name of the source system.

Release differences

Project Settings shows the Generator Version of your project.

  • Generator 2.0: Target Object Comment Type replaced the value Target Platform with Tables and Tables and Views, default None.
  • Generator 2.2: added the project properties Persist Validity, Create Result Views and the four Hash properties, and the model object property Increment Management Type on Stage. Deployment Options added Generate Create Schema If Not Exists Statements, selected by default, and LoadControl Options added Airflow.

Related articles