--- title: Databricks - Stage description: Databricks - Stage generates a Stage layer for Databricks as notebooks with SQL statements on Delta tables. --- [Skip to content](https://knowledge.bigenius-x.com/databricks-stage#main-content) [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en) [![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/) Open main navigation Close main navigation - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new) - [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal) How can we help you? - There are no suggestions because the search field is empty. 1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en) 2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en) 3. [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) # Databricks - Stage **Applies to** Platform: Databricks Architecture: Stage Databricks - Stage generates a Stage layer for Databricks as notebooks with SQL statements on Delta tables. Select the Generator Configuration `Databricks Stage` when you add a project whose Stage layer reads tables of a Databricks catalog directly. For files use [Databricks - Stage Files](https://knowledge.bigenius-x.com/databricks-stage-files?hsLang=en), for other databases [Databricks - Stage JDBC](https://knowledge.bigenius-x.com/databricks-stage-jdbc?hsLang=en). ### Scope and output - Platform: Databricks. Each layer is a schema of a Unity Catalog catalog or, with [Use Unity Catalog](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en) off, a Hive metastore database. - Layers: Source and Stage. Dataflow Modeling shows Stage in the Bronze column. See [Understand the Layering concept](https://knowledge.bigenius-x.com/understand-the-layering-concept?hsLang=en). - Output: notebooks in `Jupyter` and `LoadControl`, and `deploy.ps1`, described in [Databricks - Artifacts](https://knowledge.bigenius-x.com/databricks-artifacts?hsLang=en). ### Prerequisites - A Databricks workspace with a catalog for the project, or with the Hive metastore a storage location per layer. The deployment creates the schemas or databases. - An all-purpose cluster and a user who can create tables and run jobs on it, and read the source tables. - PowerShell 7 and the Databricks CLI for `deploy.ps1`. The setup is described in [Databricks - Target environment with Unity Catalog](https://knowledge.bigenius-x.com/databricks-unity-target-environment?hsLang=en) and [Databricks - Target environment with the Hive metastore](https://knowledge.bigenius-x.com/databricks-target-environment?hsLang=en), the placeholders in [Databricks - Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en). ### Sources and linked projects Stage source views read each source table as `..` with placeholders of the source system. [Discover a Databricks Unity Catalog](https://knowledge.bigenius-x.com/discover-a-databricks-unity-catalog?hsLang=en) creates the source model objects; [Source Systems per Generator Configurations](https://knowledge.bigenius-x.com/source-systems-per-generator-configurations?hsLang=en) lists the discovery types. You can [export this project as a linked project](https://knowledge.bigenius-x.com/export-a-project-to-be-used-as-linked-project?hsLang=en). The configuration cannot import linked projects. ### Supported model objects - [Source](https://knowledge.bigenius-x.com/create-a-source?hsLang=en) - [Stage](https://knowledge.bigenius-x.com/create-a-stage?hsLang=en) ### Capabilities and limitations - Generated objects per Stage: the table `stg_st_`, the views `_source` and `_result`, and a loader notebook. - Load: the loader replaces the rows of the table with `INSERT OVERWRITE` from the source view. - Loaders: one notebook per loader in the folder `Jupyter`, for example `STG_ST_bdc_customer_Loader`, with the load timestamp as the parameter `bg_loadtimestamp`. - Tables: Delta tables. [File Format](https://knowledge.bigenius-x.com/file-format-model-object?hsLang=en) offers only `delta`, and [Optimization Method](https://knowledge.bigenius-x.com/optimization-method-model-object?hsLang=en) offers `Partitioned By` and `Cluster By`. - Keys: Stage tables have no hash columns and no key on the business key terms. Indexes: index management is not available. - Deployment: with the default [Deployment Options](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en), each deployment replaces the tables and the load control log tables with empty ones, see [Redeploy and keep data](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en#redeploy-and-keep-data) in [Databricks - Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en). - Governance: [Apply Sensitivity Level Tags](https://knowledge.bigenius-x.com/apply-sensitivity-level-tags-project?hsLang=en) adds a tags notebook that the deployment does not run. ### Project properties The configuration offers no Hash Algorithm and no End of Universe. It offers these project properties, with their defaults: - [Hash Casing](https://knowledge.bigenius-x.com/hash-casing-project?hsLang=en): `None`; also `Upper` and `Lower` - [Hash Concatenation Separator](https://knowledge.bigenius-x.com/hash-concatenation-separator-project?hsLang=en): `_` - [Hash Null Placeholder](https://knowledge.bigenius-x.com/hash-null-placeholder-project?hsLang=en): `_NULL_` - [Hash Trim](https://knowledge.bigenius-x.com/hash-trim-project?hsLang=en): off - [Begin of Universe](https://knowledge.bigenius-x.com/begin-of-universe-project?hsLang=en): `1900-01-01` - [Persist Validity](https://knowledge.bigenius-x.com/persist-validity-project?hsLang=en): on - [Create Result Views](https://knowledge.bigenius-x.com/create-result-views-project?hsLang=en): on - [Target Object Comment Type](https://knowledge.bigenius-x.com/target-object-comment-type-project?hsLang=en): `None`; also `Tables` and `Tables and Views` - [Use Unity Catalog](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en): on - [Use Unique Constraints](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en): on - [Deployment Options](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en): `Generate Drop Table Statements` and `Generate Create Schema If Not Exists Statements`; also `Generate Alter View Body Statements` - [Notebook Deployment Objects Limit](https://knowledge.bigenius-x.com/notebook-deployment-objects-limit-project?hsLang=en): `100` - [Apply Sensitivity Level Tags](https://knowledge.bigenius-x.com/apply-sensitivity-level-tags-project?hsLang=en): off - [Sensitivity Level Tag Name](https://knowledge.bigenius-x.com/sensitivity-level-tag-name-project?hsLang=en): `sensitivity_level` - [Configure LoadControl](https://knowledge.bigenius-x.com/configure-loadcontrol-project?hsLang=en): on - [Deploy LoadControl](https://knowledge.bigenius-x.com/deploy-loadcontrol-project?hsLang=en): on - [LoadControl Options](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en): `Multi-Threading Notebook`; also `Airflow` - [Logging Method](https://knowledge.bigenius-x.com/logging-method-project?hsLang=en): `Delta Table`; also `Python Logger` Hash Casing, Hash Concatenation Separator, Hash Null Placeholder, Hash Trim and Persist Validity have no effect on Stage objects. ### Data types The Default Terms get these data types: - Load timestamp, `Timestamp`: `BG_LoadTimestamp` - Text, `String`: `BG_SourceSystem`, `BG_DataflowName`, `BG_DataflowSetName`, `BG_IncrementLoadStatus` - Number, `Long`: `BG_ID` ### Platform differences The table compares the configuration with [Microsoft SQL Server - Stage](https://knowledge.bigenius-x.com/microsoft-sql-server-stage?hsLang=en) of the same release: | Item | Databricks | Microsoft SQL Server | | --- | --- | --- | | Timestamp Default Terms | `Timestamp` | `datetimeoffset` | | Text Default Terms | `String` | `nvarchar` | | Persist Validity default | on | off | | Deployment Options default | drop tables, create schemas | drop tables | | Index management | not available | available | Both platforms offer the LoadControl Option `Airflow`. The native load control is `Multi-Threading Notebook` on Databricks. ### Example A Stage `bdc_customer` reads the source table `bdc_customer`. It generates this table and source view. The excerpt shows the example catalog `dwh` with the schema `stage`, and the source catalog `src` with the schema `bfm`, in place of the placeholders: ``` CREATE OR REPLACE TABLE `dwh`.`stage`.`stg_st_bdc_customer` ( `bg_loadtimestamp` Timestamp NOT NULL ,`bg_sourcesystem` String ,`bdc_shop_id` String NOT NULL ,`bdc_customer_id` String NOT NULL ,`first_name` String ,`last_name` String ,`date_of_birth` Date ,`location` String ) USING delta CREATE OR REPLACE VIEW `dwh`.`stage`.`stg_st_bdc_customer_source` AS SELECT CAST(NULL AS Timestamp) AS `bg_loadtimestamp` ,'source' AS `bg_sourcesystem` ,`s1`.`bdc_shop_id` AS `bdc_shop_id` ,`s1`.`bdc_customer_id` AS `bdc_customer_id` ,`s1`.`first_name` AS `first_name` ,`s1`.`last_name` AS `last_name` ,`s1`.`date_of_birth` AS `date_of_birth` ,`s1`.`location` AS `location` FROM `src`.`bfm`.`bdc_customer` AS `s1` ``` The loader `STG_ST_bdc_customer_Loader` fills the table from this view and sets `bg_loadtimestamp` to its parameter. `source` is the name of the source system. ### Release differences Project Settings shows the Generator Version of your project. - Generator 2.0: Use Unity Catalog is on by default, off up to Generator 1.10. Target Object Comment Type replaced `Target Platform` with `Tables` and `Tables and Views`, default `None`. Deployment Options added `Generate Alter View Body Statements`. - Generator 2.2: added the project properties Persist Validity, Create Result Views, the four Hash properties and the two Sensitivity Level properties, and the model object property Increment Management Type on Stage. Deployment Options added `Generate Create Schema If Not Exists Statements`, selected by default. - Generator 2.4: added the project property [Use Unique Constraints](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en). ### Related articles - Previous: - Databricks: [Target environment with Unity Catalog](https://knowledge.bigenius-x.com/databricks-unity-target-environment?hsLang=en), [Target environment with the Hive metastore](https://knowledge.bigenius-x.com/databricks-target-environment?hsLang=en) - Next: [Generate artifacts](https://knowledge.bigenius-x.com/generate-artifacts?hsLang=en), [Replace the placeholders in the artifacts](https://knowledge.bigenius-x.com/replace-the-placeholders-in-the-artifacts?hsLang=en) - Databricks: [Artifacts](https://knowledge.bigenius-x.com/databricks-artifacts?hsLang=en), [Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en), [Load data - Native Load Control](https://knowledge.bigenius-x.com/databricks-load-data-native-load-control?hsLang=en), [Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/databricks-load-data-apache-airflow-load-control?hsLang=en) - Reference: [Use Unity Catalog (Project)](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en), [Use Unique Constraints (Project)](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en), [Deployment Options (Project)](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en), [Notebook Deployment Objects Limit (Project)](https://knowledge.bigenius-x.com/notebook-deployment-objects-limit-project?hsLang=en), [LoadControl Options (Project)](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en), [Optimization Method (Model Object)](https://knowledge.bigenius-x.com/optimization-method-model-object?hsLang=en) - Databricks: [Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en) - See also: [Available Generator Configurations](https://knowledge.bigenius-x.com/available-generator-configurations?hsLang=en), [Understand the Generator Configuration](https://knowledge.bigenius-x.com/understand-the-generator-configuration?hsLang=en), [Source Systems per Generator Configurations](https://knowledge.bigenius-x.com/source-systems-per-generator-configurations?hsLang=en), [Microsoft SQL Server - Stage](https://knowledge.bigenius-x.com/microsoft-sql-server-stage?hsLang=en) - Databricks: [Stage Files](https://knowledge.bigenius-x.com/databricks-stage-files?hsLang=en), [Stage JDBC](https://knowledge.bigenius-x.com/databricks-stage-jdbc?hsLang=en), [Data Store](https://knowledge.bigenius-x.com/databricks-data-store?hsLang=en), [DataVault](https://knowledge.bigenius-x.com/databricks-datavault?hsLang=en), [Dimensional](https://knowledge.bigenius-x.com/databricks-dimensional?hsLang=en) - [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content) - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines) - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working) - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account) - [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en) - [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en) - [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content) - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration) - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions) - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features) - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects) - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches) - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections) - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview) - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps) - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms) - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping) - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships) - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality) - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes) - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling) - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts) - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings) - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace) - [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content) - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse) - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse) - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake) - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server) - [Oracle](https://knowledge.bigenius-x.com/generators?hsLang=en#oracle) - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts) - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders) - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment) - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment) - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline) - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment) - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment) - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control) - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type) - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties) - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms) - [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content) - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations) - [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content) - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches) - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases) - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules) - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules) - [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en) - [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content) - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application) - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application) - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators) - [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content) - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs) - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits) - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) - [Customer Portal](https://knowledge.bigenius-x.com/customer-portal) - [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login) [![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com) All rights reserved © 2025, biGENIUS AG