--- title: Databricks - Datamart description: Databricks - Datamart generates a Mart layer from the objects of linked projects for Databricks as notebooks with SQL statements on Delta tables. --- [Skip to content](https://knowledge.bigenius-x.com/databricks-datamart#main-content) [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en) [![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/) Open main navigation Close main navigation - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new) - [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal) How can we help you? - There are no suggestions because the search field is empty. 1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en) 2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en) 3. [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) # Databricks - Datamart **Applies to** Platform: Databricks Architecture: Data Mart Databricks - Datamart generates a Mart layer from the objects of linked projects for Databricks as notebooks with SQL statements on Delta tables. Select the Generator Configuration `Databricks DataMart` when you add a project whose data marts run on Databricks. For dbt, select `Databricks Dbt DataMart` instead, see [Databricks dbt - Datamart](https://knowledge.bigenius-x.com/databricks-dbt-datamart?hsLang=en). ### Scope and output - Platform: Databricks. Each layer is a schema of a Unity Catalog catalog or, with [Use Unity Catalog](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en) off, a Hive metastore database. - Layers: Mart, and the layers of the imported linked projects. Dataflow Modeling shows the imported objects in the Linked column in Silver and the Mart model objects in the Datamart column, which is Gold. See [Understand the Layering concept](https://knowledge.bigenius-x.com/understand-the-layering-concept?hsLang=en). - Output: notebooks in `Jupyter` and `LoadControl`, and `deploy.ps1`, described in [Databricks - Artifacts](https://knowledge.bigenius-x.com/databricks-artifacts?hsLang=en). ### Prerequisites - A Databricks workspace with a catalog for the project, or with the Hive metastore a storage location per layer. The deployment creates the schemas or databases. - An all-purpose cluster and a user who can create tables and run jobs on it, and read the schemas of the linked projects. - PowerShell 7 and the Databricks CLI for `deploy.ps1`. The setup is described in [Databricks - Target environment with Unity Catalog](https://knowledge.bigenius-x.com/databricks-unity-target-environment?hsLang=en) and [Databricks - Target environment with the Hive metastore](https://knowledge.bigenius-x.com/databricks-target-environment?hsLang=en), the placeholders in [Databricks - Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en). ### Sources and linked projects The configuration reads only model objects of linked projects, for example of a Data Vault or Data Store project. It has no discovery of its own. [Import a linked project](https://knowledge.bigenius-x.com/import-a-project-to-be-used-as-a-linked-project?hsLang=en) before you model the Mart layer. You can also [export this project as a linked project](https://knowledge.bigenius-x.com/export-a-project-to-be-used-as-linked-project?hsLang=en). Mart source views read the layers of the linked project with their own catalog and schema placeholders. ### Supported model objects - [Mart Dimension](https://knowledge.bigenius-x.com/create-a-mart-dimension?hsLang=en) - [Mart Fact](https://knowledge.bigenius-x.com/create-a-mart-fact?hsLang=en) - [Mart Composite](https://knowledge.bigenius-x.com/create-a-mart-composite?hsLang=en) - [Mart Hierarchy Flatten](https://knowledge.bigenius-x.com/create-a-mart-hierachy-flatten?hsLang=en) - [Mart Out](https://knowledge.bigenius-x.com/create-a-mart-out?hsLang=en), since Generator 2.2 ### Capabilities and limitations - Loaders: one notebook per loader in the folder `Jupyter`, for example `DM_MD_bk_multipart_not_nullable_Loader`, with the load timestamp as the parameter `bg_loadtimestamp`. - Tables: Delta tables. [File Format](https://knowledge.bigenius-x.com/file-format-model-object?hsLang=en) offers only `delta`, and [Optimization Method](https://knowledge.bigenius-x.com/optimization-method-model-object?hsLang=en) offers `Partitioned By` and `Cluster By`. - Keys, with Unity Catalog only: a Mart Dimension has its identity column `_sk` as primary key and, with Use Unique Constraints, a UNIQUE constraint on its business key terms. Databricks does not enforce them, see [Constraints on Databricks](https://docs.databricks.com/aws/en/tables/constraints). Databricks Runtime 17.3 LTS rejects UNIQUE constraints, see [Databricks - Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en#a-table-with-a-unique-constraint-is-not-created). - Foreign keys: [Foreign Key Mode](https://knowledge.bigenius-x.com/foreign-key-mode-model-object?hsLang=en) offers `None` and `Create Disabled` (default), not `Create Enforced`. `Create Disabled` adds the constraint as `NOT ENFORCED`. - Unknown member: the deployment adds a row with the surrogate key `-1` to each Mart Dimension. [Custom Singleton Values](https://knowledge.bigenius-x.com/custom-singleton-values-model-object?hsLang=en) sets its values. - Hashing: row hashes are `UNHEX(SHA2(..., 256))` in `Binary` columns, see [How biGENIUS-X builds hash values](https://knowledge.bigenius-x.com/how-bigenius-x-builds-hash-values?hsLang=en). - Validity: with Persist Validity on (default), historized tables store `bg_validtotimestamp`; off, the result view computes it. - Deployment: with the default [Deployment Options](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en), each deployment replaces the tables and the load control log tables with empty ones, see [Redeploy and keep data](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en#redeploy-and-keep-data) in [Databricks - Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en). - Governance: [Apply Sensitivity Level Tags](https://knowledge.bigenius-x.com/apply-sensitivity-level-tags-project?hsLang=en) adds a tags notebook that the deployment does not run. ### Project properties The configuration offers these project properties, with their defaults: - [Hash Algorithm](https://knowledge.bigenius-x.com/hash-algorithm-project?hsLang=en): `SHA-256`, the only value - [Hash Casing](https://knowledge.bigenius-x.com/hash-casing-project?hsLang=en): `None`; also `Upper` and `Lower` - [Hash Concatenation Separator](https://knowledge.bigenius-x.com/hash-concatenation-separator-project?hsLang=en): `_` - [Hash Null Placeholder](https://knowledge.bigenius-x.com/hash-null-placeholder-project?hsLang=en): `_NULL_` - [Hash Trim](https://knowledge.bigenius-x.com/hash-trim-project?hsLang=en): off - [Begin of Universe](https://knowledge.bigenius-x.com/begin-of-universe-project?hsLang=en): `1900-01-01` - [End of Universe](https://knowledge.bigenius-x.com/end-of-universe-project?hsLang=en): `9999-12-31` - [Persist Validity](https://knowledge.bigenius-x.com/persist-validity-project?hsLang=en): on - [Create Result Views](https://knowledge.bigenius-x.com/create-result-views-project?hsLang=en): on - [Target Object Comment Type](https://knowledge.bigenius-x.com/target-object-comment-type-project?hsLang=en): `None`; also `Tables` and `Tables and Views` - [Use Unity Catalog](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en): on - [Use Unique Constraints](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en): on - [Deployment Options](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en): `Generate Drop Table Statements` and `Generate Create Schema If Not Exists Statements`; also `Generate Alter View Body Statements` - [Notebook Deployment Objects Limit](https://knowledge.bigenius-x.com/notebook-deployment-objects-limit-project?hsLang=en): `100` - [Apply Sensitivity Level Tags](https://knowledge.bigenius-x.com/apply-sensitivity-level-tags-project?hsLang=en): off - [Sensitivity Level Tag Name](https://knowledge.bigenius-x.com/sensitivity-level-tag-name-project?hsLang=en): `sensitivity_level` - [Configure LoadControl](https://knowledge.bigenius-x.com/configure-loadcontrol-project?hsLang=en): on - [Deploy LoadControl](https://knowledge.bigenius-x.com/deploy-loadcontrol-project?hsLang=en): on - [LoadControl Options](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en): `Multi-Threading Notebook`; also `Airflow` - [Logging Method](https://knowledge.bigenius-x.com/logging-method-project?hsLang=en): `Delta Table`; also `Python Logger` ### Data types The Default Terms get these data types: - Timestamps, `Timestamp`: `BG_LoadTimestamp`, `BG_UpdateTimestamp`, `BG_ValidFromTimestamp`, `BG_ValidToTimestamp`, `BG_EffectiveTimestamp` - Row hashes, `Binary`: `BG_RowHash`, `BG_RowHashScd1`, `BG_RowHashScd2` - Text, `String`: `BG_SourceSystem`, `BG_DataflowName`, `BG_DataflowSetName`, `BG_ErrorDescription`, `BG_IncrementLoadStatus` - Numbers, `Long`: `BG_ID`, `BG_DataQuality_ID`, `BG_MartDimension_Identity`, `BG_MartFact_ForeignKey_Identity`; `Integer`: `BG_ErrorCount` - Flags, `Integer`: `BG_IsDeleted` ### Platform differences The table compares the configuration with [Microsoft SQL Server - Datamart](https://knowledge.bigenius-x.com/microsoft-sql-server-datamart?hsLang=en) of the same release: | Item | Databricks | Microsoft SQL Server | | --- | --- | --- | | Timestamp Default Terms | `Timestamp` | `datetimeoffset` | | Hash Default Terms | `Binary` | `binary(20)` | | Text Default Terms | `String` | `nvarchar` | | Hash Algorithm default | `SHA-256` | `SHA1` | | Persist Validity default | on | off | | Deployment Options default | drop tables, create schemas | drop tables | | Enforced foreign keys | not available | available | | Index management | not available | available | Both platforms offer the LoadControl Option `Airflow`. The native load control is `Multi-Threading Notebook` on Databricks. ### Example A Mart Dimension `bk_multipart_not_nullable` with the business key terms `bdc_shop_id` and `bdc_customer_id` generates this table. The excerpt shows the example catalog `dwh` and schema `mart` in place of the placeholders: ``` CREATE OR REPLACE TABLE `dwh`.`mart`.`dm_md_bk_multipart_not_nullable` ( `bg_sourcesystem` String ,`bg_loadtimestamp` Timestamp NOT NULL ,`bk_multipart_not_nullable_sk` Long NOT NULL GENERATED BY DEFAULT AS IDENTITY (START WITH 1 INCREMENT BY 1) ,`bdc_shop_id` String NOT NULL ,`bdc_customer_id` String NOT NULL ,`first_name` String ,`last_name` String ,`date_of_birth` Date ,`location` String ,`bg_rowhashscd1` Binary NOT NULL ,CONSTRAINT `pk_dm_md_bk_multipart_not_nullable` PRIMARY KEY (`bk_multipart_not_nullable_sk`) ,CONSTRAINT `uc_dm_md_bk_multipart_not_nullable` UNIQUE (`bdc_shop_id`, `bdc_customer_id`) ) USING delta ``` The deployment then inserts the unknown-member row with `bk_multipart_not_nullable_sk` `-1`, the text `Unknown` and the date `1900-01-01`. Its source view reads `bdv_current_bdc_customer` of the linked Data Vault project. ### Release differences Project Settings shows the Generator Version of your project. - Generator 2.0: Use Unity Catalog is on by default, off up to Generator 1.10. Target Object Comment Type replaced `Target Platform` with `Tables` and `Tables and Views`, default `None`. Deployment Options added `Generate Alter View Body Statements`. - Generator 2.2: added Mart Out, the project properties Persist Validity, Create Result Views, the four Hash properties and the two Sensitivity Level properties, and the model object property Increment Management Type on Mart Fact. Deployment Options added `Generate Create Schema If Not Exists Statements`, selected by default. - Generator 2.3: added the model object property [Custom Singleton Values](https://knowledge.bigenius-x.com/custom-singleton-values-model-object?hsLang=en) on Mart Dimension. - Generator 2.4: added the project property [Use Unique Constraints](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en). - Generator 1.10: the project property [Delta Detection](https://knowledge.bigenius-x.com/delta-detection-model-object?hsLang=en). Since Generator 2.0, Delta Detection is a property of the Mart Dimension. ### Related articles - Previous: - Databricks: [Target environment with Unity Catalog](https://knowledge.bigenius-x.com/databricks-unity-target-environment?hsLang=en), [Target environment with the Hive metastore](https://knowledge.bigenius-x.com/databricks-target-environment?hsLang=en) - Next: [Generate artifacts](https://knowledge.bigenius-x.com/generate-artifacts?hsLang=en), [Replace the placeholders in the artifacts](https://knowledge.bigenius-x.com/replace-the-placeholders-in-the-artifacts?hsLang=en) - Databricks: [Artifacts](https://knowledge.bigenius-x.com/databricks-artifacts?hsLang=en), [Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en), [Load data - Native Load Control](https://knowledge.bigenius-x.com/databricks-load-data-native-load-control?hsLang=en), [Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/databricks-load-data-apache-airflow-load-control?hsLang=en) - Reference: [Foreign Key Mode (Model Object)](https://knowledge.bigenius-x.com/foreign-key-mode-model-object?hsLang=en), [Custom Singleton Values (Model Object)](https://knowledge.bigenius-x.com/custom-singleton-values-model-object?hsLang=en), [Delta Detection (Model Object)](https://knowledge.bigenius-x.com/delta-detection-model-object?hsLang=en), [Use Unity Catalog (Project)](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en), [Use Unique Constraints (Project)](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en), [Deployment Options (Project)](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en), [Notebook Deployment Objects Limit (Project)](https://knowledge.bigenius-x.com/notebook-deployment-objects-limit-project?hsLang=en), [LoadControl Options (Project)](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en), [Optimization Method (Model Object)](https://knowledge.bigenius-x.com/optimization-method-model-object?hsLang=en), [BG\_MartDimension\_Identity](https://knowledge.bigenius-x.com/bg_martdimension_identity?hsLang=en), [BG\_MartFact\_ForeignKey\_Identity](https://knowledge.bigenius-x.com/bg_martfact_foreignkey_identity?hsLang=en) - Databricks: [Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en) - See also: [Available Generator Configurations](https://knowledge.bigenius-x.com/available-generator-configurations?hsLang=en), [Understand the Generator Configuration](https://knowledge.bigenius-x.com/understand-the-generator-configuration?hsLang=en), [Source Systems per Generator Configurations](https://knowledge.bigenius-x.com/source-systems-per-generator-configurations?hsLang=en), [Use Linked Projects](https://knowledge.bigenius-x.com/use-linked-projects?hsLang=en), [Microsoft SQL Server - Datamart](https://knowledge.bigenius-x.com/microsoft-sql-server-datamart?hsLang=en) - Databricks: [DataVault](https://knowledge.bigenius-x.com/databricks-datavault?hsLang=en), [DataVault and Mart](https://knowledge.bigenius-x.com/databricks-datavault-and-mart?hsLang=en), [Data Store](https://knowledge.bigenius-x.com/databricks-data-store?hsLang=en), [Dimensional and Mart](https://knowledge.bigenius-x.com/databricks-dimensional-and-mart?hsLang=en) - Databricks dbt: [Datamart](https://knowledge.bigenius-x.com/databricks-dbt-datamart?hsLang=en), [Target environment](https://knowledge.bigenius-x.com/databricks-dbt-target-environment?hsLang=en) - [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content) - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines) - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working) - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account) - [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en) - [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en) - [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content) - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration) - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions) - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features) - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects) - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches) - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections) - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview) - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps) - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms) - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping) - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships) - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality) - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes) - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling) - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts) - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings) - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace) - [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content) - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse) - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse) - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake) - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server) - [Oracle](https://knowledge.bigenius-x.com/generators?hsLang=en#oracle) - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts) - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders) - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment) - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment) - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline) - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment) - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment) - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control) - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type) - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties) - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms) - [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content) - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations) - [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content) - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches) - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases) - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules) - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules) - [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en) - [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content) - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application) - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application) - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators) - [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content) - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs) - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits) - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) - [Customer Portal](https://knowledge.bigenius-x.com/customer-portal) - [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login) [![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com) All rights reserved © 2025, biGENIUS AG