--- title: Databricks - Data Store description: Databricks - Data Store generates a Stage and a Store layer for Databricks as notebooks with SQL statements on Delta tables. --- [Skip to content](https://knowledge.bigenius-x.com/databricks-data-store#main-content) [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en) [![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/) Open main navigation Close main navigation - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new) - [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal) How can we help you? - There are no suggestions because the search field is empty. 1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en) 2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en) 3. [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) # Databricks - Data Store **Applies to** Platform: Databricks Architecture: Data Store Databricks - Data Store generates a Stage and a Store layer for Databricks as notebooks with SQL statements on Delta tables. Select the Generator Configuration `Databricks DataStore` when you add a project whose Data Store runs on Databricks. ### Scope and output - Platform: Databricks. Each layer is a schema of a Unity Catalog catalog or, with [Use Unity Catalog](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en) off, a Hive metastore database. - Layers: Source, Stage and Store. Dataflow Modeling shows Stage and the Data Store in the Bronze column. See [Understand the Layering concept](https://knowledge.bigenius-x.com/understand-the-layering-concept?hsLang=en). - Output: notebooks in `Jupyter` and `LoadControl`, and `deploy.ps1`, described in [Databricks - Artifacts](https://knowledge.bigenius-x.com/databricks-artifacts?hsLang=en). ### Prerequisites - A Databricks workspace with a catalog for the project, or with the Hive metastore a storage location per layer. The deployment creates the schemas or databases. - An all-purpose cluster and a user who can create tables and run jobs on it, and read the source tables. - PowerShell 7 and the Databricks CLI for `deploy.ps1`. The setup is described in [Databricks - Target environment with Unity Catalog](https://knowledge.bigenius-x.com/databricks-unity-target-environment?hsLang=en) and [Databricks - Target environment with the Hive metastore](https://knowledge.bigenius-x.com/databricks-target-environment?hsLang=en), the placeholders in [Databricks - Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en). ### Sources and linked projects Stage source views read each source table as `..` with placeholders of the source system. [Discover a Databricks Unity Catalog](https://knowledge.bigenius-x.com/discover-a-databricks-unity-catalog?hsLang=en) creates the source model objects; [Source Systems per Generator Configurations](https://knowledge.bigenius-x.com/source-systems-per-generator-configurations?hsLang=en) lists the discovery types. You can [import a linked project](https://knowledge.bigenius-x.com/import-a-project-to-be-used-as-a-linked-project?hsLang=en), for example a [Databricks - Stage Files](https://knowledge.bigenius-x.com/databricks-stage-files?hsLang=en) or [Databricks - Stage JDBC](https://knowledge.bigenius-x.com/databricks-stage-jdbc?hsLang=en) project, and [export this project as a linked project](https://knowledge.bigenius-x.com/export-a-project-to-be-used-as-linked-project?hsLang=en). ### Supported model objects - [Source](https://knowledge.bigenius-x.com/create-a-source?hsLang=en) - [Stage](https://knowledge.bigenius-x.com/create-a-stage?hsLang=en) - [Store Entity](https://knowledge.bigenius-x.com/create-a-store-entity?hsLang=en) - [Store Composite](https://knowledge.bigenius-x.com/create-a-store-composite?hsLang=en) ### Capabilities and limitations - Loaders: one notebook per loader in the folder `Jupyter`, for example `DS_SE_bk_multipart_not_nullable_Loader`, with the load timestamp as the parameter `bg_loadtimestamp`. - Tables: Delta tables. [File Format](https://knowledge.bigenius-x.com/file-format-model-object?hsLang=en) offers only `delta`, and [Optimization Method](https://knowledge.bigenius-x.com/optimization-method-model-object?hsLang=en) offers `Partitioned By` and `Cluster By`. - Keys, with Unity Catalog only: a Store Entity has its business key as primary key; a historized one adds `bg_validfromtimestamp`. With a nullable business key term, the key is a UNIQUE constraint instead. Databricks does not enforce them, see [Constraints on Databricks](https://docs.databricks.com/aws/en/tables/constraints). Databricks Runtime 17.3 LTS rejects UNIQUE constraints, see [Databricks - Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en#a-table-with-a-unique-constraint-is-not-created). - Delete detection: [Configure delete detection](https://knowledge.bigenius-x.com/configure-a-delete-detection-behavior?hsLang=en) adds the column `bg_isdeleted` and a delete detection loader. - Data quality: Store Entity supports [data quality rules](https://knowledge.bigenius-x.com/understand-data-quality-rules-overview?hsLang=en). - Hashing: row hashes are `UNHEX(SHA2(..., 256))` in `Binary` columns, see [How biGENIUS-X builds hash values](https://knowledge.bigenius-x.com/how-bigenius-x-builds-hash-values?hsLang=en). - Validity: with Persist Validity on (default), historized tables store `bg_validtotimestamp`; off, the result view computes it. - Deployment: with the default [Deployment Options](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en), each deployment replaces the tables and the load control log tables with empty ones, see [Redeploy and keep data](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en#redeploy-and-keep-data) in [Databricks - Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en). - Governance: [Apply Sensitivity Level Tags](https://knowledge.bigenius-x.com/apply-sensitivity-level-tags-project?hsLang=en) adds a tags notebook that the deployment does not run. ### Project properties The configuration offers these project properties, with their defaults: - [Hash Algorithm](https://knowledge.bigenius-x.com/hash-algorithm-project?hsLang=en): `SHA-256`, the only value - [Hash Casing](https://knowledge.bigenius-x.com/hash-casing-project?hsLang=en): `None`; also `Upper` and `Lower` - [Hash Concatenation Separator](https://knowledge.bigenius-x.com/hash-concatenation-separator-project?hsLang=en): `_` - [Hash Null Placeholder](https://knowledge.bigenius-x.com/hash-null-placeholder-project?hsLang=en): `_NULL_` - [Hash Trim](https://knowledge.bigenius-x.com/hash-trim-project?hsLang=en): off - [Begin of Universe](https://knowledge.bigenius-x.com/begin-of-universe-project?hsLang=en): `1900-01-01` - [End of Universe](https://knowledge.bigenius-x.com/end-of-universe-project?hsLang=en): `9999-12-31` - [Persist Validity](https://knowledge.bigenius-x.com/persist-validity-project?hsLang=en): on - [Create Result Views](https://knowledge.bigenius-x.com/create-result-views-project?hsLang=en): on - [Target Object Comment Type](https://knowledge.bigenius-x.com/target-object-comment-type-project?hsLang=en): `None`; also `Tables` and `Tables and Views` - [Use Unity Catalog](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en): on - [Use Unique Constraints](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en): on - [Deployment Options](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en): `Generate Drop Table Statements` and `Generate Create Schema If Not Exists Statements`; also `Generate Alter View Body Statements` - [Notebook Deployment Objects Limit](https://knowledge.bigenius-x.com/notebook-deployment-objects-limit-project?hsLang=en): `100` - [Apply Sensitivity Level Tags](https://knowledge.bigenius-x.com/apply-sensitivity-level-tags-project?hsLang=en): off - [Sensitivity Level Tag Name](https://knowledge.bigenius-x.com/sensitivity-level-tag-name-project?hsLang=en): `sensitivity_level` - [Configure LoadControl](https://knowledge.bigenius-x.com/configure-loadcontrol-project?hsLang=en): on - [Deploy LoadControl](https://knowledge.bigenius-x.com/deploy-loadcontrol-project?hsLang=en): on - [LoadControl Options](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en): `Multi-Threading Notebook`; also `Airflow` - [Logging Method](https://knowledge.bigenius-x.com/logging-method-project?hsLang=en): `Delta Table`; also `Python Logger` ### Data types The Default Terms get these data types: - Timestamps, `Timestamp`: `BG_LoadTimestamp`, `BG_UpdateTimestamp`, `BG_ValidFromTimestamp`, `BG_ValidToTimestamp`, `BG_EffectiveTimestamp` - Row hashes, `Binary`: `BG_RowHash`, `BG_RowHashScd1`, `BG_RowHashScd2` - Text, `String`: `BG_SourceSystem`, `BG_DataflowName`, `BG_DataflowSetName`, `BG_ErrorDescription`, `BG_IncrementLoadStatus` - Numbers, `Long`: `BG_ID`, `BG_DataQuality_ID`; `Integer`: `BG_ErrorCount` - Flags, `Integer`: `BG_IsDeleted` ### Platform differences The table compares the configuration with [Microsoft SQL Server - Data Store](https://knowledge.bigenius-x.com/microsoft-sql-server-data-store?hsLang=en) of the same release: | Item | Databricks | Microsoft SQL Server | | --- | --- | --- | | Timestamp Default Terms | `Timestamp` | `datetimeoffset` | | Hash Default Terms | `Binary` | `binary(20)` | | Text Default Terms | `String` | `nvarchar` | | Hash Algorithm default | `SHA-256` | `SHA1` | | Persist Validity default | on | off | | Deployment Options default | drop tables, create schemas | drop tables | | Index management | not available | available | Both platforms offer the LoadControl Option `Airflow`. The native load control is `Multi-Threading Notebook` on Databricks. ### Example A Store Entity `bk_multipart_not_nullable` with the business key terms `bdc_shop_id` and `bdc_customer_id` has four SCD1 terms. It generates this table. The excerpt shows the example catalog `dwh` and schema `store` in place of the placeholders: ``` CREATE OR REPLACE TABLE `dwh`.`store`.`ds_se_bk_multipart_not_nullable` ( `bg_loadtimestamp` Timestamp NOT NULL ,`bg_updatetimestamp` Timestamp NOT NULL ,`bg_sourcesystem` String ,`bg_rowhashscd1` Binary NOT NULL ,`bdc_shop_id` String NOT NULL ,`bdc_customer_id` String NOT NULL ,`first_name` String ,`last_name` String ,`date_of_birth` Date ,`location` String ,CONSTRAINT `pk_ds_se_bk_multipart_not_nullable` PRIMARY KEY (`bdc_shop_id`, `bdc_customer_id`) ) USING delta ``` The hashing view builds the row hash of the SCD1 terms. It joins them with the separator `_` and replaces NULL with `_NULL_`: ``` UNHEX(SHA2(COALESCE(`first_name`, '_NULL_') || '_' || COALESCE(`last_name`, '_NULL_') || '_' || COALESCE(CAST(`date_of_birth` AS String), '_NULL_') || '_' || COALESCE(`location`, '_NULL_'), 256)) AS `bg_rowhashscd1` ``` The loader merges the rows of the delta view into the table with `MERGE` on the business key. ### Release differences Project Settings shows the Generator Version of your project. - Generator 2.0: Use Unity Catalog is on by default, off up to Generator 1.10. Target Object Comment Type replaced `Target Platform` with `Tables` and `Tables and Views`, default `None`. Deployment Options added `Generate Alter View Body Statements`. - Generator 2.2: added the project properties Persist Validity, Create Result Views, the four Hash properties and the two Sensitivity Level properties, the model object property Increment Management Type on Stage, and [Delete Detection Partial Terms](https://knowledge.bigenius-x.com/delete-detection-partial-terms?hsLang=en) and [Delete Detection Partial Operator](https://knowledge.bigenius-x.com/delete-detection-partial-operator?hsLang=en) on Store Entity. Deployment Options added `Generate Create Schema If Not Exists Statements`, selected by default. - Generator 2.4: added the project property [Use Unique Constraints](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en). - Generator 1.10: the project property [Delta Detection](https://knowledge.bigenius-x.com/delta-detection-model-object?hsLang=en). Since Generator 2.0, Delta Detection is a property of the Store Entity. ### Related articles - Previous: - Databricks: [Target environment with Unity Catalog](https://knowledge.bigenius-x.com/databricks-unity-target-environment?hsLang=en), [Target environment with the Hive metastore](https://knowledge.bigenius-x.com/databricks-target-environment?hsLang=en) - Next: [Generate artifacts](https://knowledge.bigenius-x.com/generate-artifacts?hsLang=en), [Replace the placeholders in the artifacts](https://knowledge.bigenius-x.com/replace-the-placeholders-in-the-artifacts?hsLang=en) - Databricks: [Artifacts](https://knowledge.bigenius-x.com/databricks-artifacts?hsLang=en), [Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en), [Load data - Native Load Control](https://knowledge.bigenius-x.com/databricks-load-data-native-load-control?hsLang=en), [Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/databricks-load-data-apache-airflow-load-control?hsLang=en) - Reference: [Delta Detection (Model Object)](https://knowledge.bigenius-x.com/delta-detection-model-object?hsLang=en), [Use Unity Catalog (Project)](https://knowledge.bigenius-x.com/unity-catalog-project?hsLang=en), [Use Unique Constraints (Project)](https://knowledge.bigenius-x.com/use-unique-constraints-project?hsLang=en), [Deployment Options (Project)](https://knowledge.bigenius-x.com/deployment-options-project?hsLang=en), [Notebook Deployment Objects Limit (Project)](https://knowledge.bigenius-x.com/notebook-deployment-objects-limit-project?hsLang=en), [LoadControl Options (Project)](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en), [Optimization Method (Model Object)](https://knowledge.bigenius-x.com/optimization-method-model-object?hsLang=en) - Databricks: [Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en) - See also: [Available Generator Configurations](https://knowledge.bigenius-x.com/available-generator-configurations?hsLang=en), [Understand the Generator Configuration](https://knowledge.bigenius-x.com/understand-the-generator-configuration?hsLang=en), [Source Systems per Generator Configurations](https://knowledge.bigenius-x.com/source-systems-per-generator-configurations?hsLang=en), [Microsoft SQL Server - Data Store](https://knowledge.bigenius-x.com/microsoft-sql-server-data-store?hsLang=en) - Databricks: [Stage](https://knowledge.bigenius-x.com/databricks-stage?hsLang=en), [Stage Files](https://knowledge.bigenius-x.com/databricks-stage-files?hsLang=en), [Stage JDBC](https://knowledge.bigenius-x.com/databricks-stage-jdbc?hsLang=en), [DataVault](https://knowledge.bigenius-x.com/databricks-datavault?hsLang=en), [Datamart](https://knowledge.bigenius-x.com/databricks-datamart?hsLang=en) - [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content) - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines) - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working) - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account) - [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en) - [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en) - [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content) - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration) - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions) - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features) - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects) - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches) - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections) - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview) - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps) - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms) - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping) - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships) - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality) - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes) - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling) - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts) - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings) - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace) - [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content) - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse) - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse) - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake) - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server) - [Oracle](https://knowledge.bigenius-x.com/generators?hsLang=en#oracle) - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts) - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders) - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment) - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment) - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline) - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment) - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment) - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control) - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type) - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties) - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms) - [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content) - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations) - [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content) - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches) - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases) - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules) - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules) - [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en) - [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content) - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application) - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application) - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators) - [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content) - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs) - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits) - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) - [Customer Portal](https://knowledge.bigenius-x.com/customer-portal) - [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login) [![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com) All rights reserved © 2025, biGENIUS AG