Databricks Data Vault and Mart

Applies to
Platform: Databricks
Architecture: Data Vault · Data Mart

Databricks Data Vault and Mart generates a Stage, Raw Vault, Business Vault and Mart for Databricks as notebooks with SQL statements on Delta tables. Select the Generator Configuration Databricks DataVault & Mart when you add a project whose Data Vault and data marts run on Databricks.

For dbt, select Databricks Dbt DataVault & Mart instead, see Databricks dbt Data Vault and Mart.

Scope and output

Platform: Databricks. Each layer is a schema of a Unity Catalog catalog or, with Use Unity Catalog off, a Hive metastore database.
Layers: Source, Stage, Raw Vault, Business Vault and Mart. Dataflow Modeling shows Stage in Bronze, both vault layers in Silver and the Mart model objects in the Datamart column. See Understand the Layering concept.
Output: notebooks in Jupyter and LoadControl, and deploy.ps1, described in Databricks - Artifacts.

Prerequisites

A Databricks workspace with a catalog for the project, or with the Hive metastore a storage location per layer. The deployment creates the schemas or databases.
An all-purpose cluster and a user who can create tables and run jobs on it, and read the source tables.
PowerShell 7 and the Databricks CLI for deploy.ps1.

The setup is described in Databricks - Target environment with Unity Catalog and Databricks - Target environment with the Hive metastore, the placeholders in Databricks - Placeholder values.

Sources and linked projects

Stage source views read each source table as <catalog>.<schema>.<table> with placeholders of the source system. Discover a Databricks Unity Catalog creates the source model objects; Source systems per generator configuration lists the discovery types.

You can import a linked project, for example a Databricks Stage Files or Databricks Stage JDBC project, and export this project as a linked project.

Supported model objects

Mart

Capabilities and limitations

Loaders: one notebook per loader in the folder Jupyter, for example DM_MD_bk_multipart_not_nullable_Loader, with the load timestamp as the parameter bg_loadtimestamp.
Tables: Delta tables. File Format offers only delta, and Optimization Method offers Partitioned By and Cluster By.
Keys, with Unity Catalog only: Hubs and Links have their hash key, Mart Dimensions their identity column <model object name>_sk, as primary key and, with Use Unique Constraints, a UNIQUE constraint on the business key terms of all three. Databricks does not enforce them, see Constraints on Databricks. Databricks Runtime 17.3 LTS rejects UNIQUE constraints, see Databricks - Deployment.
Foreign keys: Foreign Key Mode offers None and Create Disabled (default), not Create Enforced. Create Disabled adds the constraint as NOT ENFORCED.
Unknown member: Hubs, Links, Satellites and Link Satellites get a row with the hash key X'00', Mart Dimensions one with the surrogate key -1. Custom Singleton Values sets their values.
Hashing: hash keys and row hashes are UNHEX(SHA2(..., 256)) in Binary columns, see How biGENIUS-X builds hash values.
Validity: with Persist Validity on (default), historized tables store bg_validtotimestamp; off, the result view computes it.
Change detection: changes to Satellite terms selected in Excluded from Delta Detection alone create no new version.
Loading order: Use Embryo Loader on a relationship inserts missing Hub or Link keys before the Satellite loader runs.
Deployment: with the default Deployment Options, each deployment replaces the tables and the load control log tables with empty ones, see Redeploy and keep data in Databricks - Deployment.
Governance: Apply Sensitivity Level Tags adds a tags notebook that the deployment does not run.

Project properties

The configuration offers these project properties, with their defaults:

Hash Algorithm: SHA-256, the only value
Hash Casing: None; also Upper and Lower
Hash Concatenation Separator: _
Hash Null Placeholder: _NULL_
Hash Trim: off
Begin of Universe: 1900-01-01
End of Universe: 9999-12-31
Persist Validity: on
Create Result Views: on
Target Object Comment Type: None; also Tables and Tables and Views
Use Unity Catalog: on
Use Unique Constraints: on
Deployment Options: Generate Drop Table Statements and Generate Create Schema If Not Exists Statements; also Generate Alter View Body Statements
Notebook Deployment Objects Limit: 100
Apply Sensitivity Level Tags: off
Sensitivity Level Tag Name: sensitivity_level
Configure LoadControl: on
Deploy LoadControl: on
LoadControl Options: Multi-Threading Notebook; also Airflow
Logging Method: Delta Table; also Python Logger

Older projects can still show the legacy property Behaviour for Nulled Business Keys in Links; the data quality rules replace it, see Understand Data Quality rules overview.

Data types

The Default Terms get these data types:

Timestamps, Timestamp: BG_LoadTimestamp, BG_UpdateTimestamp, BG_ValidFromTimestamp, BG_ValidToTimestamp, BG_EffectiveTimestamp, DVPIT_ForeignKey_LoadDate, BG_Effectivity_Begin, BG_Effectivity_End
Hash keys and row hashes, Binary: Hub_HK, Link_HK, DrivingKeyLink_HK, PIT_HK, DVLINK_ForeignKey_Identity, DVPIT_ForeignKey_Identity, BG_RowHash, BG_RowHashScd1, BG_RowHashScd2
Text, String: BG_SourceSystem, BG_DataflowName, BG_DataflowSetName, BG_ErrorDescription, BG_IncrementLoadStatus
Numbers, Long: BG_ID, BG_DataQuality_ID, BG_MartDimension_Identity, BG_MartFact_ForeignKey_Identity; Integer: BG_ErrorCount, BG_Sequence
Flags, Integer: BG_IsDeleted, BG_MAS_HubDifferenceFound; Boolean: BG_Appearance

Platform differences

The table compares the configuration with Microsoft SQL Server Data Vault and Mart of the same release:

Item	Databricks	Microsoft SQL Server
Timestamp Default Terms	`Timestamp`	`datetimeoffset`
Effectivity Default Terms	`Timestamp`	`datetime2`
Hash Default Terms	`Binary`	`binary(20)`
Text Default Terms	`String`	`nvarchar`
Hash Algorithm default	`SHA-256`	`SHA1`
Persist Validity default	on	off
Deployment Options default	drop tables, create schemas	drop tables
Enforced foreign keys	not available	available
Index management	not available	available

Both platforms offer the LoadControl Option Airflow. The native load control is Multi-Threading Notebook on Databricks.

Example

A Mart Dimension bk_multipart_not_nullable reads the Business Current View bdv_current_bdc_customer of the same project. The excerpt shows the example catalog dwh with its layer schemas:

CREATE OR REPLACE VIEW
    `dwh`.`mart`.`dm_md_bk_multipart_not_nullable_source`
AS
SELECT
     CAST(NULL AS String) AS `bg_sourcesystem`
    ,`s1`.`bdc_shop_id` AS `bdc_shop_id`
    ,`s1`.`bdc_customer_id` AS `bdc_customer_id`
    ,`s1`.`first_name` AS `first_name`
    ,`s1`.`last_name` AS `last_name`
    ,`s1`.`date_of_birth` AS `date_of_birth`
    ,`s1`.`location` AS `location`
FROM `dwh`.`businessvault`.`bdv_current_bdc_customer` AS `s1`

Release differences

Project Settings shows the Generator Version of your project.

Generator 2.0: Use Unity Catalog is on by default, off up to Generator 1.10. Target Object Comment Type replaced Target Platform with Tables and Tables and Views, default None. Deployment Options added Generate Alter View Body Statements. Added Business Driving Key Link.
Generator 2.2: added Vault Out, Mart Out, the project properties Persist Validity, Create Result Views, the four Hash properties and the two Sensitivity Level properties, the model object property Increment Management Type, and the relationship property Use Embryo Loader. Deployment Options added Generate Create Schema If Not Exists Statements, selected by default.
Generator 2.3: added the model object property Custom Singleton Values.
Generator 2.4: added the project property Use Unique Constraints; Non-Historized Satellite and Non-Historized Business Satellite; the model object property Excluded from Delta Detection.
Generator 1.10: the project property Delta Detection. Since Generator 2.0, Delta Detection is a property of the Mart Dimension.