Skip to content
Contact us Go to Customer Portal
  • There are no suggestions because the search field is empty.

Databricks - DataVault

Applies to
Platform: Databricks
Architecture: Data Vault

Databricks - DataVault generates a Stage, Raw Vault and Business Vault for Databricks as notebooks with SQL statements on Delta tables. Select the Generator Configuration Databricks DataVault when you add a project whose Data Vault runs on Databricks.

For dbt, select Databricks Dbt DataVault instead, see Databricks dbt - DataVault.

Scope and output

  • Platform: Databricks. Each layer is a schema of a Unity Catalog catalog or, with Use Unity Catalog off, a Hive metastore database.
  • Layers: Source, Stage, Raw Vault and Business Vault. Dataflow Modeling shows Stage in the Bronze column and both vault layers in the Silver column. See Understand the Layering concept.
  • Output: notebooks in Jupyter and LoadControl, and deploy.ps1, described in Databricks - Artifacts.

Prerequisites

  • A Databricks workspace with a catalog for the project, or with the Hive metastore a storage location per layer. The deployment creates the schemas or databases.
  • An all-purpose cluster and a user who can create tables and run jobs on it, and read the source tables.
  • PowerShell 7 and the Databricks CLI for deploy.ps1.

The setup is described in Databricks - Target environment with Unity Catalog and Databricks - Target environment with the Hive metastore, the placeholders in Databricks - Placeholder values.

Sources and linked projects

Stage source views read each source table as <catalog>.<schema>.<table> with placeholders of the source system. Discover a Databricks Unity Catalog creates the source model objects; Source Systems per Generator Configurations lists the discovery types.

You can import a linked project, for example a Databricks - Stage Files or Databricks - Stage JDBC project, and export this project as a linked project.

Supported model objects

Source and Stage

Raw Vault

Business Vault

Capabilities and limitations

  • Loaders: one notebook per loader in the folder Jupyter, for example RDV_HUB_common_customer_Loader, with the load timestamp as the parameter bg_loadtimestamp.
  • Tables: Delta tables. File Format offers only delta, and Optimization Method offers Partitioned By and Cluster By.
  • Keys, with Unity Catalog only: Hubs and Links have their hash key as primary key and, with Use Unique Constraints, a UNIQUE constraint on the business key terms of Hubs and Links. Databricks does not enforce them, see Constraints on Databricks. Databricks Runtime 17.3 LTS rejects UNIQUE constraints, see Databricks - Deployment.
  • Foreign keys: Foreign Key Mode offers None and Create Disabled (default), not Create Enforced. Create Disabled adds the constraint as NOT ENFORCED.
  • Unknown member: Hubs, Links, Satellites and Link Satellites get a row with the hash key X'00', and the result views filter it out. Custom Singleton Values sets its values.
  • Hashing: hash keys and row hashes are UNHEX(SHA2(..., 256)) in Binary columns, see How biGENIUS-X builds hash values.
  • Validity: with Persist Validity on (default), historized tables store bg_validtotimestamp; off, the result view computes it.
  • Change detection: changes to Satellite terms selected in Excluded from Delta Detection alone create no new version.
  • Loading order: Use Embryo Loader on a relationship inserts missing Hub or Link keys before the Satellite loader runs.
  • Deployment: with the default Deployment Options, each deployment replaces the tables and the load control log tables with empty ones, see Redeploy and keep data in Databricks - Deployment.
  • Governance: Apply Sensitivity Level Tags adds a tags notebook that the deployment does not run.

Project properties

The configuration offers these project properties, with their defaults:

Older projects can still show the legacy property Behaviour for Nulled Business Keys in Links; the data quality rules replace it, see Understand Data Quality rules overview.

Data types

The Default Terms get these data types:

  • Timestamps, Timestamp: BG_LoadTimestamp, BG_UpdateTimestamp, BG_ValidFromTimestamp, BG_ValidToTimestamp, BG_EffectiveTimestamp, DVPIT_ForeignKey_LoadDate, BG_Effectivity_Begin, BG_Effectivity_End
  • Hash keys and row hashes, Binary: Hub_HK, Link_HK, DrivingKeyLink_HK, PIT_HK, DVLINK_ForeignKey_Identity, DVPIT_ForeignKey_Identity, BG_RowHash, BG_RowHashScd1, BG_RowHashScd2
  • Text, String: BG_SourceSystem, BG_DataflowName, BG_DataflowSetName, BG_ErrorDescription, BG_IncrementLoadStatus
  • Numbers, Long: BG_ID, BG_DataQuality_ID; Integer: BG_ErrorCount, BG_Sequence
  • Flags, Integer: BG_IsDeleted, BG_MAS_HubDifferenceFound; Boolean: BG_Appearance

Platform differences

The table compares the configuration with Microsoft SQL Server - DataVault of the same release:

ItemDatabricksMicrosoft SQL Server
Timestamp Default TermsTimestampdatetimeoffset
Effectivity Default TermsTimestampdatetime2
Hash Default TermsBinarybinary(20)
Text Default TermsStringnvarchar
Hash Algorithm defaultSHA-256SHA1
Persist Validity defaultonoff
Deployment Options defaultdrop tables, create schemasdrop tables
Enforced foreign keysnot availableavailable
Index managementnot availableavailable

Both platforms offer the LoadControl Option Airflow. The native load control is Multi-Threading Notebook on Databricks.

Example

A Hub common_customer with the business key term customer_id generates this table. The excerpt shows the example catalog dwh and schema rawvault in place of the placeholders:

CREATE OR REPLACE TABLE `dwh`.`rawvault`.`rdv_hub_common_customer` (
     `bg_loadtimestamp` Timestamp NOT NULL
    ,`hub_hk` Binary NOT NULL
    ,`bg_sourcesystem` String
    ,`customer_id` Decimal(38,0) NOT NULL
    ,CONSTRAINT `pk_rdv_hub_common_customer` PRIMARY KEY (`hub_hk`)
    ,CONSTRAINT `uc_rdv_hub_common_customer` UNIQUE (`customer_id`)
)
USING delta

The deployment then inserts the unknown-member row with hub_hk X'00', customer_id 0 and bg_sourcesystem Unknown. The hashing view rdv_hub_common_customer_hashing computes the hash key:

UNHEX(SHA2(COALESCE(CAST(`customer_id` AS String), '_NULL_'), 256))
    AS `hub_hk`

Release differences

Project Settings shows the Generator Version of your project.

  • Generator 2.0: Use Unity Catalog is on by default, off up to Generator 1.10. Target Object Comment Type replaced Target Platform with Tables and Tables and Views, default None. Deployment Options added Generate Alter View Body Statements. Added Business Driving Key Link.
  • Generator 2.2: added Vault Out, the project properties Persist Validity, Create Result Views, the four Hash properties and the two Sensitivity Level properties, the model object property Increment Management Type, and the relationship property Use Embryo Loader. Deployment Options added Generate Create Schema If Not Exists Statements, selected by default.
  • Generator 2.3: added the model object property Custom Singleton Values.
  • Generator 2.4: added the project property Use Unique Constraints; Non-Historized Satellite and Non-Historized Business Satellite; the model object property Excluded from Delta Detection.
  • Generator 1.10: the project property Delta Detection. Generator 2.0 no longer offers it.

Related articles