Databricks dbt - DataVault and Mart

Applies to
Platform: Databricks dbt
Architecture: Data Vault · Data Mart

Databricks dbt - DataVault and Mart generates a Stage, Raw Vault, Business Vault and Mart for Databricks as one dbt project. Select it in Project Settings, where Generator Configuration shows Databricks Dbt DataVault & Mart, when the Mart reads the Data Vault of the same project instead of a linked project. This article lists what the configuration supports and where it differs from Databricks - DataVault and Mart.

The Raw Vault and Business Vault objects have the properties, Implementation Types and dbt models described in Databricks dbt - DataVault, the Mart objects those described in Databricks dbt - Datamart. This article covers what changes when both are in one project. The configuration offers no load control, no deployment scripts and no data quality checks.

Scope and output

Platform: Databricks with Unity Catalog. The configuration is available since Generator 2.2; Generator 2.1 contained a preview.
Layers: Source, Stage, Raw Vault, Business Vault and Mart. See Understand the Layering concept.
Output: one dbt project with the model folders models/stage, models/rawvault, models/businessvault and models/mart, described in dbt - Artifacts. dbt creates and loads the tables, see dbt - Deployment.

Prerequisites

A Databricks workspace with Unity Catalog, dbt Core with the adapter dbt-databricks, and a connection profile, as described in Databricks dbt - Target environment. dbt 2 is not supported with Databricks dbt.
The values in dbt_project.yml, listed in dbt - Placeholder values.

How the Mart reads the vault

dbt writes each layer into the schema <profile schema>_<layer schema> in the catalog of the connection profile. <profile schema> is the schema of the profile. <layer schema> is the +schema value of the layer in dbt_project.yml.

The Mart models read the vault models with ref(), not as dbt sources. Therefore one dbt run builds Stage, Raw Vault, Business Vault and Mart in dependency order, and each Mart model runs after the vault models it reads. The project needs no linked project and no variables for the vault, unlike Databricks dbt - Datamart.

With Create Result Views off (default), the Mart models read the vault tables and views, for example rdv_lsat_<model object name>. With it on, they read the result view of a vault object that has one, for example rdv_lsat_<model object name>_result.

Sources of the Mart objects

Source systems have the type Spark, see Discover a Databricks Unity Catalog.
The Mart objects read the Raw Vault and Business Vault objects of the project, Vault Out and other Mart objects. They cannot read Source or Stage objects. The article of each Mart object lists the model object types it accepts.
Vault Out is a source of Mart Composite, Mart Dimension, Mart Fact and Mart Hierarchy Flatten since Generator 2.2.
The wizards Create Mart Composite, Create Mart Dimension and Create Mart Fact are available on Vault Out and on the vault objects. Create Mart Out is available on the Mart objects. Mart Hierarchy Flatten has no wizard.

Supported model objects

Source, Stage, Raw Vault and Business Vault: the model object types listed in Databricks dbt - DataVault, including Vault Out.

Mart:

Capabilities and limitations

Loading: dbt runs the models of all layers in the order of their references. Vault objects and Mart Facts with more than one dataflow get one loader model per dataflow.
Incremental reads: a vault model, or a Mart Fact with the Fact Load Strategy Insert that reads a vault object, reads only new rows when BG_IncrementalFilter is mapped to the source's BG_LoadTimestamp. See Configure an incremental load.
Historization: Mart Dimensions with SCD2 terms allow one dataflow.
Validity: historized satellites and Mart Dimensions with SCD2 terms need Persist Validity or Create Result Views. With both off, validation reports the error Model object uses historization, but result views are disabled and validity is not persisted. for each historized satellite, such as a Satellite or Link Satellite. Mart Dimensions with SCD2 terms get no validation error, but generation fails on them.
Keys: the model YAML files declare primary keys, unique constraints on business keys and, with Foreign Key Mode Create Disabled (default), foreign keys. In the Mart only Mart Facts get foreign keys, to the Mart Dimensions they reference. No foreign key points from a Mart object to a vault object. Foreign Key Mode offers None and Create Disabled.
Hashes: Hash Algorithm offers only SHA-256. Hash keys and row hashes have the data type Binary.

The properties that Databricks dbt - DataVault and Databricks dbt - Datamart list as not available are missing here too, among them the load control properties, Load Caching and Use Embryo Loader (see Embryo Loader).

Project properties

Project Settings shows the project properties of Databricks dbt - DataVault with the same defaults, among them Persist Validity on, Create Result Views off and Use Unique Constraints on.

Platform differences

Compared with Databricks - DataVault and Mart of the same release:

Output: a dbt project with SQL models instead of PySpark notebooks and deployment scripts. There is no load control and no Apache Airflow option.
Create Result Views: off by default; on by default in Databricks - DataVault and Mart.
Foreign Key Mode: also shown on Mart Dimension, Mart Composite, Mart Hierarchy Flatten and Mart Out, where it has no effect.
Target Object Comment Type: no value Tables.
Unity Catalog: always used; Databricks - DataVault and Mart has the property Use Unity Catalog.

Example

A Mart Dimension common_salestransaction reads the Link Satellite SALESTRANSACTION of the Raw Vault. Its model dm_md_common_salestransaction_source reads the vault model with ref(). Excerpt without the other columns:

    SELECT
         CAST(NULL AS String) AS `bg_sourcesystem`
        ,`s1`.`link_hk` AS `link_hk`
    FROM {{ ref('rdv_lsat_salestransaction') }} AS `s1`

List the models that a run of this dimension includes:

dbt ls --select +dm_md_common_salestransaction --output name

dbt lists 26 models and five source tables. The models are the Mart Dimension, the Link Satellite, its Link, five Hubs and five Stage objects, each with its _source model. dbt run with the same selection builds the Stage models first, then the Raw Vault models, then the Mart Dimension.

With Create Result Views on, the excerpt reads ref('rdv_lsat_salestransaction_result').

Release differences

Project Settings shows the Generator Version of your project.

Generator 2.2: first release after the preview in Generator 2.1. Compared with the preview: Vault Out, also as a source of the Mart objects; Create Result Views, default off; the Foreign Key Mode value Create Disabled with primary key and foreign key constraints; the incremental read filter on BG_LoadTimestamp; loaders for more than one dataflow; Reference Tables without an unknown-member row.
Generator 2.3: the catalog and schema of each source system are variables in dbt_project.yml; the model object property Custom Singleton Values; a fix of the Mart Out deduplication. See Generators Release - 2.3.0.
Generator 2.4: Non-Historized Satellite and Non-Historized Business Satellite; Excluded from Delta Detection; Use Unique Constraints; the materializations apply the constraints; table columns use the modeled data types. See Generators Release - 2.4.0.
Generator 2.4.1: Driving Key Links and Business Driving Key Links with more than one dataflow no longer report the validation warning ID-10065. The generated models stay the same. See Generators Release - 2.4.1.