Skip to content
Contact us Go to Customer Portal
  • There are no suggestions because the search field is empty.

Databricks dbt - DataVault

Applies to
Platform: Databricks dbt
Architecture: Data Vault

Databricks dbt - DataVault generates a Stage, Raw Vault and Business Vault for Databricks as a dbt project. Select it in Project Settings, where Generator Configuration shows Databricks Dbt DataVault, when you load your Data Vault with dbt on Databricks. This article lists what the configuration supports and where it differs from Databricks - DataVault.

The configuration offers no load control, no deployment scripts and no data quality checks. The full list of features that are not available is under Capabilities and limitations.

Scope and output

  • Platform: Databricks with Unity Catalog. The configuration is available since Generator 2.2; Generator 2.1 contained a preview.
  • Layers: Source, Stage, Raw Vault and Business Vault. See Understand the Layering concept.
  • Output: one dbt project with SQL models, model YAML files and macros, described in dbt - Artifacts. dbt creates and loads the tables, see dbt - Deployment.

Prerequisites

Sources and linked projects

Source systems have the type Spark, as with Databricks - DataVault. To create source model objects from Unity Catalog tables, see Discover a Databricks Unity Catalog.

You can export this project as a linked project, for example for a project with the configuration Databricks Dbt DataMart. With Create Result Views on, the export names each object by its result view, for example RDV_HUB_common_customer_Result. With it off, the export names the table, RDV_HUB_common_customer. The dbt models carry the same names in lower case.

Supported model objects

Source and Stage

Raw Vault

Business Vault

Capabilities and limitations

  • Loading: dbt runs the models in the order of their references. A model object with more than one dataflow gets one loader model per dataflow.
  • Incremental reads: a model reads only new source rows when BG_IncrementalFilter is mapped to the source's BG_LoadTimestamp, see Configure an incremental load.
  • Keys: the model YAML files declare primary keys, unique constraints on business keys and, with Foreign Key Mode Create Disabled (default), foreign keys. Foreign Key Mode offers None and Create Disabled.
  • Hashes: Hash Algorithm offers only SHA-256. Hash keys and row hashes have the data type Binary.
  • Data quality: data quality checks are not available. Manage Data Quality is greyed out in the menu of the model objects.
  • Reference Table: the table has no unknown-member row.

Implementation Type offers these values per model object type:

  • Hubs, Links, all satellite types and Reference Table: Permanent
  • PIT and Business PIT: Permanent or Cached
  • Stage: Cached (default) or Virtual
  • RawVault Composite, BusinessVault Composite and Vault Out: Cached or Virtual (default)
  • Hierarchy Flatten: Cached
  • Current View, History View and their Business counterparts: Virtual

These properties of Databricks - DataVault are not available:

Project properties

Project Settings shows these properties in the groups Governance, Hashing, Historization and Implementation, with their defaults:

Model object properties: Deduplication Method, Sort Order within Partition, Implementation Type, Version Valid From Term, Delete Detection Method, Foreign Key Mode, Custom Singleton Values, File Format (delta), Optimization Method, Optimization Method Columns, Driving Key, Calculate Effectivity End From Begin and Excluded from Delta Detection.

Platform differences

Compared with Databricks - DataVault of the same release:

  • Output: a dbt project with SQL models instead of PySpark notebooks and deployment scripts.
  • Loading: dbt runs the models; there is no load control and no Apache Airflow option.
  • Create Result Views: off by default; on by default in Databricks - DataVault.
  • Target Object Comment Type: no value Tables.
  • Unity Catalog: always used; Databricks - DataVault has the property Use Unity Catalog.

Example

A Hub common_customer with the business key term CUSTOMER_ID gives the models rdv_hub_common_customer_source and rdv_hub_common_customer. The excerpt of models/rawvault/models_rawvault.yml declares its keys:

  - name: rdv_hub_common_customer
    constraints:
      - type: primary_key
        columns: [hub_hk]
        name: pk_rdv_hub_common_customer
      - type: unique
        columns: [customer_id]
        name: uc_rdv_hub_common_customer
    columns:
      - name: bg_loadtimestamp
        data_type: Timestamp
      - name: bg_sourcesystem
        data_type: String
      - name: customer_id
        data_type: Decimal(38,0)
      - name: hub_hk
        data_type: Binary

The excerpt leaves out descriptions, the not_null constraints and warn_unsupported: false. With Use Unique Constraints off, the unique constraint is left out.

Release differences

Project Settings shows the Generator Version of your project.

  • Generator 2.2: first release after the preview in Generator 2.1. Compared with the preview: Vault Out; Create Result Views, default off; the incremental read filter on BG_LoadTimestamp; primary key and foreign key constraints; loaders for more than one dataflow on all model object types; Reference Tables without an unknown-member row.
  • Generator 2.3: the catalog and schema of each source system are variables in dbt_project.yml; the model object property Custom Singleton Values, see Generators Release - 2.3.0.
  • Generator 2.4: Non-Historized Satellite and Non-Historized Business Satellite; Excluded from Delta Detection; Use Unique Constraints; the materializations apply the constraints; table columns use the modeled data types. See Generators Release - 2.4.0.
  • Generator 2.4.1: Driving Key Links and Business Driving Key Links with more than one dataflow no longer report the validation warning ID-10065. The generated models stay the same. See Generators Release - 2.4.1.

Related articles