Skip to content
Contact us Go to Customer Portal
  • There are no suggestions because the search field is empty.

dbt - Artifacts

Applies to
Platform: Snowflake dbt · Databricks dbt

The artifacts of a Snowflake dbt or Databricks dbt project are a complete dbt project: models, model YAML files and macros. This article explains its folders, the models of each model object and the file you edit. Use it to find the model of a model object and to see what dbt run builds.

The dbt artifacts contain no deployment scripts, no load control, no replace_placeholders.ps1 and no replacement_config.json. You set the values as described in dbt - Placeholder values, and dbt creates and loads the tables as described in dbt - Deployment.

Files and folders

  • dbt_project.yml: the project name, the schema of each layer, one pair of variables per source system and the name of the connection profile.
  • README.md: generator version, project, branch and the number of model objects and validation warnings.
  • ParseError.log: present when generation reported validation warnings, with the text of each warning.
  • models/<layer>/<model object type>/: the models of a layer, one folder per model object type, for example models/rawvault/hub. Stage models are in models/stage/<source system>/.
  • models/<layer>/models_<layer>.yml: the columns, data types and constraints of the models of the layer.
  • models/stage/sources_<source system>.yml: the dbt source of a source system with its tables and columns. Its catalog and schema come from the variables in dbt_project.yml.
  • macros/: the materializations, such as bg_append, and helper macros, such as bg_close_validity_merge.

The project contains no dbt tests, no packages.yml and no selectors.yml. You do not need dbt deps.

Models of a model object

Model names are in lower case and start with the prefix of the model object type, for example stg_, rdv_hub_, rdv_sat_ or bdv_hub_.

  • Source: <prefix><model object name>_source, the view that prepares the rows to load.
  • Storage: <prefix><model object name>, the table or view of the model object.
  • Result: <prefix><model object name>_result, the result view, only with Create Result Views on.
  • Dataflow source: <prefix><model object name>_<dataflow name>_source, the view of one dataflow.
  • Loader: <prefix><model object name>_<dataflow name>_loader, which inserts the rows of one dataflow.

Notes on these models:

  • With more than one dataflow, the storage model only creates the table. Each loader inserts into it after the loader of the previous dataflow.
  • The materialization follows the Implementation Type. Permanent uses bg_append: the first run creates the table, later runs insert new rows. Cached uses the dbt materialization table, which rebuilds the table on every run. Virtual gives a view.
  • Historized satellite types, such as Satellite, Link Satellite and Effectivity Satellite, and PITs carry the post-hook bg_close_validity_merge. It sets BG_ValidToTimestamp of the previous version of a key, see Persist Validity.

Constraints and contracts

dbt_project.yml sets +contract: enforced: true, so dbt checks the columns and data types of each model against the model YAML file. The YAML files declare not_null columns and the constraints primary_key, unique and foreign_key. Use Unique Constraints controls the unique constraints, and Foreign Key Mode the foreign keys. The bg_ materializations apply the constraints to the tables.

With Target Object Comment Type set to Tables and Views, dbt_project.yml also sets +persist_docs. With Apply Sensitivity Level Tags on, it adds the post-hook apply_column_tags.

Data Mart projects on Databricks dbt

  • Databricks Dbt DataMart: the models are in the layer folder mart, with one schema entry. The project reads the objects of its linked projects as dbt sources. Each linked source system gets a file models/stage/sources_<source system>.yml and the variables <source system>_catalog and <source system>_schema.
  • Databricks Dbt DataVault & Mart: one project with the layer folders stage, rawvault, businessvault and mart and four schema entries. Variables exist only for the source systems. The Mart models read the vault models with ref().

Example

Part of the artifacts of a Data Vault project with the source system bfmmssql. The Hub common_customer has one dataflow. The Satellite load_customer_sat has the dataflows Dataflow1 and Dataflow2:

dbt_project.yml
README.md
macros/bg_close_validity_merge.sql
macros/materializations/bg_append.sql
models/stage/sources_bfmmssql.yml
models/stage/bfmmssql/stg_st_customer_source.sql
models/stage/bfmmssql/stg_st_customer.sql
models/rawvault/models_rawvault.yml
models/rawvault/hub/rdv_hub_common_customer_source.sql
models/rawvault/hub/rdv_hub_common_customer.sql
models/rawvault/satellite/rdv_sat_load_customer_sat.sql
models/rawvault/satellite/rdv_sat_load_customer_sat_dataflow1_source.sql
models/rawvault/satellite/rdv_sat_load_customer_sat_dataflow1_loader.sql
models/rawvault/satellite/rdv_sat_load_customer_sat_dataflow2_source.sql
models/rawvault/satellite/rdv_sat_load_customer_sat_dataflow2_loader.sql

The Hub has a source view and a table model. The Satellite has an owner model rdv_sat_load_customer_sat and a source view and a loader for each dataflow.

Platform differences

  • Snowflake dbt: the source variables are <source system>_database and <source system>_schema. Each layer also has +snowflake_warehouse. The macros include bg_apply_tags and no process_config_changes. Snowflake dbt is available since Generator 2.4 and offers only the configuration Snowflake Dbt DataVault.
  • Databricks dbt: the macro process_config_changes applies changes of Optimization Method to existing tables, see dbt - Deployment.

Release differences

  • Generator 2.3: the catalog and schema of each source system are variables in dbt_project.yml. Up to Generator 2.2, sources_<source system>.yml contained catalog: <placeholder>.
  • Generator 2.4: the bg_ materializations apply the constraints. Generator 2.2 and 2.3 applied them with separate post-hooks.

Related articles