Skip to content
Contact us Go to Customer Portal
  • There are no suggestions because the search field is empty.

dbt incremental read filter not applied (Business Rule)

Applies to
Platform: Snowflake dbt · Databricks dbt
Architecture: Data Vault · Data Mart

These business rules warn when the dbt incremental read filter ignores the mapping of BG_IncrementalFilter or uses one watermark for several dataflows. A mapping to anything other than the source's BG_LoadTimestamp raises ID-34000, a read filter on a model object with several dataflows raises ID-34001, and a mapping on a Truncate Insert Mart Fact raises ID-34002. All three are warnings, and generation continues.

Messages

Each message names the model object and its type in place of <model object name> and <type>. The Generate Artifacts page lists the messages without the rule ID.

Incremental filter not mapped to the load timestamp

The client shows a warning with this text:

Model object '<model object name>' of type '<type>' has an
incremental filter that is not mapped to the source's
BG_LoadTimestamp. The dbt incremental read filter currently supports
only BG_LoadTimestamp; none applied. Map BG_IncrementalFilter to the
source's BG_LoadTimestamp to enable it.

On the model object card, point to the warning icon to see this message with the rule ID: Warning ID-34000.

Cause: the expression of BG_IncrementalFilter does not use the BG_LoadTimestamp of a source of its dataflow set. The rule checks the satellite types and Mart Facts with Fact Load Strategy Insert. The model then reads all source rows on every run.
Fix: map BG_IncrementalFilter to the source's BG_LoadTimestamp in Manage Term Mappings. When the model object should read all rows, clear the mapping instead.

Incremental filter on a model object with several dataflows

The client shows a warning with this text:

Model object '<model object name>' of type '<type>' uses an
incremental read filter with a single global watermark;
per-dataflow watermarking is not yet implemented and this may miss
data.

On the model object card, point to the warning icon to see this message with the rule ID: Warning ID-34001.

Cause: the model object has more than one dataflow, and BG_IncrementalFilter is mapped to the source's BG_LoadTimestamp. Each dataflow reads the source rows that are newer than its own last load. The rule checks the satellite types and Mart Facts with Fact Load Strategy Insert.
Fix: if validation shows ID-34001, clear the mapping of BG_IncrementalFilter to read all rows of every dataflow. Without the mapping, a satellite reads all source rows on every run, and a Mart Fact with Fact Load Strategy Insert inserts all source rows again on every run.

Incremental filter on a full-rebuild Mart Fact

The client shows a warning with this text:

Model object '<model object name>' of type '<type>' is a
Truncate-Insert Mart Fact with an incremental filter mapped; the dbt
incremental read filter does not apply to full-rebuild facts and is
not applied.

On the model object card, point to the warning icon to see this message with the rule ID: Warning ID-34002.

Cause: a Mart Fact with Fact Load Strategy Truncate Insert maps BG_IncrementalFilter. The Mart Fact is rebuilt in full on every run. Only Databricks dbt offers Data Mart projects.
Fix: clear the mapping of BG_IncrementalFilter.

Available since Generator 2.2.

Check the result

The warning icon is gone from the model object.

Example

In a Data Mart project on Databricks dbt, the Mart Fact deduplication_partition_mart_fact maps BG_IncrementalFilter to the term TRANSACTION_TIME of its source. The client shows:

Model object 'deduplication_partition_mart_fact' of type 'Mart Fact'
has an incremental filter that is not mapped to the source's
BG_LoadTimestamp. The dbt incremental read filter currently supports
only BG_LoadTimestamp; none applied. Map BG_IncrementalFilter to the
source's BG_LoadTimestamp to enable it.

Related articles