Skip to content
Contact us Go to Customer Portal
  • There are no suggestions because the search field is empty.

Partition Filter (Target Term)

Applies to
Platform: Databricks · Fabric Lakehouse
Architecture: Stage

Partition Filter (PartitionFilter) marks the term of a file Stage that holds the partition of the source files, for example a date folder. The loader then loads only the rows of the last loaded partition and of later ones.

The property exists only in the Databricks and Fabric Lakehouse Stage Files generators.

Where to set it

Set the property when you edit a term of a Stage. Set it on the term mapped to the partition column of the source files.

Values and default

  • false (default): the loader loads all rows of the source path.
  • true: the loader filters the rows by this term.

Effect on generated output

The Stage gets an increment table with the column bg_partition_filter_<term name>. Each load works in three steps:

  1. The loader reads the value stored by the previous load. On the first load there is none, and the filter is 1 = 1.
  2. It reads the source rows whose partition value is greater than or equal to the stored value and reloads the Stage table with them.
  3. It stores the highest partition value of the Stage table, when it differs from the stored one.

The last loaded partition is therefore read again on the next load.

Example

The Stage BRANCH on Databricks has Partition Filter true on the term dt. Its loader builds the filter as follows:

partition_filter_clause = '1 = 1'
...
partition_filter_clause = f"dt >= '{bg_partition_filter_dt_value}'"

After a load whose highest dt is 2026-10-27, the next load loads the rows with dt from 2026-10-27 on.

Related articles