---
title: Configure an incremental load
description: "If you run a full load at each load, the time required to process is monotonically increasing because today’s load will always have more data than yesterday’s.  A best practice is to use incremental loads, which only load the delta.  The benefits are:  Run faster since they touch less data Usually steady over time Can preserve historical data. Many source systems purge old data periodically"
---

[Skip to content](https://knowledge.bigenius-x.com/configure-an-incremental-load#main-content)

[Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en)

[![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/)

Open main navigation

Close main navigation

- [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new)
- [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal)

 How can we help you?

- There are no suggestions because the search field is empty.

1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en)
2. [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en)
3. [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases)

# Configure an incremental load

If you run a full load every time, the processing time will **increase continuously**, as each day’s load will include more data than the previous one.

A best practice is to use incremental loads, which only process the **delta**—the changes since the last load.

The benefits of incremental loads include:

- **Faster performance**, as they process less data.
- **More consistent execution time** over the long term.
- **Preservation of historical data**, since many source systems periodically purge old records.

 

We can configure the Incremental Load behavior for the following Model Objects:

- Stage
- PIT
- Satellite
- Link Satellite
- Multi-Active Satellite
- Effectivity Satellite
- Record Tracking Satellite
- Effectivity Link Satellite
- Record Tracking Link Satellite
- Non-Historized Link Satellite
- Business PIT
- Business Satellite
- Business Link Satellite
- Effectivity Business Satellite
- Record Tracking Business Link Satellite
- Effectivity Business Link Satellite
- Non-Historized Business Link Satellite
- Mart Fact
- Fact

 

To configure incremental loading, you must map the [***BG\_IncrementalFilter***](https://knowledge.bigenius-x.com/bg_incrementalfilter?hsLang=en) **Default Term** to:

- **a Source Term** that contains the value used for the increment, for example:  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Oct-04-2023-01-33-26-6949-PM.png?width=670&height=231&name=image-png-Oct-04-2023-01-33-26-6949-PM.png)
- **a function composed of several Source Terms,** for example:  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Oct-04-2023-01-34-02-5380-PM.png?width=670&height=225&name=image-png-Oct-04-2023-01-34-02-5380-PM.png)

The relevant Model Object should **not** have the **Implementation Type property set to Virtual.**

See [Implementation Type (Model Object)](https://knowledge.bigenius-x.com/implementation-type-model-object?hsLang=en) for more details.

 

The ***BG\_IncrementalFilter*** is created by default with an ***unsupported*** Data Type.

To **configure incremental load behavior**, please **change it to the correct Data Type** according to the mapped Source(s) Term(s).

All Source terms can be used with data types such as datetime, int...

Example:  
![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Oct-04-2023-01-35-50-4165-PM.png?width=670&height=231&name=image-png-Oct-04-2023-01-35-50-4165-PM.png)

Depending on the Model Object Type, the incremental load implementation is different:

- Stage: As a Stage loads data from the Source system, we prioritize to access this data only once to avoid overloading the Source System
- Other Model Object Types: For example, a Mart Fact can contain a large volume of data, so we prioritize minimizing the number of columns.

### Stage Model Object Type

If an incremental load behavior is set for a Stage, the ***BG\_IncrementalFilter*** Default Term contains the **increment value used to select rows for the incremental load**. 

So the **max value of the Default Term**.

Example of values:

![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Oct-04-2023-01-38-24-7950-PM.png?width=670&height=78&name=image-png-Oct-04-2023-01-38-24-7950-PM.png)

This increment is stored after each load in a **new table,** ***XXX\_INC.*** For example, the table below shows where two loads occurred:

![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jan-11-2024-10-29-39-6233-AM.png?width=611&height=174&name=image-png-Jan-11-2024-10-29-39-6233-AM.png)

During deployment, this table is empty. It is populated during the load process.

**Reset or Override of the Increment**

Once the incremental load is configured, all increments are derived from the logs of previous loads.

However, there are specific scenarios where you want to override this behavior manually:

- to re-populate an incrementally loaded table (e.g., after a truncate or a re-deployment)
- to start the incremental load at a specific point (e.g., when enabling incremental load on an already populated table)
- to ignore all data before a particular value during the initial load (e.g., to apply a specific value of cut-off)
- to correct issues with a previous load where data was manually modified or removed  in an incrementally loaded table

To configure a manual override for a specific loader, update the column ***BG\_IncrementalFilter*** in the table ***XXX\_Increment***.

You have two options, depending on your objective:

- **To re-populate** an incrementally loaded table, **truncate the *XXX\_Increment t*able**.
- **To start** from a specific increment, **insert a new row** with the desired increment value.

### Other Model Object Types

If an incremental load behavior was set for a PIT, a Business PIT, or a Mart Fact, the ***BG\_IncrementalFilter*** Default Term does **not** **contain the increment value used to select rows for the incremental load**.

This increment is stored after each load in a **new table,** ***XXX\_INC.*** For example, the table below shows where two loads occurred:

![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jan-11-2024-10-29-39-6233-AM.png?width=611&height=174&name=image-png-Jan-11-2024-10-29-39-6233-AM.png)

During the deployment, this table already contains **one row per Dataflow and Dataflow Set for the Model Object:**

![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Apr-03-2025-08-28-28-5846-AM.png?width=620&height=41&name=image-png-Apr-03-2025-08-28-28-5846-AM.png)

It is populated during the load.

The view ***XXX\_IncrementSource*** is used to determine which rows should be loaded based on the increment.

**Reset or Override of the Increment**

Once the incremental load is configured, all increments are derived from the logs of previous loads.

However, there are specific scenarios where you want to override this behavior manually:

- to re-populate an incrementally loaded table (e.g., after a truncate or a re-deployment)
- to start the incremental load at a specific point (e.g., when enabling incremental load on an already populated table)
- to ignore all data before a particular value during the initial load (e.g., to apply a specific value of cut-off)
- to correct issues with a previous load where data was manually modified or removed  in an incrementally loaded table

To configure a manual override for a specific loader, update the column ***BG\_IncrementalFilter*** in the table ***XXX\_Increment***.

You have two options, depending on your objective:

- **To re-populate** an incrementally loaded table, **delete the affected rows.**
- To **start the incremental load at a specific increment**, **insert a new row** with the desired increment value.

**Do not delete** the rows created during deployment **that contain a NULL value** for the increment. These rows are required by the generated code to manage the initial load.

### Increment Management Types

By default, the incremental load is **automatically managed by the loaders**. They will query the data sources to find and load new data and then update the increment tables to reflect up to which point data was loaded. However, since all fresh data is always loaded in one transaction, this **could overwhelm the database system** in an environment **with very large datasets**.

User managed incremental load is available in generators with version 2.2.0 and above.

Therefore, you can choose to control the incremental load yourself, by configuring user managed increment management type. In that case the user is responsible to identify and specify the next increment that should be loaded.

To **configure user managed incremental load**, select the option ***User*** for the property ***Increment Management Type*** on the model object:

![](https://knowledge.bigenius-x.com/hs-fs/hubfs/undefined-Aug-14-2026-09-30-13-2957-AM.png?width=670&height=524&name=undefined-Aug-14-2026-09-30-13-2957-AM.png)

After deployment, and **before each load** you have to prepare the increments in the corresponding increment table ***XXX\_Increment*** for this model object.

![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-14-2026-09-37-48-6696-AM.png?width=670&height=40&name=image-png-Aug-14-2026-09-37-48-6696-AM.png)

For each data flow and data flow set create a new row and **specify the lower and upper boundary** by which the ***BG\_IncrementalFilter*** column should be filtered using the columns ***BG\_LowerThreshold*** and ***BG\_UpperThreshold***.

To ensure consistency during loads, the specified value for the **lower boundary will be excluded**, the value for the **upper boundary will be included**. To avoid gaps or duplicated data in concurrent runs, the lower boundary therefore should match the upper boundary of the previous run. For both boundaries you can use ***NULL*** to not filter for a specific value and rather load all available rows.

The column ***BG\_IncrementLoadStatus*** will be used to indicate the load status: ***NULL*** means the load is configured but didn't happen yet; after the load the loader will update this field to ***done*** to indicate a successful load.

![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-14-2026-09-56-45-5335-AM.png?width=670&height=45&name=image-png-Aug-14-2026-09-56-45-5335-AM.png)

You can **prepare multiple rows** (e.g. for setting up an initial load); they will be **processed one row for each load, data flow and data flow set** ascending by ***BG\_ID***.

We recommend that you automate this using your orchestration framework.

 

- [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content)

    - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines)
    - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working)
    - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account)
- [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en)
- [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en)
- [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content)

    - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration)
    - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions)
    - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features)
    - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects)
    - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches)
    - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections)
    - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview)
    - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps)
    - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms)
    - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping)
    - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships)
    - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality)
    - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes)
    - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling)
    - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts)
    - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings)
    - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace)
- [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content)

    - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse)
    - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse)
    - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks)
    - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake)
    - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server)
    - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts)
    - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders)
    - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment)
    - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment)
    - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline)
    - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment)
    - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment)
    - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control)
    - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow)
    - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type)
    - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties)
    - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms)
- [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content)

    - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations)
- [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content)

    - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches)
    - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases)
    - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules)
    - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules)
- [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en)
- [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content)

    - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application)
    - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application)
    - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators)
- [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content)

    - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs)
    - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits)

- [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en)
- [Customer Portal](https://knowledge.bigenius-x.com/customer-portal)
- [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login)

[![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com)

<https://www.linkedin.com/company/bigenius> <https://www.youtube.com/@biGENIUS_DWA>

All rights reserved © 2025, biGENIUS AG