--- title: Deduplication Method (Dataflow) description: Deduplication Method (DeduplicationMethod) removes duplicate source rows of a dataflow before the loader applies them. --- [Skip to content](https://knowledge.bigenius-x.com/deduplication-method-model-object#main-content) [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en) [![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/) Open main navigation Close main navigation - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new) - [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal) How can we help you? - There are no suggestions because the search field is empty. 1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en) 2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en) 3. [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties) # Deduplication Method (Dataflow) **Applies to** Platform: Microsoft SQL Server · Snowflake · Snowflake dbt · Databricks · Databricks dbt · Fabric Lakehouse · Fabric Warehouse · Oracle Deduplication Method (`DeduplicationMethod`) removes duplicate source rows of a dataflow before the loader applies them. Use it when a source can deliver the same key more than once in one load. ### Where to set it Set the property when you [edit a dataflow](https://knowledge.bigenius-x.com/edit-a-data-flow?hsLang=en). Each dataflow of a model object has its own value. It is offered on Stage, Store Entity, Store Composite, Entity, Fact, Core Composite, Mart Fact, Mart Composite, Mart Out and all Raw Vault and Business Vault types except PIT, Business PIT, Current View, History View, Business Current View, Business History View and Hierarchy Flatten. Mart Dimension and Mart Hierarchy Flatten do not offer it. On a Stage, validation shows a warning for `Distinct` and `Partition`, see [No deduplication for Stage objects (Business Rule)](https://knowledge.bigenius-x.com/no-deduplication-for-stage-objects-business-rule?hsLang=en). ### Values and default - `None` (default): no deduplication. - `Distinct`: the deduplication view selects `DISTINCT` rows over all mapped terms. - `Partition`: the deduplication view numbers the rows of each partition with `ROW_NUMBER()` and keeps the first row. The Microsoft SQL Server - Stage OpenQuery generator offers `None` and `Distinct` only. ### How Partition builds the partition - Objects with business keys partition by the business keys. - Satellites and Link Satellites partition by their foreign keys, which carry the business keys of the Hub or Link. - Objects without business keys partition by all non-default terms. Validation shows a warning for them, see [Partition deduplication without business keys (Business Rule)](https://knowledge.bigenius-x.com/model-object-has-deduplication-method-partitionorder-but-no-business-keys-defined-business-rule?hsLang=en). The rows of a partition are ordered by [Sort Order within Partition](https://knowledge.bigenius-x.com/deduplication-partition-model-object?hsLang=en). Without it, they are ordered by the partition terms, so any of the duplicates can be kept. ### Example A Store Entity with the business key `ARTICLE_NUMBER`, Partition and Sort Order within Partition `OFFER_DATE` ascending. The deduplication view on Microsoft SQL Server, shortened: ``` SELECT [BG_Partition].[ARTICLE_NUMBER], ... FROM ( SELECT [BG_Source].[ARTICLE_NUMBER], ... ,ROW_NUMBER() OVER (PARTITION BY [BG_Source].[ARTICLE_NUMBER] ORDER BY [BG_Source].[OFFER_DATE] ASC) AS [BG_DeduplicationPartitionOrder] FROM [DS_SE_deduplication_partition_se_Source] AS [BG_Source] ) AS [BG_Partition] WHERE [BG_Partition].[BG_DeduplicationPartitionOrder] = 1 ``` For each article, the row with the earliest offer date is loaded. ### Related articles - Reference: [Sort Order within Partition (Dataflow)](https://knowledge.bigenius-x.com/deduplication-partition-model-object?hsLang=en) - See also: [Edit a Dataflow](https://knowledge.bigenius-x.com/edit-a-data-flow?hsLang=en), [Supported Deduplication Methods (Business Rule)](https://knowledge.bigenius-x.com/supported-deduplication-methods-business-rule?hsLang=en), [No deduplication for Stage objects (Business Rule)](https://knowledge.bigenius-x.com/no-deduplication-for-stage-objects-business-rule?hsLang=en), [Partition deduplication without business keys (Business Rule)](https://knowledge.bigenius-x.com/model-object-has-deduplication-method-partitionorder-but-no-business-keys-defined-business-rule?hsLang=en) - [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content) - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines) - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working) - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account) - [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en) - [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en) - [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content) - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration) - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions) - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features) - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects) - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches) - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections) - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview) - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps) - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms) - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping) - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships) - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality) - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes) - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling) - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts) - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings) - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace) - [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content) - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse) - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse) - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake) - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server) - [Oracle](https://knowledge.bigenius-x.com/generators?hsLang=en#oracle) - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts) - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders) - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment) - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment) - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline) - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment) - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment) - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control) - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type) - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties) - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms) - [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content) - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations) - [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content) - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches) - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases) - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules) - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules) - [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en) - [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content) - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application) - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application) - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators) - [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content) - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs) - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits) - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) - [Customer Portal](https://knowledge.bigenius-x.com/customer-portal) - [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login) [![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com) All rights reserved © 2025, biGENIUS AG