Applies to
Platform: Microsoft SQL Server · Snowflake · Databricks · Fabric Lakehouse · Fabric Warehouse · Oracle
Architecture: Stage
This business rule warns when a dataflow of a Stage has Deduplication Method Distinct or Partition. It raises a warning, and generation continues. The rule ID is ID-10047.
Message
The client shows a warning with this text on the model object. <model object name> and <type> stand for its name and type.
Model object '<model object name>' of type '<type>' has deduplication method 'Distinct' or 'Partition' configured. This is usually not considered as best practice. Consider using a Composite Model Object for deduplication.
On the model object card, point to the warning icon to see this message with the rule ID: Warning ID-10047. The Generate Artifacts page lists the message without the ID.
Cause
A dataflow of the Stage has Deduplication Method Distinct or Partition. The rule checks Stage only.
Fix
Do one of the following:
- Open the dataflow of the Stage and set Deduplication Method to
None, see Edit a Dataflow. The warning disappears. - Deduplicate in a composite model object that reads from the Stage instead, for example a Store Composite, Core Composite or Mart Composite.
If you keep the deduplication on the Stage, the warning stays, and generation continues.
Check the result
The warning icon is gone from the Stage.
Example
In a Stage project on Microsoft SQL Server, the Stage deduplication_distinct has Deduplication Method Distinct. The client shows:
Model object 'deduplication_distinct' of type 'Stage' has deduplication method 'Distinct' or 'Partition' configured. This is usually not considered as best practice. Consider using a Composite Model Object for deduplication.