Optimization Method without columns or with parallel dataflows (Business Rule)

Applies to
Platform: Databricks · Databricks dbt · Fabric Lakehouse

These business rules check the Optimization Method of a model object on Databricks and Fabric Lakehouse. An Optimization Method without Optimization Method Columns raises error ID-31002, which blocks generation in the client. Dataflows that load in parallel without Optimization Method Partitioned By raise warning ID-31001.

Messages

Each message names the model object and its type in place of <model object name> and <type>. The Generate Artifacts page lists the messages without the rule ID.

Optimization Method without columns

The client shows an error with this text:

Model object '<model object name>' of type '<type>' has an
optimization method, but no optimization method columns defined.

On the model object card, point to the error icon to see this message with the rule ID: Error ID-31002.

Cause: Optimization Method is Partitioned By or Cluster By, and Optimization Method Columns is empty. The rule also runs on Databricks dbt.
Fix: select the columns in Optimization Method Columns, or clear Optimization Method.

Parallel dataflows without partitioning

The client shows a warning with this text:

Model object '<model object name>' of type '<type>' has multiple
dataflows, uses parallel dataflow execution but has no optimization
method defined. This could cause the Spark runtime error
io.delta.exceptions.ConcurrentAppendException.

On the model object card, point to the warning icon to see this message with the rule ID: Warning ID-31001.

Cause: the model object has more than one dataflow, Use Sequential Dataflow Execution is false, and Optimization Method is not Partitioned By. Cluster By also raises the warning. The dbt generators do not offer Use Sequential Dataflow Execution.
Fix: set Use Sequential Dataflow Execution to true, or set Optimization Method to Partitioned By with columns that give each dataflow its own partition.

Available since Generator 2.0.

Check the result

The error or warning icon is gone from the model object.

Example

In a Data Vault project on Databricks, the Hub common_item has Optimization Method Cluster By and no Optimization Method Columns. The client shows:

Model object 'common_item' of type 'Hub' has an optimization method,
but no optimization method columns defined.