Skip to content
Contact us Go to Customer Portal
  • There are no suggestions because the search field is empty.

Use Stream (Model Object)

Applies to
Platform: Snowflake

Use Stream (UseStream) makes a dataflow of a model object read its source through a Snowflake stream instead of the source view. The stream returns the changes of the source since the previous load that consumed it.

Use Stream is a preview feature of biGENIUS-X. It covers the common load patterns; support for further scenarios and edge cases is still being extended.

Where to set it

Only the native Snowflake generators offer the property; Snowflake dbt does not. Use Stream is set per dataflow. Set it when you edit a dataflow. Each dataflow of a model object has its own value. It is offered on Stage, Store Entity, Store Composite, Entity, Fact, Core Composite, Mart Dimension, Mart Fact, Mart Composite, Mart Out, Vault Out, Reference Table, RawVault Composite, BusinessVault Composite and all Hub, Link and Satellite types, including the Business variants.

Values and default

  • false (default): the loader reads the source view.
  • true: the loader reads a stream on the source view.

Effect on generated output

With true, the generator renames the source view to <source view>_STREAMSOURCE and creates a stream with the original name of the source view on it. The stream is created with SHOW_INITIAL_ROWS = TRUE, so the first load reads all existing rows. For a Reference Table the stream is a standard stream. For all other types it has APPEND_ONLY = TRUE. The Snowflake documentation of CREATE STREAM describes the stream types. The generator creates no dynamic tables.

Every deployment runs CREATE OR REPLACE STREAM ... SHOW_INITIAL_ROWS = TRUE and so recreates the stream. The first load after a deployment therefore reads all rows of the source view again, not only the changes (CREATE STREAM).

Restrictions

Validation rejects these combinations with an error:

  • A model object with Implementation Type Virtual: has implementation type 'Virtual' and usage of stream defined. This is only supported with implementation type 'Permanent', 'FactLoadStrategy' or 'Cached'. (ID-30000)
  • A model object with an incremental load: has both incremental load and usage of streams configured. Use only one of each. (ID-30004)

Example

A Satellite with Use Stream true gets, shortened:

CREATE OR REPLACE VIEW
    <database>.<schema>.RDV_SAT_<model object name>_SOURCE_STREAMSOURCE
    COPY GRANTS
...
CREATE OR REPLACE STREAM
    <database>.<schema>.RDV_SAT_<model object name>_SOURCE
ON VIEW
    <database>.<schema>.RDV_SAT_<model object name>_SOURCE_STREAMSOURCE
APPEND_ONLY = TRUE
SHOW_INITIAL_ROWS = TRUE;

<model object name> stands for the name of the Satellite in capitals. <database> and <schema> stand for the replaced placeholder values of the Raw Vault.

Related articles