Skip to content
  • There are no suggestions because the search field is empty.

Add a Discovery

Add a Discovery

To add a Discovery for an existing Source System in a Project, open the Source System left menu option in the Data Connections overview page:

If your source is a database with multiple schemas, you should create a separate Source system for each schema.

Then, click on the Add Discovery button:

The Import a discovered file side panel window opens. It consists of 3 steps: Add Discovery, Compare, then Add Source:

Each Target Technology has a native Source System. This refers to a Source System that does not require any special configuration or method to access source data.

For example:

  • For Snowflake, the native Source System is Snowflake.

  • For Microsoft SQL Server, it is Microsoft SQL Server.

  • For Spark, it is Spark.

In the case of Spark, the native format is defined using Catalog objects. These are tables that are already populated by some process and registered in the Catalog (such as Hive or Unity).

These tables may be made available through:

  • Delta Sharing

  • External tables pointing to existing data files

  • Tables deployed and loaded by another process

When performing a discovery with a Source System of type Spark, you can use a Parquet/Delta discovery, since these formats are compatible with Spark.

When accessing source objects in this context, it is assumed that the objects are already registered in the catalog. They are queried directly (e.g., SELECT * FROM <source_table_name>) instead of being accessed via their file storage path (e.g., spark.read...).

Step one: Add Discovery

Fill in the following information:

  1. Discovery Filetype: Select one of the following types:
    1. DataHub: if you are using Discovery files from your usage of DataHub
    2. biGENIUS Discovery: if you are using YAML Discovery files from the biGENIUS-X Discovery Application
  2. Select File: select the Discovery File from your local system.

If you do not select a valid file type or a file that is not properly structured, you will receive the following error: This discovery file is not compatible.

Then, click NEXT.

Step two: Compare

The Compare step provides an overview of the differences from the previous Source Model Objects created with the prior Discovery by showing the deleted and modified Source Model Objects.
  • M: Modified Source Model Objects
  • D: Deleted Source Model Objects

Then, click NEXT.

Step three: Add Source

 You can now select the new available Source Model Objects to create.

The existing Source Model Objects for the Discovery have already been selected in the side panel window.

It is not possible to deselect them.

If you want to specifically delete a Source Model Object that is in the Discovery, see Delete a Model Object.

By clicking the UPDATE button:

    • The M - Modified Source Model Objects will be updated
    • The D - Deleted Source Model Objects will be deleted
    • The newly added Source Model Objects will be created.

The Discovery will appear in the Source System list:



A notification confirms the correct creation:

Demonstration video