Skip to content
  • There are no suggestions because the search field is empty.

Add a Discovery

Add a Discovery

To create a Discovery for a Project, open the Source System left menu option in the Data Connections overview page:

If your source is a database with multiple schemas, you should create a separate Source system for each schema.

Then, click on the Add Discovery button:

The Import a discovered file side panel window opens. It consists of 3 steps: Add Discovery, Compare, then Create Source:

or

or

Each Target Technology has a native Source System. This refers to a Source System that does not require any special configuration or method to access source data.

For example:

  • For Snowflake, the native Source System is Snowflake.

  • For Microsoft SQL Server, it is Microsoft SQL Server.

  • For Spark, it is Spark.

In the case of Spark, the native format is defined using Catalog objects. These are tables that are already populated by some process and registered in the Catalog (such as Hive or Unity).

These tables may be made available through:

  • Delta Sharing

  • External tables pointing to existing data files

  • Tables deployed and loaded by another process

When performing a discovery with a Source System of type Spark, you can use a Parquet/Delta discovery, since these formats are compatible with Spark.

When accessing source objects in this context, it is assumed that the objects are already registered in the catalog. They are queried directly (e.g., SELECT * FROM <source_table_name>) instead of being accessed via their file storage path (e.g., spark.read...).

Fill in the following information:

  1. Add to an existing Source System: There are two options here: add a new Source System or add the Discovery to an existing Source System.

    1. Add a Discovery in a new Source System:
      Select the "- Add new Source System -" option in the Source System list:

    2. Add a Discovery for an existing Source System:
      Select the Source System in the Source System list:

  2. Source System Type: several Source System Types can be available depending on the Generator Configuration.
    1. Example:
  3. Source: 3 possibilities:

    1. File Upload: select this option to upload a discovery file 
      Then fill:

      1. Source System Name: fill in a name for your Source System
      2. Discovery Filetype: Select one of the following types:

        1. DataHub: if you are using Discovery files from your usage of DataHub
        2. biGENIUS Discovery: if you are using YAML Discovery files from the biGENIUS-X Discovery Application
      3. Select File: select the Discovery File from your local system.
    2. Data Marketplace: select this option to choose a Data Product Contract from the Data Marketplace
      Some info here: Understand the Data Marketplace overview
      Then fill:
      1. Source System Name: choose the Source System available in the Data Marketplace:
      2. Data Contract: choose a Data Contract version of the Source System
    3. Manual: select this option to start modeling without any Discovery.
      Then fill:
      1. Source System Name: fill in a name for your Source System
  4. Description (optional): fill in a description for your Source System

If you do not select a valid file type or a file that is not properly structured, you will receive the following error: This discovery file is not compatible.

Then, click on the NEXT button:

  • If you created a new Source System, the Add Source step is displayed in the side panel window. See Create Source Objects from a Discovery.
  • If you added a Discovery in an existing Source System, the Compare step will be displayed: it contains an overview of the differences with the previous Source Model Objects created with the previous Discovery:
    • A: Added Source Model Objects
    • M: Modified Source Model Objects
    • D: Deleted Source Model Objects

    • Click on the NEXT button
    • You can select the new available Source Model Object to create:
    • By clicking on the UPDATE button:
      • The A - Added Source Model Objects you selected in step 3 will be created
      • The M - Modified Source Model Objects will be updated
      • The D - Deleted Source Model Objects will be deleted
  • The Discovery (and Source system if a new one was created) will appear in the Source System list:

    • The Discovery's name is the Date and Time of the Discovery creation
  • A notification confirms the correct creation:

Depending on your license, you may have limitations on the number of Source Systems per Project.

If this is the case, when this limitation is reached, you will see the following message and will not be able to create a new Source system:

Demonstration video