Source Systems
      Back to home
      1. biGENIUS-X Knowledge Base
      2. Application Modules
      3. Source Systems

      Add a Discovery

      Add a Discovery

      To create a Discovery for a Project, access the Source Systems overview page for a Project Branch:

      If your source is a database with multiple schemas, you should create one Source system for each schema.

      Then click on the Add Discovery button:

      The Import a discovered file side panel window is opened. It is composed of 2 steps: Add Discovery, then Create Source:

      or

      Each Target Technology has a native Source System. That is a Source System where no special configuration/method is necessary to gather source data. For example, for Snowflake, it is Snowflake; for Microsoft SQL Server, it is Microsoft SQL Server; for Spark, it is Spark.  

      For Spark, the native format was defined as Catalog objects. These tables are already populated by some process and registered in the Catalog (Hive or Unity). It can happen through Delta Sharing, specifying just an external table on an existing data file somewhere, or using a table deployed and loaded by some other process.

      For the discovery with a Source System Type Spark, you can use a Parquet/Delta discovery, as these data types can be mapped to Spark. When you then access the source objects, it is just assumed, that they are already known to the catalog and they are accessed like select * from <source_table_name>, instead of loading the files with their storage path (spark.read....).

      Fill in the following information:

      1.  Add to an existing Source System: There are two options here: add a new Source System or add the Discovery to an existing Source System.

        1. Add a Discovery in a new Source System:
          Select the entry "- Add new Source System -" in the Source system list:

        2. Add a Discovery for an existing Source System:
          Select the Source System in the Source system list:

      2. Source System Type: several Source System Types can be available depending on the Generator Configuration.
        1. Example:
      3. Source: 2 possibilities:

        1. File Upload: select this entry to upload a discovery file 
          Then fill:

          1. Source System Name: fill in a name for your Source System
          2. Discovery Filetype: Select one of the following types:

            1. DataHub: if you are using Discovery files from your usage of DataHub
            2. biGENIUS Discovery: if you are using YAML Discovery files from the biGENIUS-X Discovery Application
          3. Select File: select the Discovery File from your local system.
        2. Data Marketplace: select this entry to choose a Data Product Contract from the Data Marketplace
          Some info here: Understand the Data Marketplace overview
          Then fill:
          1. Source System Name: choose the Source System available in the Data Marketplace:
          2. Data Contract: choose a Data Contract version of the Source System
      4. Description (optional): fill in a description for your Source System

      If you do not select a valid file type or a file that is not properly structured, you will receive the following error: This discovery file is not compatible.

      Then, click on the NEXT button:

      • If you created a new Source System, the Create Source step is displayed in the side panel window. See Create Source Objects from a Discovery.
      • If you added a Discovery in an existing Source System, the Impact of updating current Source Model with step will be displayed: it contains an overview of the differences with the previous Source Model Objects created with the previous Discovery:
        • A: Added Source Model Objects
        • M: Modified Source Model Objects
        • D: Deleted Source Model Objects
      • By clicking on the UPDATE SOURCE MODEL button:
        • The A - Added Source Model Objects will not be created - Please Create Source Objects from a Discovery to add them
        • The M - Modified Source Model Objects will be updated
        • The D - Deleted Source Model Objects will be deleted
      • The Discovery (and Source system if a new one was created) will appear in the Source System list:

        • The Discovery's name is the Date and Time of the Discovery creation
      • A notification confirms the correct creation:

      Depending on your license, you may have limitations on the number of Source Systems per Project.

      If this is the case, when this limitation is reached, you will see the following message and will not be able to create a new Source system:

      Demonstration video