Skip to content
  • There are no suggestions because the search field is empty.

Do a Discovery with the biGENIUS Discovery App CLI (Walkthrough)

 

 

Prerequisites

To follow this walkthrough, you will need a CSV test dataset and the biGENIUS Discovery App CLI for your operating system.

This guide is based on the biGENIUS Discovery App CLI version 2.0.0.
 

First, extract the test files archive directly into the directory of your Discovery App.

The resulting structure should look something like this:

C:/discovery-app
│   bigenius-discovery-cli.exe
└───bigenius_discovery_test_data_bfm
        BDC_CUSTOMER.csv
        BDC_EMPLOYEE.csv
        BDC_ITEM_BASIC.csv
        BDC_ITEM_OFFER.csv
        ...

Start biGENIUS Discovery App

Open the integrated terminal of your operating system and navigate to the directory containing the bigenius-discovery executable.

  • Windows
    In your terminal, run:

./bigenius-discovery-cli.exe 

 

  • Linux
    First, you need to make the application executable. In your terminal, run the command:
chmod +x bigenius-discovery-cli

 

Then, you can start the application by running the command:

./bigenius-discovery-cli

 

  • macOS
    In your terminal, try running the app with the command:
    ./bigenius-discovery-cli

    If you receive a warning from the OS that the app cannot be started, you may need to bypass the macOS Gatekeeper service

    Run the command:
    xattr -d com.apple.quarantine bigenius-discovery-cli

    Then, try to execute the app again:
    ./bigenius-discovery-cli

 

When running the app for the first time on your machine, you should see something like this in your terminal output:

>> .\bigenius-discovery-cli.exe
User Settings Directory not found. The directory will be automatically created...
[INF]: Setting up prerequisites...
User Settings Directory not found. The directory will be automatically created...
[INF]: Initializing database context...
[INF]: Database context initialized successfully
[INF]: biGENIUS Discovery CLI version 2.0.0

biGENIUS Discovery CLI - Interactive Mode
Type 'help' to see available commands or 'exit' to quit
Use up/down arrow keys to navigate command history

[INF]: Loading discovery configurations...
[INF]: No discovery configurations found
>
 

When the app is started for the first time, some resources are created in the background.
You can find these resources in the respective user-related directories of your operating system:

  • Windows: %LOCALAPPDATA%\\biGENIUS\\DiscoveryApp

  • Linux: ~/.local/share/biGENIUS/DiscoveryApp/

  • macOS: ~/.local/share/biGENIUS/DiscoveryApp

Create New Discovery Configuration


After starting the app in your terminal, we can now create our first discovery configuration using the test data we downloaded earlier.

First, let's check what kind of discovery configurations we can create in the app. Since our source data consists of simple CSV files, we are looking to create a discovery related to CSV files.

Run the command:

create --list

This will list all available discovery types that we can use to create a new discovery configuration and looks like this:

The list shows us a Discovery Type called "CSV Local", which is exactly what we need.

In the app, we can create a new discovery configuration using the create command. For the create command, we need to provide a handful of parameters. We can determine these by inspecting the discovery type CSV Local. Run the command:

inspect --type "CSV Local"


This will show us a full description of the discovery type and looks like this:

> inspect --type "CSV Local"
Properties for discovery type 'CSV Local':

 ------------------------------------------------------
 | Property Name   | Data Type | Required | Sensitive |
 ------------------------------------------------------
 | FilePathType    | String    | Yes      | No        |
 ------------------------------------------------------
 | FilePath        | String    | Yes      | No        |
 ------------------------------------------------------
 | ColumnDelimiter | String    | Yes      | No        |
 ------------------------------------------------------
 | Length          | String    | No       | No        |
 ------------------------------------------------------
 | FieldQuote      | String    | No       | No        |
 ------------------------------------------------------
 | RowDelimiter    | String    | Yes      | No        |
 ------------------------------------------------------
 | Name            | String    | Yes      | No        |
 ------------------------------------------------------

 Count: 7

Allowed values for lookup properties:

  PathTypes:
    - Single CSV File
    - Directory
    - Directory With Partitioned Entities

  RowDelimiters:
    -
    - \n
    - \r\n
    - \r

Using this information, we can construct our create command.

  • The FilePathType for our example is Directory since all our CSV files are in a single directory.

  • The FilePath will be a relative path to the example data directory in our Discovery App folder: bigenius_discovery_test_data_bfm/

  • The ColumnDelimiter for all our files is a single pipe symbol |.

  • The RowDelimiter of our files will be \\\\r\\\\n. We need to escape the backslashes for the configuration to work correctly from the command line.

  • FieldQuote and Length can be ignored for this example.

  • The Name property is special, as all configurations require one, and it is provided differently.

Let's construct our command. The create command needs the following arguments:

  • Discovery Type: In our case, this will be CSV Local.

  • Name: You can choose the name yourself. A suggestion is csv_local_bfm_disc_01.

  • Properties: The values we decided on earlier should be provided as a space-separated list to this argument, like: FilePathType="Directory" FilePath="bigenius_discovery_test_data_bfm/" ColumnDelimiter="|" RowDelimiter="\\\\r\\\\n"

Execute the full command:

create "CSV Local" --name csv_local_bfm_disc_01 --properties FilePathType="Directory" FilePath="bigenius_discovery_test_data_bfm/" ColumnDelimiter="|" RowDelimiter="\\r\\n"

When the command completes, you should see output like this:

[INF]: Creating new discovery configuration of type CSV Local with name csv_local_bfm_disc_01 ...
[INF]: Setting property 'name' to csv_local_bfm_disc_01
[INF]: Setting property FilePathType to Directory
[INF]: Setting property FilePath to bigenius_discovery_test_data_bfm
[INF]: Setting property ColumnDelimiter to |
[INF]: Setting property RowDelimiter to \r\n
Successfully saved 1 items to CsvDirectories directory.
[INF]: csv_local_bfm_disc_01 has been added successfully.
[INF]: Successfully created discovery configuration 'csv_local_bfm_disc_01' of type 'CSV Local

We can now inspect the contents of the discovery configuration to see if the values were set properly.
Run the command:

inspect --name csv_local_bfm_disc_01

This will show us the content of the configuration as we defined it in the create command:

 

Run Discovery

Before we run our discovery, let's check the available discovery configurations. Run the command:

discover --list

You will see a list of all existing discovery configurations. If you haven't created any other configurations, you will see the one we just created. The output should look like this:

> discover --list

Available discovery configurations:

 -------------------------------------
 | Type      | Name                  |
 -------------------------------------
 | CSV Local | csv_local_bfm_disc_01 |
 -------------------------------------

 Count: 1 
 
Now we can run the discovery and let the app generate the discovery file that we need for biGENIUS-X. Usually, the discovery file will be written by default into a directory structure within your operating system's user-specific directories. We can override this behavior and define our own output directory and file.

Execute the command:

discover "csv_local_bfm_disc_01" --output "discovery_file_csv_local_bfm_disc_01.yaml"

If the discovery generation process was successful, you will see output like this:

[INF]: Running discovery for csv_local_bfm_disc_01
[INF]: Found discovery configuration 'csv_local_bfm_disc_01' (CSV Local)
[INF]: Loading source system data for 'csv_local_bfm_disc_01 (CSV Local)'...
[INF]: Found and selected 29 Source Objects for discovery configuration csv_local_bfm_disc_01 (CSV Local)
[INF]: Generating discovery file for 'csv_local_bfm_disc_01 (CSV Local)'...
[DBG]: Generating discovery file for source system csv_local_bfm_disc_01 with version 2.0.0.
[INF]: Discovery file successfully created at discovery_file_csv_local_bfm_disc_01.yaml
Successfully saved 1 items to CsvDirectories directory.

Find the generated discovery file in the same directory as the application. The content of the generated discovery file should look like this:

id: '8ab62f79-5ce0-4baa-8b9c-3dee4e9f683d'
name: 'csv_local_bfm_disc_01'
timestamp: '2025-06-02T08:43:27+00:00'
appVersion: '2.0.0'
sourceObjects:
  - name: 'BDC_CUSTOMER'
    terms:
      - name: 'BDC_SHOP_ID'
        isBusinessKey: false
        isDefaultTerm: false
...

You can now take this file and upload it to a Source System in biGENIUS-X.

 

Run Discovery Against biGENIUS-X Data Marketplace

If you are using the biGENIUS-X Data Marketplace to manage your source systems, you can also directly upload a discovery to the Data Marketplace without creating the file locally and uploading it manually.

Create a new Source System called BFM CSV in the Data Marketplace in biGENIUS-X.

Afterwards, you can run the discover command again, but this time we will provide additional information to connect directly to the biGENIUS-X API to upload the generated discovery file. Run the command:

 discover "csv_local_bfm_disc_01" --api-connection-properties ClientId="YOUR CLIENT ID" ClientSecret="YOUR CLIENT SECRET" SourceSystemName="BFM CSV" Version="1.0.0"
 
If the upload was successful, the output should look like this:
 
[INF]: Running discovery for csv_local_bfm_disc_01
[INF]: Found discovery configuration 'csv_local_bfm_disc_01' (CSV Local)
[INF]: Applied API Storage Adapter property ClientId
[INF]: Applied API Storage Adapter property ClientSecret
[INF]: Applied API Storage Adapter property SourceSystemName
[INF]: Applied API Storage Adapter property Version
[INF]: Loading source system data for 'csv_local_bfm_disc_01 (CSV Local)'...
[INF]: Found and selected 29 Source Objects for discovery configuration csv_local_bfm_disc_01 (CSV Local)
[INF]: Generating discovery file for 'csv_local_bfm_disc_01 (CSV Local)'...
[DBG]: Generating discovery file for source system csv_local_bfm_disc_01 with version 2.0.0.
[INF]: File successfully uploaded via API.
 
 

Side Note: Run Discovery with Sensitive Properties

The biGENIUS Discovery App does not save any sensitive information for source systems that require passwords or access keys. When running a discovery against such a source system, we need to provide the values for sensitive properties in the discover command directly.

Let's say we have a Microsoft SQL Server database that we want to discover. When using SQL Authentication, we have to provide a password in the discover command.

To find out what properties are sensitive and need to be provided by the user, we can get a list of properties directly in the CLI by running the command discover --name <Discovery Name> --list-sensitive.

Running this command against a SQL Server discovery configuration, the result would look like this:

Required sensitive properties for discovery configurations:

 -----------------------------------------------------
 | Type        | Name          | Sensitive Properties |
 -----------------------------------------------------
 | SQL Server  | 01-sql-server | Password             |
 -----------------------------------------------------

 Count: 1

 

This tells us that we need to provide a value for the Password property.

The command for our imaginary source system would look like this:

discover "sql_server_prod" --property Password="mySecretPassword"

 

Create Discovery with External Config File

In an earlier step, we created a discovery configuration by providing each parameter through the command line. This can sometimes be tedious when we need to provide many parameters.

To make this process easier, we also support loading external configuration files that contain the parameter values for a discovery configuration.

In the directory where your executable is located, create a new file called csv_config.yaml.

Add the following content to the file and save it:

FilePathType: "Directory"
FilePath: "bigenius_discovery_test_data_bfm/"
ColumnDelimiter: "|"
RowDelimiter: "\r\n"

This is the same content that we also provided via the parameters for the create command earlier.

Let's create a new discovery configuration by using the --properties-file argument instead of providing the parameters themselves. Run the command:

create "CSV Local" --name csv_local_bfm_disc_02 --properties-file "csv_config.yaml"
 

The output is similar to the output from the previous create command:

> create "CSV Local" --name csv_local_bfm_disc_02 --properties-file "csv_config.yaml"
[INF]: Creating new discovery configuration of type CSV Local with name csv_local_bfm_disc_02 ...
[INF]: Setting property 'name' to csv_local_bfm_disc_02
[INF]: Setting property FilePathType to Directory
[INF]: Setting property FilePath to bigenius_discovery_test_data_bfm/
[INF]: Setting property ColumnDelimiter to |
[INF]: Setting property RowDelimiter to \r\n
Successfully saved 2 items to CsvDirectories directory.
[INF]: csv_local_bfm_disc_02 has been added successfully.
[INF]: Successfully created discovery configuration 'csv_local_bfm_disc_02' of type 'CSV Local'

 

We can also inspect the newly created configuration:

inspect --name csv_local_bfm_disc_02

This will show something like this:

> inspect --name csv_local_bfm_disc_02
[INF]: Inspecting discovery configuration csv_local_bfm_disc_02...

Configuration 'csv_local_bfm_disc_02' (CSV Local):


 ------------------------------------------------------------
 | Property Name     | Value                                  |
 ------------------------------------------------------------
 | FilePathType      | Directory                              |
 ------------------------------------------------------------
 | FilePath          | bigenius_discovery_test_data_bfm/      |
 ------------------------------------------------------------
 | ColumnDelimiter   | |                                      |
 ------------------------------------------------------------
 | Length            |                                        |
 ------------------------------------------------------------
 | FieldQuote        |                                        |
 ------------------------------------------------------------
 | RowDelimiter      | \r\n                                   |
 ------------------------------------------------------------
 | Id                | c7b08903-30cb-4e76-9557-1483d9fb6ca1   |
 ------------------------------------------------------------
 | Name              | csv_local_bfm_disc_02                  |
 ------------------------------------------------------------
 | LastExecutionTime | null                                   |
 ------------------------------------------------------------

 Count:

The discover command for this newly created configuration follows the same pattern as before:

discover "csv_local_bfm_disc_02" --output "discovery_file_csv_local_bfm_disc_02.yaml"

This results in a response like this:

> discover "csv_local_bfm_disc_02" --output "discovery_file_csv_local_bfm_disc_02.yaml"
[INF]: Running discovery for csv_local_bfm_disc_02
[INF]: Found discovery configuration 'csv_local_bfm_disc_02' (CSV Local)
[INF]: Loading source system data for 'csv_local_bfm_disc_02 (CSV Local)'...
[INF]: Found and selected 29 Source Objects for discovery configuration csv_local_bfm_disc_02 (CSV Local)
[INF]: Generating discovery file for 'csv_local_bfm_disc_02 (CSV Local)'...
[DBG]: Generating discovery file for source system csv_local_bfm_disc_02 with version 2.0.0.
[INF]: Discovery file successfully created at discovery_file_csv_local_bfm_disc_02.yaml

 

Create and Discover in One Command

The Discovery CLI also supports a single-command solution if you want to create, run, and delete a discovery configuration using a single command.
Instead of using the create command, then the discover command, and potentially the delete command, you can use the create-and-discover command with the --delete-after argument.

Run the following command to see it in action:

create-and-discover "CSV Local" --name "csv_local_bfm_disc_03" --properties-file "csv_config.yaml" --output "discovery_file_csv_local_bfm_disc_03.yaml" --delete-after

The result should look like this:

[INF]: Creating new discovery configuration of type CSV Local with name csv_local_bfm_disc_03 ...
[INF]: Setting property 'name' to csv_local_bfm_disc_03
[INF]: Setting property FilePathType to Directory
[INF]: Setting property FilePath to bigenius_discovery_test_data_bfm/
[INF]: Setting property ColumnDelimiter to |
[INF]: Setting property RowDelimiter to \r\n
Successfully saved 3 items to CsvDirectories directory.
[INF]: csv_local_bfm_disc_03 has been added successfully.
[INF]: Successfully created discovery configuration 'csv_local_bfm_disc_03' of type 'CSV Local'

[INF]: Running discovery for csv_local_bfm_disc_03
[INF]: Found discovery configuration 'csv_local_bfm_disc_03' (CSV Local)
[INF]: Loading source system data for 'csv_local_bfm_disc_03 (CSV Local)'...
[INF]: Found and selected 29 Source Objects for discovery configuration csv_local_bfm_disc_03 (CSV Local)
[INF]: Generating discovery file for 'csv_local_bfm_disc_03 (CSV Local)'...
[DBG]: Generating discovery file for source system csv_local_bfm_disc_03 with version 2.0.0.
[INF]: Discovery file successfully created at discovery_file_csv_local_bfm_disc_03.yaml

[INF]: Deleting discovery configuration csv_local_bfm_disc_03...
[INF]: Successfully deleted discovery configuration csv_local_bfm_disc_03
[INF]: csv_local_bfm_disc_03 has been deleted successfully.

 

 We can see that the command performs three independent actions: First, it creates the discovery configuration, similar to the create command. Then, it runs the discovery configuration, similar to the discover command. Finally, it deletes the configuration, similar to the delete command.

Of course, this can also be combined with the --api-connection-properties argument to directly upload the discovery file to the biGENIUS Data Marketplace.

This command is useful if you want to integrate the generation of the discovery file into pipelines or other automated processes that support command-line execution.

 

Execute Commands Outside the Discovery App

The create-and-discover command provides a solution to generate discovery files in a single command. We can also execute this command from outside the Discovery App's interactive mode by providing the command and its arguments as parameters to the executable itself.

If you are still inside the Discovery App's CLI, run the exit command to return to your terminal.

Instead of running the executable and then running the create-and-discover command within the interactive shell, we can directly send the command to the executable.

Run the following command directly from your operating system's terminal while located in the Discovery App directory created earlier:

./bigenius-discovery-cli create-and-discover "CSV Local" --name "csv_local_bfm_disc_03" --properties-file "csv_config.yaml" --output "discovery_file_csv_local_bfm_disc_03.yaml" --delete-after

You will see that the app executes the provided command without entering the interactive shell mode and returns directly to the terminal:

PS C:\discovery-app\bigenius-discovery-cli-2.0.0-win-x64> ./bigenius-discovery-cli create-and-discover "CSV Local" --name "csv_local_bfm_disc_03" --properties-file "csv_config.yaml" --output "discovery_file_csv_local_bfm_disc_03.yaml" --delete-after

[INF]: Setting up prerequisites...
[INF]: Initializing database context...
[INF]: Database context initialized successfully
[INF]: biGENIUS Discovery CLI version 2.0.0
[INF]: Creating new discovery configuration of type CSV Local with name csv_local_bfm_disc_03 ...
[INF]: Setting property 'name' to csv_local_bfm_disc_03
[INF]: Setting property FilePathType to Directory
[INF]: Setting property FilePath to bigenius_discovery_test_data_bfm/
[INF]: Setting property ColumnDelimiter to |
[INF]: Setting property RowDelimiter to \r\n
[INF]: csv_local_bfm_disc_03 has been added successfully.
[INF]: Successfully created discovery configuration 'csv_local_bfm_disc_03' of type 'CSV Local'

[INF]: Running discovery for csv_local_bfm_disc_03
[INF]: Found discovery configuration 'csv_local_bfm_disc_03' (CSV Local)
[INF]: Loading source system data for 'csv_local_bfm_disc_03 (CSV Local)'...
[INF]: Found and selected 29 Source Objects for discovery configuration csv_local_bfm_disc_03 (CSV Local)
[INF]: Generating discovery file for 'csv_local_bfm_disc_03 (CSV Local)'...
[DBG]: Generating discovery file for source system csv_local_bfm_disc_03 with version 2.0.0.
[INF]: Discovery file successfully created at discovery_file_csv_local_bfm_disc_03.yaml

[INF]: Deleting discovery configuration csv_local_bfm_disc_03...
[INF]: Successfully deleted discovery configuration csv_local_bfm_disc_03
 

As mentioned, this approach allows you to integrate the discovery file generation into various automation processes, like CI/CD pipelines.

 

Demonstration video