Do a Discovery with the biGENIUS Discovery App CLI (Walkthrough)

Prerequisites

To follow this walkthrough, you will need a CSV test dataset and the biGENIUS Discovery App CLI for your operating system.

This guide is based on the biGENIUS Discovery App CLI version 3.0.2.

First, extract the test files archive directly into the directory of your Discovery App.

The resulting structure should look something like this:

C:/discovery-app
│ bigenius-discovery-cli.exe
└───bigenius_discovery_test_data_ADW
CreditCard.csv
Currency.csv
CurrencyRate.csv
Customer.csv
...

Start biGENIUS Discovery App

Open the integrated terminal of your operating system and navigate to the directory containing the bigenius-discovery executable.

Windows
- In your terminal, run:

./bigenius-discovery-cli.exe

Linux
- First, you need to make the application executable. In your terminal, run the command:

chmod +x bigenius-discovery-cli

- Then, you can start the application by running the command:

./bigenius-discovery-cli

MacOS
- In your terminal, try running the app with the command:

./bigenius-discovery-cli

- If you receive a warning from the OS that the app cannot be started, you may need to bypass the macOS Gatekeeper service
- Run the command:

xattr -d com.apple.quarantine bigenius-discovery-cli

- Then, try to execute the app again:

./bigenius-discovery-cli

When running the app for the first time on your machine, you should see something like this in your terminal output:

When the app is started for the first time, some resources are created in the background.
You can find these resources in the respective user-related directories of your operating system:

Windows: %LOCALAPPDATA%\\biGENIUS\\DiscoveryApp
Linux: ~/.local/share/biGENIUS/DiscoveryApp/
macOS: ~/.local/share/biGENIUS/DiscoveryApp

Create New Discovery Configuration

After starting the app in your terminal, we can now create our first discovery configuration using the test data we downloaded earlier.

First, let's review the types of discovery configurations we can create in the app. Since our source data consists of simple CSV files, we are looking to create a discovery related to CSV files.

Run the command:

create --list

This will list all available discovery types that we can use to create a new discovery configuration, and looks like this:

The list shows us a Discovery Type called CSV Local, which is exactly what we need.

In the app, we can create a new discovery configuration using the create command. For the create command, we need to provide a handful of parameters. We can determine these by inspecting the discovery type CSV Local.

Run the command:

inspect --type "CSV Local"

This will show us a full description of the discovery type, and it looks like this:

Using this information, we can construct our create command.

The FilePathType for our example is Directory since all our CSV files are in a single directory.
The FilePath will be a relative path to the example data directory in our Discovery App folder: bigenius_discovery_test_data_bfm/
The ColumnDelimiter for all our files is a single pipe symbol |.
The RowDelimiter of our files will be \\\\r\\\\n. We need to escape the backslashes for the configuration to work correctly from the command line.
FieldQuote and Length can be ignored for this example.
The Name property is special, as all configurations require one, and it is provided differently.

Let's construct our command. The create command needs the following arguments:

Discovery Type: In our case, this will be CSV Local.
Name: You can choose the name yourself. A suggestion is csv_local_bfm_disc_01.
Properties: The values we decided on earlier should be provided as a space-separated list to this argument, like:

FilePathType="Directory" FilePath="bigenius_discovery_test_data_ADW/" ColumnDelimiter="|" RowDelimiter="\\r\\n"

Execute the full command:

create "CSV Local" --name csv_local_ADW --properties FilePathType="Directory" FilePath="bigenius_discovery_test_data_ADW/" ColumnDelimiter="|" RowDelimiter="\\r\\n"

When the command completes, you should see output like this:

We can now inspect the discovery configuration to verify that the values were set correctly.
Run the command:

inspect --name csv_local_ADW

This will show us the content of the configuration as we defined it in the create command:

Run Discovery

Before we run our discovery, let's check the available discovery configurations. Run the command:

discover --list

You will see a list of all existing discovery configurations. If you haven't created any other configurations, you will see the one we just created. The output should look like this:

Now we can run the discovery and let the app generate the discovery file that we need for biGENIUS-X. Usually, the discovery file will be written by default into a directory structure within your operating system's user-specific directories. We can override this behavior and define our own output directory and file.

Execute the command:

discover "csv_local_ADW" --output "discovery_file_csv_local_ADW.yaml"

If the discovery generation process was successful, you will see output like this:

Find the generated discovery file in the same directory as the application. The content of the generated discovery file should look like this:

You can now take this file and upload it to a Source System in biGENIUS-X.

Run Discovery against biGENIUS-X Data Marketplace

If you are using the biGENIUS-X Data Marketplace to manage your source systems, you can also directly upload a discovery to the Data Marketplace without creating the file locally and uploading it manually.

Create a new Source System called ADW in the Data Marketplace in biGENIUS-X.

Afterwards, you can run the discover command again, but this time we will provide additional information to connect directly to the biGENIUS-X API to upload the generated discovery file.

Run the command:

discover "csv_local_ADW" --api-connection-properties ClientId="YOUR CLIENT ID" ClientSecret="YOUR CLIENT SECRET" SourceSystemName="ADW" Version="1.0.0"

If the upload was successful, the output should look like this:$

Side Note - Run Discovery with Sensitive Properties

The biGENIUS Discovery App does not save any sensitive information for source systems that require passwords or access keys. When running a discovery against such a source system, we need to provide the values for sensitive properties in the discover command directly.

Let's say we have a Microsoft SQL Server database that we want to discover. When using SQL Authentication, we have to provide a password in the discover command.

To find out what properties are sensitive and need to be provided by the user, we can get a list of properties directly in the CLI by running the command:

discover <Discovery Name> --list-sensitive

Running this command against a SQL Server discovery configuration, the result would look like this:

This tells us that we need to provide a value for the Password property.

The command for our example source system would look like this:

discover "ADW2019_Sales" --sensitive-properties Password="mySecretPassword"

Create Discovery with External Config File

In an earlier step, we created a discovery configuration by providing each parameter through the command line. This can sometimes be tedious when we need to provide many parameters.

To simplify this process, we also support loading external configuration files that contain the parameter values for a discovery configuration.

In the directory where your executable is located, create a new file called csv_config.yaml.

Add the following content to the file and save it:

FilePathType: "Directory"
FilePath: "bigenius_discovery_test_data_ADW/"
ColumnDelimiter: "|"
RowDelimiter: "\r\n"

This is the same content that we also provided via the parameters for the create command earlier.

Let's create a new discovery configuration by using the --sensitive-properties-file argument instead of providing the parameters themselves.

Run the command:

create "CSV Local" --name csv_local_ADW_2 --properties-file "csv_config.yaml"

The output is similar to the output from the previous create command:

We can also inspect the newly created configuration:

inspect --name csv_local_ADW_2

This will show something like this:

The discover command for this newly created configuration follows the same pattern as before:

discover "csv_local_ADW_2" --output "discovery_file_csv_local_ADW_2.yaml"

This results in a response like this:

Create and Discover in One Command

The Discovery CLI also supports a single-command solution if you want to create, run, and delete a discovery configuration using a single command.
Instead of using the create command, then the discover command, and potentially the delete command, you can use the create-and-discover command with the --delete-after argument.

Run the following command to see it in action:

create-and-discover "CSV Local" --name "csv_local_ADW_3" --properties-file "csv_config.yaml" --output "csv_local_ADW_3.yaml" --delete-after

The result should look like this:

We can see that the command performs three independent actions: First, it creates the discovery configuration, similar to the create command. Then, it runs the discovery configuration, similar to the discover command. Finally, it deletes the configuration, similar to the delete command.

Of course, this can also be combined with the --api-connection-properties argument to directly upload the discovery file to the biGENIUS Data Marketplace.

This command is useful if you want to integrate the discovery file generation into pipelines or other automated processes that support command-line execution.

Execute Commands Outside the Discovery App

The create-and-discover command provides a solution to generate discovery files in a single command. We can also execute this command from outside the Discovery App's interactive mode by providing the command and its arguments as parameters to the executable itself.

If you are still inside the Discovery App's CLI, run the exit command to return to your terminal.

Instead of running the executable and then running the create-and-discover command within the interactive shell, we can directly send the command to the executable.

Run the following command directly from your operating system's terminal while located in the Discovery App directory created earlier:

./bigenius-discovery-cli create-and-discover "CSV Local" --name "csv_local_bfm_disc_03" --properties-file "csv_config.yaml" --output "discovery_file_csv_local_bfm_disc_03.yaml" --delete-after

You will see that the app executes the provided command without entering the interactive shell mode and returns directly to the terminal:

As mentioned, this approach allows you to integrate discovery file generation into various automation processes, like CI/CD pipelines.