Skip to content
Contact us Go to Customer Portal
  • There are no suggestions because the search field is empty.

Databricks - Target environment with Unity Catalog

Applies to
Platform: Databricks

A Databricks target environment with Unity Catalog needs a workspace enabled for Unity Catalog, a catalog, an all-purpose cluster and the Databricks CLI. Prepare it before you replace the placeholders and deploy the artifacts of a Databricks generator. Afterwards deploy.ps1 can upload the notebooks and run the deployment on the cluster.

This article applies when the project property Use Unity Catalog is selected. For projects without it, see Databricks - Target environment with the Hive metastore. biGENIUS support does not cover the installation and configuration of Databricks beyond the example in this article.

Prerequisites

  • A Databricks workspace enabled for Unity Catalog, for example an Azure Databricks workspace.
  • A user who can create schemas in the target catalog, for example as its owner, and who can create an all-purpose cluster.
  • Use Unity Catalog is selected in Project Settings, the default since Generator 2.0:
    Project property Use Unity Catalog selected
  • PowerShell 7 on the deploying machine. deploy.ps1 requires it.

Prepare the catalog

  1. Create a catalog, for example in Catalog Explorer or with CREATE CATALOG datalakehouse;, or choose an existing one. The deployment creates the schemas of the layers and of the load control, but not the catalog.
  2. Enter the catalog in the database_name placeholders and the schemas in the schema_name placeholders, see Databricks - Placeholder values. The layer schemas are prefilled with the layer names; the load control schema is empty.

Create the cluster

deploy.ps1 runs the deployment on an all-purpose cluster that you name, and the Airflow jobs run on an existing cluster too.

  1. In the workspace, open Compute and choose Create compute.
  2. Enter a name, for example dwh_cluster, and set:
    • Access mode: Dedicated (single user).
    • Databricks runtime version: 18 LTS.
    • Single node with a node type such as Standard_DS3_v2.
  3. Choose Create compute and wait until the cluster is running.

With Databricks Runtime 17.3 LTS, the deployment of tables with a UNIQUE constraint, such as hubs, fails while Use Unique Constraints is selected, see Databricks - Deployment.

Install the Databricks CLI

  1. Install the Databricks CLI, for example with winget install Databricks.DatabricksCLI.
  2. Open a new PowerShell 7 window and run databricks -v. The CLI prints its version.

On its first run, deploy.ps1 creates a CLI profile and signs you in through the browser, see Databricks - Deployment.

Provide the source data

Which source system types a generator configuration accepts is listed in Source systems per generator configuration. To create source model objects from tables of a catalog, see Discover a Databricks Unity Catalog. A Data Mart project has no discovery; it uses model objects of linked projects, see Manage Linked Projects.

Files with Stage Files

Parquet, Delta or CSV files are loaded by a separate Databricks Stage Files project. A second project, for example a Data Vault project, then uses its Stage model objects as a linked project.

  1. Upload the files into a Unity Catalog volume, one folder per source model object, for example /Volumes/<catalog>/<schema>/<volume>/sales/customer/.
  2. Enter the parent folder with a trailing slash as file_path, here /Volumes/<catalog>/<schema>/<volume>/sales/. The loader reads the folder named like the source model object below it.

A volume needs no storage settings on the cluster.

Databases with Stage JDBC

Tables of MariaDB, Microsoft SQL Server, MySQL, Oracle or PostgreSQL are loaded by a separate Databricks Stage JDBC project, used as a linked project in the same way. The cluster must reach the database. For Azure SQL Database, select Allow Azure services and resources to access this server in the networking settings of the server:

Azure SQL server networking exception Allow Azure services and resources to access this server selected

Enter the connection in the jdbc_url, jdbc_driver, jdbc_user and jdbc_password placeholders, and the source schema in schema_name of the source system.

Check the result

In a notebook attached to the cluster, run SHOW SCHEMAS IN <catalog>;. It lists the schemas of the catalog without an error. For Stage Files, LIST '<file_path>' lists one folder per source model object.

Failure symptoms and recovery

The deployment script does not find the Databricks CLI

Message: Databricks CLI is not installed, not recognized or the PATH environment variable is not properly configured.
Cause: The CLI is not installed, or its folder is not in PATH of the PowerShell window.
Fix: Install the CLI, open a new PowerShell 7 window and check databricks -v.

For failures in other steps, see Troubleshoot generation, deployment and load failures.

Release differences

  • Use Unity Catalog is selected by default since Generator 2.0. Up to Generator 1.10 it was cleared by default.
  • Since Generator 2.2, the deployment creates the schemas with Generate Create Schema If Not Exists Statements, a default of Deployment Options. With older artifacts, create the schemas yourself.
  • Up to Generator 1.9, a Stage JDBC project took the source schema from the model object name, for example Sales.CreditCard. Since Generator 1.10 it comes from schema_name of the source system.

Related articles