--- title: Databricks - Target environment with the Hive metastore description: A Databricks target environment with the Hive metastore needs a workspace, a data lake storage account, an all-purpose cluster and the Databricks CLI. --- [Skip to content](https://knowledge.bigenius-x.com/databricks-target-environment-with-the-hive-metastore#main-content) [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en) [![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/) Open main navigation Close main navigation - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new) - [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal) How can we help you? - There are no suggestions because the search field is empty. 1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en) 2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en) 3. [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment) # Databricks - Target environment with the Hive metastore **Applies to** Platform: Databricks A Databricks target environment with the Hive metastore needs a workspace, a data lake storage account, an all-purpose cluster and the Databricks CLI. Prepare it before you replace the placeholders and deploy the artifacts of a Databricks generator. Afterwards `deploy.ps1` can upload the notebooks and run the deployment on the cluster. This article applies when the project property [Use Unity Catalog](https://knowledge.bigenius-x.com/use-unity-catalog-project?hsLang=en) is cleared. For projects with Unity Catalog, see [Databricks - Target environment with Unity Catalog](https://knowledge.bigenius-x.com/databricks-target-environment-with-unity-catalog?hsLang=en). biGENIUS support does not cover the installation and configuration of Databricks beyond the example in this article. ### Prerequisites - An Azure Databricks workspace and a user who can create an all-purpose cluster. - An Azure storage account with hierarchical namespace for the data lake, and a service principal that can read and write it. Its client ID and secret are stored in a Databricks secret scope. - Use Unity Catalog is cleared in Project Settings: ![Project property Use Unity Catalog cleared](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Sep-09-2024-08-30-20-1228-AM.png?width=640&height=43&name=image-png-Sep-09-2024-08-30-20-1228-AM.png) - PowerShell 7 on the deploying machine. `deploy.ps1` requires it. ### Prepare the data lake 1. Create a container or folder for the data lake in the storage account, for example `datalake`. 2. Enter one folder per layer in the `database_directory` placeholders, for example `abfss://datalake@.dfs.core.windows.net/rawvault/`. See [Databricks - Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en). The deployment creates each layer database with `CREATE DATABASE IF NOT EXISTS` and the location of its `database_directory`. The database names are the `database_name` placeholders, prefilled with the layer names of the project, for example `stage`, `rawvault`, `businessvault` and `mart`. ### Create the cluster 1. In the workspace, open Compute and choose Create compute. 2. Enter a name, for example `dwh_cluster`, and choose the node type, for example `Standard_DS3_v2` as a single node. 3. Under Advanced, Spark, enter the Spark configuration for each storage account the cluster reads or writes. With a service principal, the lines are: ``` fs.azure.account.auth.type..dfs.core.windows.net OAuth fs.azure.account.oauth.provider.type..dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider fs.azure.account.oauth2.client.id..dfs.core.windows.net {{secrets//clientid}} fs.azure.account.oauth2.client.secret..dfs.core.windows.net {{secrets//clientsecret}} fs.azure.account.oauth2.client.endpoint..dfs.core.windows.net https://login.microsoftonline.com//oauth2/token ``` Here `` is the storage account name and `` the secret scope with the secrets `clientid` and `clientsecret`. Details are in [Connect to Azure Data Lake Storage](https://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage). 4. Choose Create compute and wait until the cluster is running. ### Install the Databricks CLI 1. Install the [Databricks CLI](https://docs.databricks.com/aws/en/dev-tools/cli/install), for example with `winget install Databricks.DatabricksCLI`. 2. Open a new PowerShell 7 window and run `databricks -v`. The CLI prints its version. On its first run, `deploy.ps1` creates a CLI profile and signs you in through the browser, see [Databricks - Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en). ### Provide the source data Which source system types a generator configuration accepts is listed in [Source systems per generator configuration](https://knowledge.bigenius-x.com/source-systems-per-generator-configuration?hsLang=en). A Data Mart project has no discovery; it uses model objects of linked projects, see [Manage Linked Projects](https://knowledge.bigenius-x.com/manage-linked-projects?hsLang=en). #### Files with Stage Files Parquet, Delta or CSV files are loaded by a separate Databricks Stage Files project. A second project, for example a Data Vault project, then uses its Stage model objects as a linked project. 1. Upload the files into a storage account the cluster can read, one folder per source model object, for example `abfss://source@.dfs.core.windows.net/sales/customer/`. 2. Enter the parent folder with a trailing slash as `file_path`, here `abfss://source@.dfs.core.windows.net/sales/`. The loader reads the folder named like the source model object below it. To create the source model objects from the files, see [Discover Parquet or Delta files on Azure](https://knowledge.bigenius-x.com/discover-parquet-or-delta-files-on-azure?hsLang=en). #### Databases with Stage JDBC Tables of MariaDB, Microsoft SQL Server, MySQL, Oracle or PostgreSQL are loaded by a separate Databricks Stage JDBC project, used as a linked project in the same way. The cluster must reach the database. For Azure SQL Database, select Allow Azure services and resources to access this server in the networking settings of the server: ![Azure SQL server networking exception Allow Azure services and resources to access this server selected](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Feb-22-2024-08-57-29-5471-AM.png?width=389&height=77&name=image-png-Feb-22-2024-08-57-29-5471-AM.png) The cluster has the JDBC drivers of Microsoft SQL Server, MariaDB and PostgreSQL. For Oracle and MySQL, install the JDBC driver as a library of the cluster, for example from Maven: `com.oracle.database.jdbc:ojdbc11` or `com.mysql:mysql-connector-j`, and enter its driver class in `jdbc_driver` (`oracle.jdbc.OracleDriver`, `com.mysql.cj.jdbc.Driver`). Enter the connection in the `jdbc_url`, `jdbc_driver`, `jdbc_user` and `jdbc_password` placeholders, and the source schema in `schema_name` of the source system. ### Check the result In a notebook attached to the cluster, list the data lake and, for Stage Files, the source folder: ``` dbutils.fs.ls("abfss://datalake@.dfs.core.windows.net/") dbutils.fs.ls("") ``` The first call lists the container content without an access error. The second lists one folder per source model object. ### Failure symptoms and recovery #### The deployment script does not find the Databricks CLI Message: `Databricks CLI is not installed, not recognized or the PATH environment variable is not properly configured.` Cause: The CLI is not installed, or its folder is not in `PATH` of the PowerShell window. Fix: Install the CLI, open a new PowerShell 7 window and check `databricks -v`. For failures in other steps, see [Troubleshoot generation, deployment and load failures](https://knowledge.bigenius-x.com/troubleshoot-generation-deployment-and-load-failures?hsLang=en). ### Release differences - Up to Generator 1.10, Use Unity Catalog was cleared by default. Since Generator 2.0 it is selected by default; clear it for the Hive metastore. - Up to Generator 1.9, a Stage JDBC project took the source schema from the model object name, for example `Sales.CreditCard`. Since Generator 1.10 it comes from `schema_name` of the source system. ### Related articles - Next: [Databricks - Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en), [Databricks - Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en) - Reference: [Use Unity Catalog (Project)](https://knowledge.bigenius-x.com/use-unity-catalog-project?hsLang=en), [Databricks - Stage Files](https://knowledge.bigenius-x.com/databricks-stage-files?hsLang=en), [Databricks - Stage JDBC](https://knowledge.bigenius-x.com/databricks-stage-jdbc?hsLang=en), [Source systems per generator configuration](https://knowledge.bigenius-x.com/source-systems-per-generator-configuration?hsLang=en) - See also: [Databricks - Target environment with Unity Catalog](https://knowledge.bigenius-x.com/databricks-target-environment-with-unity-catalog?hsLang=en), [Databricks - Artifacts](https://knowledge.bigenius-x.com/databricks-artifacts?hsLang=en), [Discover Parquet or Delta files on Azure](https://knowledge.bigenius-x.com/discover-parquet-or-delta-files-on-azure?hsLang=en), [Manage Linked Projects](https://knowledge.bigenius-x.com/manage-linked-projects?hsLang=en), [Native - Load control environment](https://knowledge.bigenius-x.com/native-load-control-environment?hsLang=en), [Apache Airflow - Load control environment](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en) - [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content) - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines) - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working) - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account) - [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en) - [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en) - [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content) - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration) - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions) - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features) - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects) - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches) - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections) - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview) - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps) - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms) - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping) - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships) - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality) - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes) - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling) - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts) - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings) - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace) - [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content) - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse) - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse) - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake) - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server) - [Oracle](https://knowledge.bigenius-x.com/generators?hsLang=en#oracle) - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts) - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders) - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment) - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment) - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline) - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment) - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment) - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control) - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type) - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties) - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms) - [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content) - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations) - [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content) - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches) - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases) - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules) - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules) - [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en) - [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content) - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application) - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application) - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators) - [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content) - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs) - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits) - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) - [Customer Portal](https://knowledge.bigenius-x.com/customer-portal) - [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login) [![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com) All rights reserved © 2025, biGENIUS AG