---
title: Databricks - Load data - Apache Airflow Load Control
description: Load the deployed Databricks solution with Apache Airflow by running the DAG that the generator writes.
---

[Skip to content](https://knowledge.bigenius-x.com/databricks-load-data-apache-airflow-load-control#main-content)

[Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en)

[![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/)

Open main navigation

Close main navigation

- [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new)
- [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal)

 How can we help you?

- There are no suggestions because the search field is empty.

1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en)
2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en)
3. [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow)

# Databricks - Load data - Apache Airflow Load Control

**Applies to**  
Platform: Databricks

Load the deployed Databricks solution with Apache Airflow by running the DAG that the generator writes. The DAG has one task per loader, runs them in dependency order and starts for each one a Databricks job that runs the loader notebook. Afterwards the Graph of the run shows the state of every loader task.

The DAG is written for Apache Airflow 2. It uses `DatabricksRunNowOperator` of the provider `apache-airflow-providers-databricks` and the Airflow connection `databricks_default`.

### Prerequisites

- [LoadControl Options](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en) contains `Airflow`, and [Configure LoadControl](https://knowledge.bigenius-x.com/configure-loadcontrol-project?hsLang=en) is `true`.
- The `databricks_job` placeholders are replaced, see [Databricks - Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en), and the artifacts are [deployed](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en) without errors.
- An Airflow 2 environment with the Databricks provider, for example the one of [Apache Airflow - Load control environment](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en).
- A Databricks personal access token of the user the jobs run as.

### Create the Databricks jobs

The folder `LoadControl` of the artifacts holds one job file `<loader>_Loader_Job.json` per loader. The DAG starts these jobs by name.

1. Download [deploy\_jobs.ps1](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/deploy_jobs.ps1) into the artifacts folder, next to `LoadControl`:  
   ![Artifacts folder with deploy\_jobs.ps1 next to the Helpers, Jupyter and LoadControl folders](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-26-36-2931-PM.png?width=239&height=223&name=image-png-Aug-08-2024-01-26-36-2931-PM.png)
2. Open PowerShell 7 in the artifacts folder and run:
   
   ```
   .\deploy_jobs.ps1 -databricksAccountName <workspace host> `
     -jobsFolderName LoadControl -databricksToken <token>
   ```
   
    Here `<workspace host>` is the host name of the workspace URL without `https://`, and `<token>` the personal access token.
3. Check the output. For each job file, the script prints `creating job <job name>`. A job with the same name is deleted first: `job already present, removing <job name>`.

### Run the DAG

1. Copy `LoadControl\<project>_dag.py` into the folder `dags` of Airflow. `<project>` is the project name in lower case without spaces, with underscores for dots; the DAG ID is the file name without `_dag.py`.
2. In the Airflow UI, open Admin, choose Connections and select + (Add a new record).  
   ![Airflow Admin menu with Connections highlighted](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-36-12-8250-AM.png?width=507&height=168&name=image-png-Jun-06-2024-06-36-12-8250-AM.png)  
   ![Airflow List Connection page with the add button highlighted](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-37-07-9860-AM.png?width=503&height=139&name=image-png-Jun-06-2024-06-37-07-9860-AM.png)
3. Enter the connection: 
     - Connection Id: `databricks_default`
     - Connection Type: Databricks
     - Host: the workspace URL, `https://<workspace host>`
     - Login: `token`
     - Password: the personal access token
   ![Airflow Add Connection form for connection type Databricks with empty fields](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-13-41-9567-PM.png?width=640&height=522&name=image-png-Aug-08-2024-01-13-41-9567-PM.png)
4. Select Test; it shows `Connection successfully tested` when Airflow reaches the workspace. Then select Save.  
   ![Test button on the connection form](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-52-25-8266-AM.png?width=158&height=49&name=image-png-Jun-06-2024-06-52-25-8266-AM.png)  
   ![Connection successfully tested message](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-52-00-3336-AM.png?width=640&height=54&name=image-png-Jun-06-2024-06-52-00-3336-AM.png)  
   ![Save button on the connection form](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-52-54-6833-AM.png?width=158&height=49&name=image-png-Jun-06-2024-06-52-54-6833-AM.png)
5. Open DAGs and switch on the toggle left of the DAG.  
   ![Airflow menu bar with DAGs highlighted](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-54-19-7985-AM.png?width=541&height=42&name=image-png-Jun-06-2024-06-54-19-7985-AM.png)
6. Select Trigger DAG in the column Actions.  
   ![Trigger DAG button in the Actions column](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-55-31-9128-AM.png?width=129&height=87&name=image-png-Jun-06-2024-06-55-31-9128-AM.png)

All tasks of a DAG run pass the same load timestamp to their loaders: the logical date of the run. The DAG has no schedule; it runs when you trigger it. After each loader task, a task `log_<loader>` prints the result of the Databricks job run.

### Check the result

Select the DAG, then the run, and open the tab Graph. The run succeeded when every task shows the state `success`:

![Graph of a DAG run with loader tasks of type DatabricksRunNowOperator, their log tasks and the tasks begin and end](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-21-18-8508-PM.png?width=640&height=279&name=image-png-Aug-08-2024-01-21-18-8508-PM.png)

The log of each `log_<loader>` task prints the link to the Databricks job run. In Databricks, the result views show the rows, for example `SELECT COUNT(*), MAX(bg_loadtimestamp) FROM <catalog>.rawvault.rdv_hub_<hub>_result;`.

### Example

The DAG `test_vaultandmart_spark_databricks` of a DataVault and Mart project has one task per loader between the tasks `begin` and `end`:

```
run_rdv_hub_bdc_customer_loader = DatabricksRunNowOperator(
    task_id='RDV_HUB_BDC_CUSTOMER_Loader',
    databricks_conn_id='databricks_default',
    job_name='RDV_HUB_BDC_CUSTOMER_Loader_Job',
    notebook_params={'bg_loadtimestamp':
      "{{ logical_date.strftime('%Y-%m-%d %H:%M:%S.%f') }}"}
)
```

Its job `RDV_HUB_BDC_CUSTOMER_Loader_Job` runs the notebook `<notebook_task_path>RDV_HUB_BDC_CUSTOMER_Loader` on the cluster of `existing_cluster_id`.

### Failure symptoms and recovery

#### A task fails because the job does not exist

Cause: `deploy_jobs.ps1` did not run, or ran in another workspace than the connection names.  
Fix: Run it as in [Create the Databricks jobs](https://knowledge.bigenius-x.com/databricks-load-data-apache-airflow-load-control#create-the-databricks-jobs), then clear the failed task and its downstream tasks in the Graph.

#### The job does not find the loader notebook

Cause: `notebook_task_path` does not point to the folder `Jupyter` of the uploaded artifacts, or lacks the trailing slash.  
Fix: Correct `notebook_task_path` in `replacement_config.json`, unzip the artifacts again, replace the placeholders and rerun `deploy_jobs.ps1`.

#### The DAG shows an import error

Cause: The Airflow environment has no Databricks provider, or it runs Airflow 3, which no longer accepts the DAG parameters `schedule_interval` and `concurrency`.  
Fix: Use an Airflow 2 environment with `apache-airflow-providers-databricks`, as described in [Apache Airflow - Load control environment](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en).

For failures in other steps, see [Troubleshoot generation, deployment and load failures](https://knowledge.bigenius-x.com/troubleshoot-generation-deployment-and-load-failures?hsLang=en).

### Related articles

- Previous: [Databricks - Deployment](https://knowledge.bigenius-x.com/databricks-deployment?hsLang=en), [Apache Airflow - Load control environment](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en)
- Reference: [LoadControl Options (Project)](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en), [Configure LoadControl (Project)](https://knowledge.bigenius-x.com/configure-loadcontrol-project?hsLang=en), [Databricks - Placeholder values](https://knowledge.bigenius-x.com/databricks-placeholder-values?hsLang=en)
- See also: [Understand Load Control](https://knowledge.bigenius-x.com/understand-load-control?hsLang=en), [Databricks - Load data - Native Load Control](https://knowledge.bigenius-x.com/databricks-load-data-native-load-control?hsLang=en), [Databricks - Artifacts](https://knowledge.bigenius-x.com/databricks-artifacts?hsLang=en)

- [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content)

    - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines)
    - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working)
    - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account)
- [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en)
- [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en)
- [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content)

    - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration)
    - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions)
    - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features)
    - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects)
    - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches)
    - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections)
    - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview)
    - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps)
    - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms)
    - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping)
    - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships)
    - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality)
    - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes)
    - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling)
    - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts)
    - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings)
    - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace)
- [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content)

    - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse)
    - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse)
    - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks)
    - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake)
    - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server)
    - [Oracle](https://knowledge.bigenius-x.com/generators?hsLang=en#oracle)
    - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts)
    - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders)
    - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment)
    - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment)
    - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline)
    - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment)
    - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment)
    - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control)
    - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow)
    - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type)
    - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties)
    - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms)
- [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content)

    - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations)
- [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content)

    - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches)
    - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases)
    - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules)
    - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules)
- [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en)
- [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content)

    - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application)
    - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application)
    - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators)
- [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content)

    - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs)
    - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits)

- [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en)
- [Customer Portal](https://knowledge.bigenius-x.com/customer-portal)
- [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login)

[![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com)

<https://www.linkedin.com/company/bigenius> <https://www.youtube.com/@biGENIUS_DWA>

All rights reserved © 2025, biGENIUS AG