--- title: Databricks - Load data - Apache Airflow Load Control - 1.8 description: Databricks - Load data - Apache Airflow Load Control - 1.8 --- [Skip to content](https://knowledge.bigenius-x.com/databricks-load-data-apache-airflow-load-control-1.8#main-content) [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en) [![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/) Open main navigation Close main navigation - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new) - [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal) How can we help you? - There are no suggestions because the search field is empty. 1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en) 2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en) 3. [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) # Databricks - Load data - Apache Airflow Load Control - 1.8 Before loading the data from your Source(s) System(s) to your Databricks Target System, please:  - [Deploy the generated Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment) - Configure the [Load control environment for Apache Airflow](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en) You are now ready to load the data with Apache Airflow. ### Create the Databricks Jobs Airflow will use Databricks Jobs to load the data. biGENIUS-X artifacts contain configuration files to create these Jobs. Download the following helper: [deploy\_jobs.ps1](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/deploy_jobs.ps1). Copy it near the LoadControl folder of your generated artifacts: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-26-36-2931-PM.png?width=239&height=223&name=image-png-Aug-08-2024-01-26-36-2931-PM.png) Then: - Open **Powershell** - Navigate to your generated artifacts folder: ``` --Examplecd C:\XXXXXXXXX\20240219144515 ``` - Execute the Powershell script helper with the following parameters: - Check [Apache Airflow - Load Control environment](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en#databricks-target-solution) to find the host and the token ``` -- Replace by the Databricks host-- Replace by your PAT.\deploy_jobs.ps1 -databricksAccountName -jobsFolderName LoadControl -databricksToken ``` You should have the following result for each Job creation: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-33-39-0888-PM.png?width=688&height=58&name=image-png-Aug-08-2024-01-33-39-0888-PM.png) ### Load the data with Apache Airflow To load the data with Apache Airflow: - Navigate to the Airflow homepage [http://localhost:8080](http://localhost:8080) ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-02-55-31-1399-PM.png?width=688&height=230&name=image-png-Jun-05-2024-02-55-31-1399-PM.png) - Enter your credentials - You should have the following result: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-02-56-15-1446-PM.png?width=688&height=216&name=image-png-Jun-05-2024-02-56-15-1446-PM.png) - Configure a connection to your target solution environment by following the steps: - Open the menu Admin \> Connections ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-36-12-8250-AM.png?width=507&height=168&name=image-png-Jun-06-2024-06-36-12-8250-AM.png) - Click on the plus icon ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-37-07-9860-AM.png?width=503&height=139&name=image-png-Jun-06-2024-06-37-07-9860-AM.png) - Enter the connection details: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-13-41-9567-PM.png?width=688&height=561&name=image-png-Aug-08-2024-01-13-41-9567-PM.png) - **Connection id**: Must be *databricks\_default* - **Connection Type**: select Databricks - **Host**: check [Apache Airflow - Load Control environment](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en#databricks-target-solution) to find it - - - **Login**: Must be *token* (as we are using a PAT) - **Password**: Must be the PAT value (check [Apache Airflow - Load Control environment](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en#databricks-target-solution) to find it) - Check the connection by clicking on the Test button: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-52-25-8266-AM.png?width=158&height=49&name=image-png-Jun-06-2024-06-52-25-8266-AM.png) - You should have the following message: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-52-00-3336-AM.png?width=688&height=58&name=image-png-Jun-06-2024-06-52-00-3336-AM.png) - Save the connection by clicking on the Save button: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-52-54-6833-AM.png?width=158&height=49&name=image-png-Jun-06-2024-06-52-54-6833-AM.png) - Go back to the DAGs list by clicking on the DAGs menu: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-54-19-7985-AM.png?width=541&height=42&name=image-png-Jun-06-2024-06-54-19-7985-AM.png) - To launch a data load, click on the Trigger DAG button for the relevant DAG: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-06-2024-06-55-31-9128-AM.png?width=129&height=87&name=image-png-Jun-06-2024-06-55-31-9128-AM.png) - The data load started: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-20-45-2150-PM.png?width=688&height=121&name=image-png-Aug-08-2024-01-20-45-2150-PM.png) - You can follow the data load by clicking on the DAG, then on the current execution (green bar), and finally on the Graph tab: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-21-18-8508-PM.png?width=688&height=299&name=image-png-Aug-08-2024-01-21-18-8508-PM.png) - If you want to check the logs of the execution, click on the Audit Log tab: ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-01-22-08-3096-PM.png?width=688&height=309&name=image-png-Aug-08-2024-01-22-08-3096-PM.png) - You can now check that your data were correctly loaded by creating a Notebook in Databricks and executing with the following code: ``` -- Adapt the database and table names-- Add as many union as target tablesdf = spark.sql("""    select '`rawvault`.`rdv_sat_creditcard_delta_satellite_result`' as table_name    , count(*) as row_count    , max(bg_loadtimestamp) as max_bg_loadtimestamp    from `rawvault`.`rdv_sat_creditcard_delta_satellite_result`    union    select '`rawvault`.`rdv_hub_salesorderheader_hub_result`' as table_name    , count(*) as row_count    , max(bg_loadtimestamp) as max_bg_loadtimestamp    from `rawvault`.`rdv_hub_salesorderheader_hub_result`""")df.show(n=1000, truncate=False) ```   - [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content) - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines) - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working) - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account) - [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en) - [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en) - [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content) - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration) - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions) - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features) - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects) - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches) - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections) - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview) - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps) - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms) - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping) - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships) - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality) - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes) - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling) - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts) - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings) - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace) - [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content) - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse) - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse) - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake) - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server) - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts) - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders) - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment) - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment) - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline) - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment) - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment) - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control) - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type) - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties) - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms) - [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content) - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations) - [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content) - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches) - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases) - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules) - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules) - [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en) - [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content) - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application) - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application) - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators) - [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content) - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs) - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits) - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) - [Customer Portal](https://knowledge.bigenius-x.com/customer-portal) - [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login) [![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com) All rights reserved © 2025, biGENIUS AG