--- title: Fabric Lakehouse - Load data - Apache Airflow Load Control description: Load a deployed Fabric Lakehouse solution with Apache Airflow by running the DAG that the generator writes. --- [Skip to content](https://knowledge.bigenius-x.com/fabric-lakehouse-load-data-apache-airflow-load-control#main-content) [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en) [![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/) Open main navigation Close main navigation - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new) - [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal) How can we help you? - There are no suggestions because the search field is empty. 1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en) 2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en) 3. [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) # Fabric Lakehouse - Load data - Apache Airflow Load Control **Applies to** Platform: Fabric Lakehouse Load a deployed Fabric Lakehouse solution with Apache Airflow by running the DAG that the generator writes. The DAG has one task per loader and runs the tasks in dependency order, one at a time. Afterwards every loader has run once. The DAG is available since Generator 2.2. With Target Platform `Fabric`, each task starts a loader notebook through the Fabric REST API. With Target Platform `Azure Synapse`, each task runs a Synapse pipeline, see [Azure Synapse](https://knowledge.bigenius-x.com/fabric-lakehouse-load-data-apache-airflow-load-control#azure-synapse). ### Prerequisites - [LoadControl Options](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en) contains `Airflow`, and [Configure LoadControl](https://knowledge.bigenius-x.com/configure-loadcontrol-project?hsLang=en) is `true`. The artifacts then contain `LoadControl\_dag.py`. - The artifacts are [deployed](https://knowledge.bigenius-x.com/fabric-lakehouse-deployment-2.0?hsLang=en) without errors. - An Apache Airflow job in the Fabric workspace of the deployment. Its requirements contain the Python packages `msal` and `requests`, which the DAG imports. - The client ID, a client secret and the tenant ID of the service principal that is a Member of the workspace, see [Fabric Lakehouse - Target environment](https://knowledge.bigenius-x.com/fabric-lakehouse-target-environment?hsLang=en). ### Run the DAG 1. In `replacement_config.json`, check the variable `airflow#airflow#fabric_conn_id`. It names the Airflow connection of the DAG and is `fabric-integration` by default. Then [replace the placeholders](https://knowledge.bigenius-x.com/replace-the-placeholders-in-the-artifacts?hsLang=en). 2. In the Apache Airflow job, create a connection with this connection ID. Enter the client ID as login, the client secret as password, and `{"tenantId": ""}` as extra. 3. Run `.\deploy.ps1` as described in [Fabric Lakehouse - Deployment](https://knowledge.bigenius-x.com/fabric-lakehouse-deployment-2.0?hsLang=en). After the notebooks, the script uploads the DAG file to the first Apache Airflow job of the workspace. It fills in the workspace ID and the ID of each loader notebook. It does this also with `-SkipDeployment`. 4. In Airflow, unpause the DAG and trigger it. The DAG ID is ``, the project name in lower case with underscores, for example `test_vaultandmart_spark_fabric`. The DAG has no schedule and is paused when it is created. Each task gets an access token for the service principal with `msal` and starts its loader notebook as a notebook job. It then reads the job status every 30 seconds until the job completes or fails. Every task passes the start of the data interval of the DAG run as load timestamp, so all loaders of one run share it. ### Check the result The DAG run succeeded when all its tasks have the state `success`. The log of each loader task ends with `Completed successfully!`. The loader notebooks write their row counts to the execution log, as in [Fabric Lakehouse - Load data - Native Load Control](https://knowledge.bigenius-x.com/fabric-lakehouse-load-data-native-load-control-2.0?hsLang=en#check-the-result). ### Example For the project `test.vaultandmart.spark.fabric`, the DAG `test_vaultandmart_spark_fabric` holds this task for the Stage loader. `deploy.ps1` replaces `` with the ID of the notebook `STG_ST_BDC_CUSTOMER_Loader`: ``` run_stg_st_bdc_customer_loader = PythonOperator( task_id="STG_ST_BDC_CUSTOMER_Loader", python_callable=run_fabric_job_and_wait, op_kwargs={ 'item_id': '', 'load_timestamp': "{{ data_interval_start.strftime( '%Y-%m-%d %H:%M:%S.%f') }}" } ) ``` The line breaks inside the template are added here for the page width. ### Azure Synapse With Target Platform `Azure Synapse`, the DAG runs one Synapse pipeline per loader with the operator `AzureDataFactoryRunPipelineOperator`. The artifacts contain the pipeline definitions as `LoadControl\_Pipeline.json`; the deployment notebooks do not create them. The DAG uses the Airflow connection ID `AzureDataFactory`, which is written into the DAG. It retries a failed task three times and passes the logical date of the DAG run as load timestamp. It uses arguments of Apache Airflow 2, such as `schedule_interval` and `concurrency`. 1. In `replacement_config.json`, enter the Spark pool as `datafactory_pipeline#datafactory_pipeline#sparkpool_name` and the linked service as `datafactory_pipeline#datafactory_pipeline#linked_service_reference_name`. Then replace the placeholders. 2. Create the pipelines in the Synapse workspace from the files `_Pipeline.json`. 3. Create the Airflow connection `AzureDataFactory` with the provider `apache-airflow-providers-microsoft-azure`. 4. Copy `LoadControl\_dag.py` into the DAG folder of Airflow, then unpause and trigger the DAG. The other pipeline variables are preset: `timeout` `0.12:00:00`, `retry` `0`, `retry_interval_in_seconds` `30`, `secure_input` and `secure_output` `false`. ### Failure symptoms and recovery #### The deployment script finds no Apache Airflow job Message: `No Airflow job found in workspace ` Cause: The workspace has no Apache Airflow job. The script stops after the deployment. Fix: Create an Apache Airflow job in the workspace and run `.\deploy.ps1 -SkipDeployment`. #### A task fails before the notebook starts Message: `Token error: ` or `Start failed: ` Cause: The connection holds a wrong client ID, secret or tenant ID, or the service principal has no access to the workspace. Fix: Correct the connection or the workspace access, then clear the task and its downstream tasks in Airflow. #### A loader notebook fails Message: `Fabric job failed with status: Failed` Cause: The loader notebook failed; its downstream tasks do not run. Fix: Read the `ERROR` rows of the execution log, correct the cause, and clear the task and its downstream tasks. ### Related articles - Previous: [Fabric Lakehouse - Deployment](https://knowledge.bigenius-x.com/fabric-lakehouse-deployment-2.0?hsLang=en), [Azure Synapse - Deployment](https://knowledge.bigenius-x.com/azure-synapse-deployment-2.0?hsLang=en) - Reference: [LoadControl Options (Project)](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en), [Configure LoadControl (Project)](https://knowledge.bigenius-x.com/configure-loadcontrol-project?hsLang=en), [Fabric Lakehouse - Placeholder values](https://knowledge.bigenius-x.com/fabric-lakehouse-placeholder-values-2.0?hsLang=en) - See also: [Understand Load Control](https://knowledge.bigenius-x.com/understand-load-control?hsLang=en), [Fabric Lakehouse - Load data - Native Load Control](https://knowledge.bigenius-x.com/fabric-lakehouse-load-data-native-load-control-2.0?hsLang=en), [Fabric Lakehouse - Artifacts](https://knowledge.bigenius-x.com/fabric-lakehouse-artifacts-2.0?hsLang=en), [Apache Airflow - Load control environment](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment?hsLang=en) - [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content) - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines) - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working) - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account) - [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en) - [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en) - [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content) - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration) - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions) - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features) - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects) - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches) - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections) - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview) - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps) - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms) - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping) - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships) - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality) - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes) - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling) - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts) - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings) - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace) - [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content) - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse) - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse) - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks) - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake) - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server) - [Oracle](https://knowledge.bigenius-x.com/generators?hsLang=en#oracle) - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts) - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders) - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment) - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment) - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline) - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment) - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment) - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control) - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow) - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type) - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties) - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms) - [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content) - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations) - [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content) - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches) - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases) - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules) - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules) - [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en) - [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content) - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application) - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application) - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators) - [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content) - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs) - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits) - [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) - [Customer Portal](https://knowledge.bigenius-x.com/customer-portal) - [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login) [![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com) All rights reserved © 2025, biGENIUS AG