---
title: Apache Airflow - Load control environment
description: This article sets up a local Apache Airflow 2.9.1 environment in Docker that runs the DAG biGENIUS-X generates when LoadControl Options contains Airflow.
---

[Skip to content](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment#main-content)

[Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en)

[![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/)

Open main navigation

Close main navigation

- [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new)
- [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal)

 How can we help you?

- There are no suggestions because the search field is empty.

1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en)
2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en)
3. [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment)

# Apache Airflow - Load control environment

**Applies to**  
Platform: Microsoft SQL Server · Snowflake · Databricks · Oracle

This article sets up a local Apache Airflow 2.9.1 environment in Docker that runs the DAG biGENIUS-X generates when [LoadControl Options](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en) contains `Airflow`. Use it to try the Airflow load control before you integrate the DAG into your own Airflow installation. Afterwards Airflow runs one task per loader in the order of the dependencies.

The generated DAGs are written for Apache Airflow 2. biGENIUS support does not cover the installation and configuration of this environment beyond the example in this article. Airflow can be installed and configured in many other ways. Fabric Lakehouse also offers `Airflow`; it runs the DAG in an Apache Airflow job of the Fabric workspace, see [Fabric Lakehouse - Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/fabric-lakehouse-load-data-apache-airflow-load-control?hsLang=en).

### Prerequisites

- [Docker Desktop](https://docs.docker.com/get-docker/) with at least 4 GB of memory and [PowerShell 7](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows).
- Artifacts generated with `Airflow` in LoadControl Options, placeholders replaced, and the solution deployed. The DAG is the file `LoadControl\<project>_dag.py` of the replaced artifacts, for example `datavault_mssql_dag.py`.
- Network access from Docker to the target platform and an account Airflow can use, for example a SQL Server login with SQL Server authentication.

### Set up and run the environment

#### Prepare the folder

1. Create a folder, for example `airflow`, with the subfolders `dags`, `logs`, `plugins` and `config`.  
   ![Folder airflow with the subfolders config, dags, logs and plugins](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-12-47-15-5559-PM.png?width=181&height=253&name=image-png-Jun-05-2024-12-47-15-5559-PM.png)
2. Download [docker-compose.yaml](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/airflow-2.9.1/docker-compose.yaml), [.env](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/airflow-2.9.1/.env), [requirements.txt](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/airflow-2.9.1/requirements.txt) and [Dockerfile](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/airflow-2.9.1/Dockerfile) into the folder. If the browser saves `.env` without its leading dot, rename it to `.env`.
3. In `requirements.txt`, remove the `#` before the provider line of your platform (see Platform differences).
4. Copy the DAG file into `dags`.  
   ![Folder airflow, subfolder dags, with a file whose name ends with \_dag.py](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-02-30-15-6489-PM.png?width=213&height=131&name=image-png-Jun-05-2024-02-30-15-6489-PM.png)

#### Start Airflow

1. Open PowerShell in the folder and create the Docker network the compose file uses: `docker network create training-network`. Skip this step when the network exists.
2. Run `docker compose up airflow-init`. Docker builds the image with your requirements, creates the metadata database and the user `airflow`. The output ends with `User "airflow" created with role "Admin"` and `2.9.1`.
3. Run `docker compose up -d`. After a few minutes, `docker compose exec airflow-scheduler airflow dags list` shows the DAG as paused. The UI is at [http://localhost:8080](http://localhost:8080), user `airflow`, password `airflow`.

#### Create the connection and run the DAG

1. Create the connection the DAG uses. For Microsoft SQL Server on the same machine:
   
   ```
   docker compose exec airflow-scheduler airflow connections add `
     dwh_azure_sql_conn --conn-type mssql `
     --conn-host host.docker.internal --conn-port 1433 `
     --conn-login <login> --conn-password <password> `
     --conn-schema <database>
   ```
   
    The command prints `Successfully added `conn_id`=dwh_azure_sql_conn`. `<database>` is the target database, `<login>` and `<password>` are the SQL Server login.
2. Unpause the DAG: `docker compose exec airflow-scheduler airflow dags unpause <dag id>`. The DAG ID is the file name without `_dag.py`.
3. Trigger it: `docker compose exec airflow-scheduler airflow dags trigger <dag id>`. The run starts in the state `queued`.

### Check the result

`docker compose exec airflow-scheduler airflow dags list-runs -d <dag id>` shows the run with the state `success` when all tasks succeeded. Each task log shows the row counts the loader returned, for example `RowCountInserted: 33`.

### Example

The DAG `datavault_mssql` of the Data Vault Test Project has 103 loader tasks between the tasks `begin` and `end`. Each task calls one loader through the connection `dwh_azure_sql_conn`:

```
op_kwargs = { 'StoProc':
  '[<database>].[<stage schema>].[STG_ST_CUSTOMER_Loader]',
  'conn_id': 'dwh_azure_sql_conn'}
```

With a local SQL Server, the run took about three minutes and ended with `success`.

### Failure symptoms and recovery

#### The DAG does not appear or shows an import error

Cause: the provider or module the DAG imports is missing, for example `oracledb` or the Databricks provider. For Microsoft SQL Server, the message `ModuleNotFoundError: No module named 'airflow.providers.microsoft.mssql.operators'` means that the provider is installed in version 4 or later, which no longer contains the `MsSqlOperator` the DAG imports.  
Fix: in `requirements.txt`, remove the `#` before the line of your platform and keep its version limit, then rebuild the environment as described in the next entry.

#### Tasks fail with a connection error

Cause: the connection ID does not exist, or Docker cannot reach the host.  
Fix: create the connection with the ID the DAG uses (see Platform differences) and check host, port and login.

#### A changed requirements file has no effect

Cause: a change in `requirements.txt` or `docker-compose.yaml` needs a new image and database.  
Fix: run `docker compose down --volumes --rmi all`, then start again with `docker compose up airflow-init`. In Docker Desktop, the same means deleting the airflow containers, the five airflow images and the two volumes.

Deleting the volumes removes the Airflow metadata database, including connections, variables and run history. Create the connections again after the reset.

### Platform differences

- Microsoft SQL Server: requirements line `apache-airflow-providers-microsoft-mssql<4`; connection `dwh_azure_sql_conn` of type `mssql`.
- Snowflake: the provider is part of the Airflow image; the DAG uses the connection `snowflake_default`.
- Databricks: requirements line `apache-airflow-providers-databricks<8`; the DAG uses the connection `databricks_default`. You can define it in `docker-compose.yaml`: remove the `#` before the line `AIRFLOW_CONN_DATABRICKS_DEFAULT: 'databricks://:@<host>?token=<token>'` and replace the placeholders, where `<host>` is the workspace URL without `https://` and `<token>` a [personal access token](https://docs.databricks.com/en/dev-tools/auth/pat.html).
- Fabric Lakehouse: the DAG imports `msal` and `requests`. The Airflow 2.9.1 image of this environment already contains both, so `requirements.txt` has no line for Fabric Lakehouse. The Airflow connection and the upload of the DAG are described in [Fabric Lakehouse - Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/fabric-lakehouse-load-data-apache-airflow-load-control?hsLang=en).
- Oracle: in `requirements.txt`, remove the `#` before `oracledb<27`. The DAG connects with one Airflow connection per layer; each connection ID is the schema name of the layer from the replacement configuration. Set host, port, login and password, and the service name as schema.

### Related articles

- Next: 
    - Microsoft SQL Server: [Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/microsoft-sql-server-load-data-apache-airflow-load-control?hsLang=en)
    - Snowflake: [Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/snowflake-load-data-apache-airflow-load-control?hsLang=en)
    - Databricks: [Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/databricks-load-data-apache-airflow-load-control?hsLang=en)
    - Oracle: [Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/oracle-load-data-apache-airflow-load-control?hsLang=en)
- Reference: [LoadControl Options (Project)](https://knowledge.bigenius-x.com/loadcontrol-options-project?hsLang=en)
- See also: [Understand Load Control](https://knowledge.bigenius-x.com/understand-load-control?hsLang=en), [Fabric Lakehouse - Load data - Apache Airflow Load Control](https://knowledge.bigenius-x.com/fabric-lakehouse-load-data-apache-airflow-load-control?hsLang=en)

- [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content)

    - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines)
    - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working)
    - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account)
- [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en)
- [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en)
- [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content)

    - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration)
    - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions)
    - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features)
    - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects)
    - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches)
    - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections)
    - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview)
    - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps)
    - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms)
    - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping)
    - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships)
    - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality)
    - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes)
    - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling)
    - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts)
    - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings)
    - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace)
- [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content)

    - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse)
    - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse)
    - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks)
    - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake)
    - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server)
    - [Oracle](https://knowledge.bigenius-x.com/generators?hsLang=en#oracle)
    - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts)
    - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders)
    - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment)
    - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment)
    - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline)
    - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment)
    - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment)
    - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control)
    - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow)
    - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type)
    - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties)
    - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms)
- [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content)

    - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations)
- [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content)

    - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches)
    - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases)
    - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules)
    - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules)
- [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en)
- [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content)

    - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application)
    - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application)
    - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators)
- [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content)

    - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs)
    - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits)

- [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en)
- [Customer Portal](https://knowledge.bigenius-x.com/customer-portal)
- [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login)

[![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com)

<https://www.linkedin.com/company/bigenius> <https://www.youtube.com/@biGENIUS_DWA>

All rights reserved © 2025, biGENIUS AG