---
title: Apache Airflow - Load control environment
description: This article outlines a potential setup for an environment for Apache Airflow load control. Please note that the installation and configuration of this target environment fall outside the scope of biG
---

[Skip to content](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment#main-content)

[Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en) [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal?hsLang=en)

[![TRI\_Logo\_biGENIUS\_2020\_RGB\_s](https://knowledge.bigenius-x.com/hs-fs/hubfs/Marketing/TRI_Logo_biGENIUS_2020_RGB_s.png?width=150&height=42&name=TRI_Logo_biGENIUS_2020_RGB_s.png)](https://www.bigenius-x.com/)

Open main navigation

Close main navigation

- [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new)
- [Go to Customer Portal](https://knowledge.bigenius-x.com/customer-portal)

 How can we help you?

- There are no suggestions because the search field is empty.

1. [biGENIUS-X Knowledge Base](https://knowledge.bigenius-x.com/?hsLang=en)
2. [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en)
3. [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment)

# Apache Airflow - Load control environment

This article outlines **a potential setup for an environment for Apache Airflow load control.**

Please note that the installation and configuration of this target environment fall **outside the scope of biGENIUS support**.

**We will not offer assistance** beyond the provided example in this article.

An Apache Airflow load control environment has numerous alternative configurations and installations.

Below is a potential environment setup for an Apache Airflow load control.

### Tools

Please install the following tools:

- Docker Desktop (download available [here](https://docs.docker.com/get-docker/))
- Powershell 7 or higher (download available [here](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.3))

### Prerequisites

Before creating the Apache Airflow environment, please execute the following steps:

- Create the following repository structure where you want to use Airflow:![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-12-47-15-5559-PM.png?width=133&height=186&name=image-png-Jun-05-2024-12-47-15-5559-PM.png)

The main repository called *airflow* in the previous picture can be adapted to your choice. The sub-repositories must have the same names as in the picture.

- Download and copy the following files inside the main repository (*airflow* in this example): 
    - [docker-compose.yaml](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/docker-compose.yaml): It is a YAML file that defines and configures all the containers to create for our Airflow environment  
          - **For Microsoft SQL Server and Databricks, please uncomment** between lines  64 to 67: the line for the connection to the target solution environment corresponding to the target technology you want to use
    - [.env](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/.env): It defines an environment variable mandatory for the environment 
          - ***After download, the . is lost. Please rename the file .env***
    - [requirements.txt](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/requirements.txt): It defines the dependencies to add depending on the target technology 
          - **Please uncomment** the lines corresponding to the target technology you want to use
    - [Dockerfile](https://f926da42ec3845beb4acf72f.blob.core.windows.net/hubspotknowledgebase/Dockerfile): It allows the use of the requirements.txt during the container's creation
- Copy the generated Python artifacts (*XXX\_dag.py*) for Airflow load control in the dags sub-repository:  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-02-30-15-6489-PM.png?width=151&height=93&name=image-png-Jun-05-2024-02-30-15-6489-PM.png)

#### Microsoft SQL Server on Docker Target solution

If you are using **Microsoft SQL Server on Docker** as a Target solution, please execute the following command in PowerShell to find out the IP address to set up in Airflow:

```
(docker inspect training-sqlserver-2022 | ConvertFrom-Json)[0].NetworkSettings.Networks.'training-network'.IPAddress
```

 If **Microsoft SQL Server** is **installed locally** on your laptop **or in a distant server**, you can find the IP address to use in the connection by executing the following command on PowerShell:

```
ipconfig | findstr IPv4
```

Update accordingly the IP address in line 65 in the docker-compose.yaml, for example:

```
AIRFLOW__CORE__SQL_ALCHEMY_CONN: sql+pythosmssql://sa:Admin2022!@192.168.112.8:1433/airflow
```

A known bug exists in the **1.9 and 1.10 Microsoft SQL Server generator dags**. Double brackets are generated in the DAG code.

Workaround:

- Open the .dag file with an editor (notepad is enough)
- Find and replace all the \[\[ by \[
- Find and replace all the \]\] by \]
- Example: 
    - 'StoProc': '**\[\[**MS\_SQL\_DVDM\].\[RDV\].\[RDV\_HUB\_CreditCard\_Hub\_Loader**\]\]**'
    - will be replaced by
    - 'StoProc': '**\[**MS\_SQL\_DVDM\].\[RDV\].\[RDV\_HUB\_CreditCard\_Hub\_Loader**\]**'
- Use this dag in Apache Airflow

#### Databricks Target solution

If you are using **Databricks** as a Target solution, please:

- Create a Personal Access Token (PAT): [https://docs.databricks.com/en/dev-tools/auth/pat.html](https://docs.databricks.com/en/dev-tools/auth/pat.html)
- Find your host URL: 
    - Databricks on Azure: this is the link used to open Databricks from Azure without the *https://*![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Aug-08-2024-12-55-50-3750-PM.png?width=670&height=202&name=image-png-Aug-08-2024-12-55-50-3750-PM.png)
    - Databricks on AWS: this is the link you are using to connect to Databricks  
      ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Mar-18-2025-09-56-20-3187-AM.png?width=670&height=225&name=image-png-Mar-18-2025-09-56-20-3187-AM.png)

- Update the connection in the *docker-compose.yaml* file:

```
--Replace host-url by your databricks host--Replace yourtoken by the PAT createdAIRFLOW_CONN_DATABRICKS_DEFAULT: 'databricks://:@host-url?token=yourtoken'
```

#### Snowflake Target solution

There is no special configuration.

### Environment setup

To set up the Airflow environment, please execute the following steps:

- Open a Powershell in the root location of your new folder structure  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-12-47-15-5559-PM.png?width=133&height=186&name=image-png-Jun-05-2024-12-47-15-5559-PM.png)
- Create a network with the following command in Powershell:

```
docker network create training-network
```

If you want to use Microsoft SQL Server on Docker as a Target solution, you must create and use the same Docker network.

If you already created it, skip this step.

- Initialize the Postgres database used by the Airflow application, run database migrations, and create the first user account with the following command in Powershell:

```
docker compose up airflow-init
```

- Let’s wait for the operations to complete, and you should see the following feedback after some minutes:  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-02-25-29-2275-PM.png?width=460&height=45&name=image-png-Jun-05-2024-02-25-29-2275-PM.png)
- Your Postgres database is created, and credentials are as follows: 
    - username: airflow
    - password: airflow
- Start your local Airflow instance with the following command in Powershell:

```
docker compose up
```

- When you can see the following feedback, the Airflow instance is ready:  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-02-26-24-1298-PM.png?width=670&height=90&name=image-png-Jun-05-2024-02-26-24-1298-PM.png)
- Navigate to the Airflow homepage [http://localhost:8080](http://localhost:8080)  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-02-55-31-1399-PM.png?width=670&height=224&name=image-png-Jun-05-2024-02-55-31-1399-PM.png)
- Enter the following credentials: 
    - username: airflow
    - password: airflow
- You should have the following result:

![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-05-2024-02-56-15-1446-PM.png?width=670&height=210&name=image-png-Jun-05-2024-02-56-15-1446-PM.png)

 

You can now [load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow).

 

If **something goes wrong** during the environment setup (you forgot to uncomment a vital line, to copy the dag at the correct location...), you should **reinitialize it completely** to be sure your changes will be taken care.

For this, open your Docker Desktop application and:

- Delete the **airflow container**:  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Dec-16-2025-04-45-06-9431-PM.png?width=670&height=303&name=image-png-Dec-16-2025-04-45-06-9431-PM.png)
- Delete the **5 airflow images**:  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Dec-16-2025-04-45-51-0710-PM.png?width=670&height=242&name=image-png-Dec-16-2025-04-45-51-0710-PM.png)
- Delete the **2 airflow volumes**:  
  ![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Dec-16-2025-04-46-38-8622-PM.png?width=670&height=135&name=image-png-Dec-16-2025-04-46-38-8622-PM.png)

Then you can [restart the environment setup](https://knowledge.bigenius-x.com/apache-airflow-load-control-environment#environment-setup).

 

- [Getting started](https://knowledge.bigenius-x.com/getting-started?hsLang=en#main-content)

    - [Guidelines](https://knowledge.bigenius-x.com/getting-started?hsLang=en#guidelines)
    - [Start working](https://knowledge.bigenius-x.com/getting-started?hsLang=en#start-working)
    - [Account](https://knowledge.bigenius-x.com/getting-started?hsLang=en#account)
- [General overview](https://knowledge.bigenius-x.com/general-overview?hsLang=en)
- [Artificial Intelligence](https://knowledge.bigenius-x.com/artificial-intelligence?hsLang=en)
- [Application Modules](https://knowledge.bigenius-x.com/application-modules?hsLang=en#main-content)

    - [Administration](https://knowledge.bigenius-x.com/application-modules?hsLang=en#administration)
    - [Solutions](https://knowledge.bigenius-x.com/application-modules?hsLang=en#solutions)
    - [Global Features](https://knowledge.bigenius-x.com/application-modules?hsLang=en#global-features)
    - [Projects](https://knowledge.bigenius-x.com/application-modules?hsLang=en#projects)
    - [Branches](https://knowledge.bigenius-x.com/application-modules?hsLang=en#branches)
    - [Data Connections](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-connections)
    - [Dataflow Modeling - Overview](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-overview)
    - [Dataflow Modeling - Wizard Steps](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-wizard-steps)
    - [Dataflow Modeling - Terms](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-terms)
    - [Dataflow Modeling - Term Mapping](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-term-mapping)
    - [Dataflow Modeling - Relationships](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-relationships)
    - [Dataflow Modeling - Data Quality](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-data-quality)
    - [Dataflow Modeling - Indexes](https://knowledge.bigenius-x.com/application-modules?hsLang=en#dataflow-modeling-indexes)
    - [Relationship Modeling](https://knowledge.bigenius-x.com/application-modules?hsLang=en#relationship-modeling)
    - [Generate Artifacts](https://knowledge.bigenius-x.com/application-modules?hsLang=en#generate-artifacts)
    - [Project Settings](https://knowledge.bigenius-x.com/application-modules?hsLang=en#project-settings)
    - [Data Marketplace](https://knowledge.bigenius-x.com/application-modules?hsLang=en#data-marketplace)
- [Generators](https://knowledge.bigenius-x.com/generators?hsLang=en#main-content)

    - [Fabric Warehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-warehouse)
    - [Fabric Lakehouse](https://knowledge.bigenius-x.com/generators?hsLang=en#fabric-lakehouse)
    - [Databricks](https://knowledge.bigenius-x.com/generators?hsLang=en#databricks)
    - [Snowflake](https://knowledge.bigenius-x.com/generators?hsLang=en#snowflake)
    - [Microsoft SQL Server](https://knowledge.bigenius-x.com/generators?hsLang=en#microsoft-sql-server)
    - [Artifacts](https://knowledge.bigenius-x.com/generators?hsLang=en#artifacts)
    - [Replace Placeholders](https://knowledge.bigenius-x.com/generators?hsLang=en#replace-placeholders)
    - [Target solution environment](https://knowledge.bigenius-x.com/generators?hsLang=en#target-solution-environment)
    - [Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment)
    - [Deployment with an Azuze DevOps pipeline](https://knowledge.bigenius-x.com/generators?hsLang=en#deployment-with-an-azuze-devops-pipeline)
    - [Delta Deployment](https://knowledge.bigenius-x.com/generators?hsLang=en#delta-deployment)
    - [Load control environment](https://knowledge.bigenius-x.com/generators?hsLang=en#load-control-environment)
    - [Load data with a native load control](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-a-native-load-control)
    - [Load data with Apache Airflow](https://knowledge.bigenius-x.com/generators?hsLang=en#load-data-with-apache-airflow)
    - [Model Object Type](https://knowledge.bigenius-x.com/generators?hsLang=en#model-object-type)
    - [Properties](https://knowledge.bigenius-x.com/generators?hsLang=en#properties)
    - [Default Terms](https://knowledge.bigenius-x.com/generators?hsLang=en#default-terms)
- [Discovery application](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#main-content)

    - [Discovery configurations](https://knowledge.bigenius-x.com/discovery-application?hsLang=en#discovery-configurations)
- [Best practices](https://knowledge.bigenius-x.com/best-practices?hsLang=en#main-content)

    - [Modeling Approaches](https://knowledge.bigenius-x.com/best-practices?hsLang=en#modeling-approaches)
    - [Use Cases](https://knowledge.bigenius-x.com/best-practices?hsLang=en#use-cases)
    - [Business Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#business-rules)
    - [Data Quality Rules](https://knowledge.bigenius-x.com/best-practices?hsLang=en#data-quality-rules)
- [FAQs](https://knowledge.bigenius-x.com/faqs?hsLang=en)
- [Product Release Notes](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#main-content)

    - [SaaS Application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#saas-application)
    - [Discovery application](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#discovery-application)
    - [Generators](https://knowledge.bigenius-x.com/product-release-notes?hsLang=en#generators)
- [Legal Documents](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#main-content)

    - [Current legal docs](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#current-legal-docs)
    - [Software Product and Limits](https://knowledge.bigenius-x.com/legal-documents?hsLang=en#software-product-and-limits)

- [Contact us](https://knowledge.bigenius-x.com/kb-tickets/new?hsLang=en)
- [Customer Portal](https://knowledge.bigenius-x.com/customer-portal)
- [Sign in](https://knowledge.bigenius-x.com/_hcms/mem/login)

[![biGENIUS logo](https://knowledge.bigenius-x.com/hubfs/TRI_Logo_biGENIUS_2020_RGB-1.svg "biGENIUS logo")](https://www.bigenius-x.com)

<https://www.linkedin.com/company/bigenius> <https://www.youtube.com/@biGENIUS_DWA>

All rights reserved © 2025, biGENIUS AG