Apache Airflow: Building and Managing Data Pipelines Training Course

Apache Airflow is an open source platform for authoring, scheduling, and monitoring workflows. With Airflow 2.x, organizations can orchestrate data pipelines at scale, integrate with cloud services, and ensure robust observability and security in production environments.

This instructor-led, live training (online or onsite) is aimed at beginner-level to intermediate-level data and DevOps professionals who wish to use Apache Airflow to build, automate, and scale end-to-end data pipelines.

By the end of this training, participants will be able to:

Install and configure Apache Airflow in different environments.
Author, schedule, and monitor DAGs using the TaskFlow API.
Integrate Airflow with data sources, cloud services, and ML workflows.
Manage ETL pipelines and ensure data reliability.
Secure and scale Airflow using executors, queues, and Kubernetes.
Apply observability and best practices for production deployments.

Format of the Course

Interactive lecture and discussion.
Lots of exercises and practice.
Hands-on implementation in a live-lab environment.

Course Customization Options

To request a customized training for this course, please contact us to arrange.

This course is available as onsite live training in Spain or online live training.

Thank you for sending your enquiry! One of our team members will contact you shortly.

Thank you for sending your booking! One of our team members will contact you shortly.

Course Outline

Introduction to Apache Airflow

What is workflow orchestration
Key features and benefits of Apache Airflow
Airflow 2.x improvements and ecosystem overview

Architecture and Core Concepts

Scheduler, web server, and worker processes
DAGs, tasks, and operators
Executors and backends (Local, Celery, Kubernetes)

Installation and Setup

Installing Airflow in local and cloud environments
Configuring Airflow with different executors
Setting up metadata databases and connections

Navigating the Airflow UI and CLI

Exploring the Airflow web interface
Monitoring DAG runs, tasks, and logs
Using the Airflow CLI for administration

Authoring and Managing DAGs

Creating DAGs with the TaskFlow API
Using operators, sensors, and hooks
Managing dependencies and scheduling intervals

Integrating Airflow with Data and Cloud Services

Connecting to databases, APIs, and message queues
Running ETL pipelines with Airflow
Cloud integrations: AWS, GCP, Azure operators

Monitoring and Observability

Task logs and real-time monitoring
Metrics with Prometheus and Grafana
Alerting and notifications with email or Slack

Securing Apache Airflow

Role-based access control (RBAC)
Authentication with LDAP, OAuth, and SSO
Secrets management with Vault and cloud secret stores

Scaling Apache Airflow

Parallelism, concurrency, and task queues
Using CeleryExecutor and KubernetesExecutor
Deploying Airflow on Kubernetes with Helm

Best Practices for Production

Version control and CI/CD for DAGs
Testing and debugging DAGs
Maintaining reliability and performance at scale

Troubleshooting and Optimization

Debugging failed DAGs and tasks
Optimizing DAG performance
Common pitfalls and how to avoid them

Summary and Next Steps

Requirements

Experience with Python programming
Familiarity with data engineering or DevOps concepts
Understanding of ETL or workflow orchestration

Audience

Data scientists
Data engineers
DevOps and infrastructure engineers
Software developers

21 Hours

Custom Corporate Training

Training solutions designed exclusively for businesses.

Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
Flexible Schedule: Dates and times adapted to your team's agenda.
Format: Online (live), In-company (at your offices), or Hybrid.

Investment

Price per private group, online live training, starting from 4800 € + VAT*

(*The final price may vary depending on the technical specialization of the course, the level of customization, the method of delivery and the number of learners)

Need help picking the right course?

Testimonials (7)

The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.

Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH