Apache Airflow: Building and Managing Data Pipelines Training Course

Apache Airflow serves as an open-source framework for designing, scheduling, and monitoring workflow processes. With the arrival of Airflow 2.x, enterprises are empowered to orchestrate large-scale data pipelines, seamlessly integrate with cloud ecosystems, and guarantee robust observability and security within production settings.

Delivered as an instructor-led live session (available online or in-person), this course is tailored for data and DevOps professionals ranging from beginner to intermediate levels who aim to leverage Apache Airflow to construct, automate, and expand end-to-end data pipelines.

Upon completion of this training, participants will be equipped to:

Deploy and configure Apache Airflow across various environments.
Design, schedule, and oversee DAGs utilizing the TaskFlow API.
Connect Airflow with data sources, cloud platforms, and ML workflows.
Oversee ETL pipelines to guarantee data integrity and reliability.
Enhance Airflow security and scalability through executors, queues, and Kubernetes.
Implement observability strategies and industry best practices for production rollouts.

Course Structure

Engaging lectures accompanied by interactive discussions.
A substantial amount of practical exercises and drills.
Real-world implementation within a live-lab setting.

Tailoring Options

For bespoke training needs related to this curriculum, please reach out to us to coordinate arrangements.

This course is available as onsite live training in Spain or online live training.

Thank you for sending your enquiry! One of our team members will contact you shortly.

Thank you for sending your booking! One of our team members will contact you shortly.

Duration 21 hours

Course Outline

Introduction to Apache Airflow

Defining workflow orchestration
Core features and advantages of Apache Airflow
Enhancements in Airflow 2.x and an overview of the ecosystem

Architecture and Fundamental Concepts

Understanding scheduler, web server, and worker components
DAGs, individual tasks, and operators
Executors and backend systems (Local, Celery, Kubernetes)

Deployment and Configuration

Installing Airflow in both local and cloud-based setups
Configuring Airflow with varied executors
Establishing metadata databases and external connections

Interacting with the Airflow UI and Command Line

Exploring the Airflow web-based interface
Tracking DAG executions, tasks, and log entries
Utilizing the Airflow CLI for administrative tasks

Creating and Administering DAGs

Building DAGs via the TaskFlow API
Leveraging operators, sensors, and hooks
Managing task dependencies and scheduling cycles

Connecting Airflow to Data and Cloud Ecosystems

Linking to databases, APIs, and message queues
Executing ETL pipelines through Airflow
Cloud connectivity: AWS, GCP, and Azure operators

Monitoring and Observability Strategies

Reviewing task logs and real-time status
Capturing metrics using Prometheus and Grafana
Setting up alerts and notifications via email or Slack

Security Measures for Apache Airflow

Implementing Role-based access control (RBAC)
Authentication protocols using LDAP, OAuth, and SSO
Managing secrets through Vault and cloud-based secret stores

Scaling Apache Airflow

Handling parallelism, concurrency, and task queuing
Utilizing CeleryExecutor and KubernetesExecutor
Deploying Airflow on Kubernetes via Helm

Production Best Practices

Managing version control and CI/CD for DAGs
Testing and debugging DAG workflows
Sustaining reliability and performance at high scale

Troubleshooting and Performance Tuning

Resolving issues in failed DAGs and tasks
Enhancing DAG execution speed and efficiency
Identifying common pitfalls and strategies to prevent them

Recap and Future Directions

Requirements

Proficiency in Python programming
Familiarity with data engineering or DevOps principles
A solid grasp of ETL processes or workflow orchestration

Target Audience

Data scientists
Data engineers
DevOps and infrastructure engineers
Software developers

Custom Corporate Training

Training solutions designed exclusively for businesses.

Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
Flexible Schedule: Dates and times adapted to your team's agenda.
Format: Online (live), In-company (at your offices), or Hybrid.

Investment

Price per private group, online live training, starting from 4350 € + VAT*

(*The final price may vary depending on the technical specialization of the course, the level of customization, the method of delivery and the number of learners)

Need help picking the right course?
info@nobleprog.es or +34 911 43 65 67

Testimonials (7)

The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.

Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH