Checkout.com Dumps Self-Managed Airflow for Cloud Composer 3

Data engineering teams know the drill: build a slick data platform, then spend all your time maintaining the orchestrator. Checkout.com’s Data Platform team was stuck in that swamp, wrestling a self-hosted Apache Airflow environment on another hyperscaler. Server management, patching, incident response—it was pulling focus from building pipelines. So they migrated to Managed Service for Apache Airflow (Gen 3) on Google Cloud. The result? A more scalable, cost-efficient, and robust data foundation. Let’s dive into the murky waters and see how they did it.

The Starting Point: Self-Managed Airflow

Before the migration, Checkout.com ran Airflow on self-managed infrastructure. It worked, but maintaining the underlying resources was a constant drain. Patching, upgrades, server management—regular interruptions that kept the team from focusing on innovation. Operational data from the past year highlighted the pain points:

• Reducing operational friction: Stability challenges, especially during high-load periods.
• Complex dependency management: Upgrading packages and ensuring compatibility was a manual struggle. With Managed Airflow (Gen 3), dependencies are handled at the image level, ensuring seamless compatibility out-of-the-box.
• DAG sync time: Syncing DAGs to the scheduler took about six minutes after deployment to S3, slowing iteration speed.
• Manual processes: Scaling required manual intervention, and onboarding new teams meant manually creating secrets and variables for dbt.

The Solution: Managed Service for Apache Airflow (Gen 3)

Checkout.com migrated to Managed Airflow to offload infrastructure responsibility and tap into Google Cloud’s managed scalability. The results were immediate and measurable across three areas: reliability, cost, and developer velocity.

Dynamic Scaling in Action

In the previous setup, the team allocated the maximum number of workers for peak loads, paying for peak capacity around the clock. Managed Airflow provides built-in dynamic scaling, automatically adjusting workers based on workload demands. When tasks spike, it scales up; when they drop, it scales down. This shift from fixed provisioning to dynamic scaling reduced monthly costs by an estimated 30%.

Reliability and DAG Isolation

Increased stability was a primary driver. In the past, a single problematic DAG could affect the entire environment. Managed Airflow introduced critical architecture improvements:

• DAG isolation: Each DAG runs in its own execution environment. If one fails or consumes excessive resources, it doesn’t affect others.
• Managed operations: Google Cloud handles patching and upgrades during scheduled windows, removing manual upgrade management.
• Improved visibility: Integration with Google Cloud Monitoring and Cloud Logging provides clear visibility into task execution. Engineers can debug issues independently without escalating to the platform team.

Faster Developer Workflows

The migration also improved day-to-day workflows for Checkout.com’s data engineers:

• Faster deployments: Using Cloud Storage for DAGs enabled near-instant syncing.
• Simpler onboarding: Teams no longer need platform support to create variables before onboarding.
• Modernizing dbt execution: By leveraging containerized dbt runs, Managed Airflow (Gen 3) eliminates dependency bottlenecks. Teams can run any required dbt model with minimal setup and no manual infrastructure overhead.
• Environment updates: No need to redeploy the entire Airflow environment to add new roles or update Python packages.
• AI-powered troubleshooting with Gemini Cloud Assist: In a self-managed environment, a failed task triggered a frantic hunt through fragmented logs. Now, Checkout.com can initiate a Gemini investigation directly from the Airflow DAG UI. Gemini generates a scorecard evaluating different hypotheses with supporting and contradictory evidence, drastically reducing mean time to recovery.

Conclusion

For Checkout.com, the move to Managed Airflow (Gen 3) marked a strategic shift, freeing engineers to focus on delivering value. As Keisi Mancellari, Data Platform Engineer at Checkout.com, put it: “With Managed Service for Apache Airflow, we’ve achieved significant improvements in efficiency, scalability, and reliability. Managed infrastructure, automated scaling, faster deployments, and isolated execution environments have transformed how we operate.” With a stable, scalable, and cost-efficient platform in place, Checkout.com is now ready to focus on the future of its data pipelines, confident that its orchestration layer is ready for whatever comes next.

Learn more about Managed Service for Apache Airflow and how it can support your data platform.

Original announcement published on Google Cloud.