Apache Airflow is an open source project for planning and managing workflows, written in Python.
Kaxil Naik, Director of Airflow Engineering at Astronomer and one of the key drivers behind Airflow, told SD Times: “It’s used to automate your day-to-day jobs or daily tasks, and the tasks can be as simple as running a Python script or as complicated as taking in all the data from 500 different data warehouses and manipulating them .”
It was created by Airbnb in 2014 and will soon celebrate its 10th anniversary this year. It joined the Apache Software Foundation in March 2016 at the Incubation level and became a top-level project in 2019.
Airflow was initially designed only for ETL use cases, but has evolved over the years to add features that make it useful for all aspects related to data engineering.
“It has continued to be a leader in this space, as we have maintained a good balance between innovation and stability. Because of these almost 10 years of Airflow in the same space, we have added so many features that allow Airflow to be very reliable and stable,” he said.
Latest edition, 2.9was released earlier this week and added new features like the ability to combine datasets and timelines, custom names for dynamic task mapping, and the ability to group task records.
The project can be found on GitHub here.