Note: Uh oh, looks like Spots got her on tech soapbox again! Feel free to skip this one if you aren’t programming-inclined.
I’ve been working on writing a lot of code against Apache Airflow at the Library. Multi-hundred-line DAGs with dozens of task groups. This is Serious Business. And while I’m mostly enjoying the way Airflow fits together, there are a few things that have been bugging me about it. Time to vent.
Task Group code is executed like “immediate” code
The code inside Task Group functions, at least in the Taskflow API, cannot access parameters. It also cannot use return values from tasks beyond sending them to other tasks. I didn’t find these limitations in top-level DAG code, or at least I don’t recall running into this problem. Why are Task Groups second-class citizens? It’s infuriating and causing me to have to write task groups differently than would feel ergonomic.
Tests feel like an afterthought
This is probably the Ruby (and before that, Objective-C) developer in me talking, but I very much ascribe to the theory of TDD (test-driven development). A proper test suite doesn’t just verify conformance to the specification, but can ensure error conditions and “odd” situations are handled correctly. I’ve had many experiences where writing tests revealed missing error handling, or in some cases even revealed more needed clarity in requirements.
I find the Airflow method of “just execute the DAG with .test()” to be really clunky. It doesn’t really make sense for most of our tasks to be their own Operators that can be imported outside of their respective DAGs, so it’s not easy/possible to write per-task test DAGs either.
So much of the documentation talks about old versions
I really appreciate the overhaul that the Airflow team did with version 3.0. A lot of things are easier, a lot of things have improved, and a lot of old cruft has been swept away. That is great and the sign of a mature project that continues to evolve.
If only the documentation kept up. I see so many references to Airflow 2.x, and sometimes even earlier versions, in the very latest documentation. It is jarring. Sometimes I even wonder if what I am reading is applicable any more, despite having “Version: latest (3.2.1)” on the side. Investing in the time and resources to overhaul Airflow 3.0 was great, but it feels like they should have given at least the same time and care to the documentation.
In closing: Airflow is still good!
I don’t want these criticisms to come across as me not liking Airflow. It is a great project. It is a solid orchestration system and overall the API is satisfying to write against. I am definitely happy that our team is using it. But even good systems have things they can work on. These criticisms are in no way meant to denigrate the hard work of the people improving Airflow every day, and my deepest gratitude is with them as they continue to do just that!