I mentioned a few days back that I was working on a personal project to help myself skill up in, among other things, Apache Airflow. That project is called Spotcheck.

To summarise the README’s Background section: I’m on a few Discord communities, and we like to have streaming parties. It’s been a challenge lately navigating which countries can stream which films and series, so enter the concept: What if I could make a site where we could see which countries can see which series / films, and have it update regularly? I found a passable API that purports to contain that information, and now I had a project with which to learn Apache Airflow properly!

I completed the initial structure late last night after a particularly annoying Uber shift involving standing in a Whataburger for 35 minutes waiting for their oven to reboot. And yes, that is exactly the phrasing the kitchen staff used when they came out to apologise. What a brave new future we now live in! But I digress. I pressed the Trigger button, and watched my code go out and fetch availability information for a small selection of initial test media: 101 Dalmatian Street, Fireman Sam, My Little Pony: Friendship is Magic, Pound Puppies, and SpongeBob.

And then the next component, the Transformation DAG, began to run. And it only ran twice, despite there being five test media to run. I had results for 101 Dalmatian Street and Fireman Sam, but there were no other results. I did some deep dive into Airflow and their issue tracker, and found that this is actually a known issue with asset-based scheduling. Sometimes multiple assets will be grouped into a single run! This isn’t well documented.

It’s actually a really nasty bug when you have asset-based workloads like this. I’m hoping to be able to add a workaround to my personal project, and we have an idea for a way forward at work. But honestly, the best and/or worst part of this was actually delivering this news to work.

Because nothing says “I’m a helpless nerd dog” more than sharing your screen on Zoom, demonstrating the bug live, and then saying “as you can see, we have results for SpongeBob, but Airflow has eaten My Little Pony” in a professional setting.