The crew who set out together from Taiwan, left to right: KuanHao, me, Peter, GuanMing Wei Lee on the far right flew in from Japan 😆
The Lakehouse sessions this year spent a lot of time on the role agents play in data infrastructure, and how agents change the way we build it.
As always, scaling came up too.
This year also featured the very first Airflow Contributor Day.
Many Airflow PMC members, committers, contributors, and users got together to discuss the future of Airflow.
Airflow Contributor Day: Community over Code, for Real#
Unfortunately, one of Airflow’s PMC members, Jens, ran into some health issues a while ago,
so they couldn’t join Airflow Summit or CommunityOverCode this year.
The good news is that Jens is doing much better now!
Jarek, one of our senior members, brought a cardboard cutout of Jens to the venue, and it arrived even before most of us had left for the conference.
About a month before CoC, Jarek invited everyone to fly to Germany together after the conference to visit Jens.
Nobody else could adjust their schedule, so we sent along some small gifts and cards
for Jarek to hand over to Jens, and to bring Jens’s latest news back to the rest of us!
This is what “Community over Code” really looks like.
We first came together for the sake of the project and its technology,
but over the long run, we’ve become friends.
On a lighter note: at last year’s Airflow Summit, I went cycling in Seattle with Jens and the group.
It turned out to be all uphill, and I couldn’t keep up with everyone.
I hope Jens recovers soon,
and that we get another chance to ride together (but let’s skip the uphill roads next time!).
Our angle was “language”: Airflow is, in a sense, a language for describing workflows.
For more than ten years, though, that language could only be written in Python. This year, you can write it in Go, TypeScript, and Java as well.
And if we can use other programming languages, can we write it in natural language too?
For my part, I ran a live demo of a retail company where three engineering teams each use a different language.
In a single Airflow environment, they build one cross-team workflow, while each team keeps its own ownership and tech stack.
I then walked through how we run multi-language workflows while keeping Airflow’s core control plane intact,
rather than reimplementing the core in every language.
Wei Lee then covered interacting with Airflow in natural language and the Common AI Provider, which lets Airflow power AI-native workflows.
This session was only confirmed about a month before the conference. Someone in the Airflow track had cancelled at the last minute,
and the architecture for the non-Python SDKs had taken concrete shape by then.
Jarek helped us get onto the Airflow track’s schedule.
Thanks again, Jarek, for giving us the chance to share this at CoC EU!
What I Saw at Lakehouse Day: The Lakehouse in the Age of Agents#
The round table’s central question was what the Lakehouse stack has to change once agents become users of the data. The speakers raised a few observations:
The impact of agent workloads: it used to be only humans reading and writing tables. Now agents interact with tables freely, so the security bar is higher and access control needs to be more granular.
The return of on-prem: on-prem is also viewed as a security measure. The Lakehouse stack can’t consider only the cloud. It has to work on-premises too.
New file formats: at the file-format and in-memory-format level, options beyond Parquet are emerging for different use cases (e.g., Lance). The community will need to converge on a standard to avoid fragmenting the ecosystem.
Influence Through Open Source: Tying a Project to Your Business#
This was a Lakehouse Day talk by Sean and his Apple colleagues on Apple’s Lakehouse adoption:
A series of migrations and architectural trade-offs on the way from HDFS to a Lakehouse
They rely mainly on Apache DataFusion Comet. When an organization has enough influence in a project, it can shape the project’s features and release cadence, aligning the project’s roadmap with internal goals.
Comet’s operator coverage needs to be as high as possible to avoid falling back to JVM-based execution or doing redundant data movement.
Lakehouse Stack Unification and Semantic Catalogs#
Table formats and compute engines have diverged, and integrating them is a shared challenge.
When an agent accesses tables, it needs a unified standard: is my “revenue” the same thing as your “revenue”? Just as UTC gives everyone a common time reference, everyone needs to follow the same definition.
Apache Ossie targets exactly this problem by proposing an open standard for business semantics. Data infra vendors and catalog services are investing in it, and Apache Polaris has already integrated it.
Centralizing metadata lookups in a REST catalog such as Apache Polaris can significantly improve the Lakehouse’s overall throughput.
Without one, the engine has to resolve metadata itself, which is very expensive.
OpenAI’s data infra team shared how they optimized Airflow 2 to handle the scaling demands of a new line of business.
Their cluster was already near its limit, and within six weeks they had to launch a much larger workload that it clearly couldn’t handle.
They named their rewritten core the Rhino API, mainly to solve these two Airflow 2 problems:
The scheduler isn’t horizontally scalable.
Every task runner pod connects directly to the DB; Rhino API removes that direct access.
Interestingly, to solve the scheduler bottleneck, their new architecture uses a single scheduler rather than replicas, and pre-fetches the task instances to run asynchronously without blocking.
Attendees did raise concerns about future scalability, but they said that after the new workload launched, their current capacity is enough for what’s coming.
From the Airflow maintainers’ perspective, though, Airflow 3 already addresses many of the problems they were solving.
If a user hits similar problems on Airflow 2 and has time to migrate,
we’d recommend upgrading straight to Airflow 3 rather than making these changes on Airflow 2,
so that the upgrade doesn’t get harder later.
Zero Downtime Kubernetes Cluster Migration for Airflow at LinkedIn#
LinkedIn has long operated one of the largest Airflow clusters in the world.
Stefan from LinkedIn’s data infra team shared how they do zero-downtime Kubernetes cluster migration on Airflow 2.
This was purely a Kubernetes cluster version upgrade, not an Airflow upgrade.
Their definition of zero downtime is strict: not even a single running task instance is interrupted, and nothing needs to be retried.
The core idea is to have the two Airflow clusters, one on the old Kubernetes cluster and one on the new, share the same metadata DB. When they switch over, the next task instance starts on the new cluster instead of the old one, and even in-progress DagRuns are unaffected.
Next, they plan to explore zero-downtime Airflow version upgrades, which will be a very interesting and challenging problem,
since it involves DB schema migrations and Airflow core code changes.
I came into this one late and missed the opening context, but it covered how OpenAI optimizes its Lakehouse internally.
They’ve built in-house optimizations for both Delta and Iceberg tables, mainly targeting the small-files problem.
Under tight business deadlines, they can’t wait for the community’s next patch or minor release,
so they fork and build their own in-house solutions. To some extent this echoes what I wrote after Apache CommunityOverCode Asia 2025.
In the long run, though, it means the organization can’t easily track upstream releases,
and may even accumulate technical debt.
Last but not least, GuanMing and KuanHao, who traveled with us, presented Apache Mahout’s QDP (Quantum Data Plane) feature and its roadmap.
It’s probably unprecedented to present something as esoteric as quantum ML at a CoC venue.
On the first day of the conference, there happened to be a Pokémon event nearby,
so the CoC organizers came up with an ASF edition of Pokémon called “Inframon”.
Maybe Apache Taipei can do something like this at our next event!
Thanks to Chia-Ping, who, despite being busy with work in Taiwan, treated the Apache Taipei folks to a big group dinner.
Thanks also to Sean for introducing us to many of the data infra folks at Apple, giving us the chance to talk with more Lakehouse experts.
This year I ran into several people I’d met at CoC NA last year. Many big tech data infra teams attend year after year.
You can see how their focus areas change, where they’re heading next, and how they solve their problems, which closes the information gap.
Showing up in person is how I learn which projects are worth watching and where they’re actively heading.
As for the Airflow project itself, I’m very glad to have been part of CoC EU,
especially with a full day of Airflow’s own track, and to have witnessed an important moment for the community.
I also talked with Peter, an Apache Ozone PMC member who came along, about how Ozone first runs the CI runner on the contributor’s fork before running the actual tests on the main repo, which helps ensure code quality and security. We also discussed how each repo should have its own workflow for automated AI review, similar to GitHub Copilot or CodeRabbit, so we control the harness configuration ourselves and avoid vendor lock-in.
Finally, special thanks to Chia-Ping. I’m still early in my career, and I didn’t get a travel grant for Apache CommunityOverCode EU this year.Without Chia-Ping’s help, I wouldn’t have been part of this Contributor Day, which meant so much to the Airflow community, and I wouldn’t have had the chance to present Airflow’s new-languages talk at CoC EU.
China’s localized data stack, streaming-first infrastructure, enterprise investment in open source, open governance, and lessons for Taiwan’s tech communities.
Exploring the AIP-82 External Event Driven Scheduling proposal in Airflow 3, the implementation details and use cases of the Common Message Queue, and how to enable real-time event-driven workflow scheduling in Airflow.
A deep dive into Airflow’s CI: Docker for reproducible environments, Breeze to unify local and CI workflows, and Selective Checks to run only the tests that matter.
