Viewing profile — caravel

Hey, creator here. I built this over the past month to try and stay sane while coordinating Claude Code, Codex, and Gemini across projects. Quick feature rundown: - Real-time colla…

So I built “Figma for AI coding” to stay sane while coordinating agents across projects, git worktrees, AI sessions and my team. Called it Agor the, "AGent ORchestrator". Built it …

A new open source project to support testing prompts at scale by the creator of Airflow & Superset. The toolkit brings many of the ideas from test-driven development to the prompt …

This is much more straightforward than your JVM-based, huge infra solutions like Flink. For when you need something that is familiar [python] and just works.

Correction: Prefect is built by one of the Airflow committers. At this time Airflow has ~50 committers. https://whimsy.apache.org/roster/ppmc/airflow

For context, I used Datastage, Informatica, Ab Initio and SSIS in previous lives an went on to write the first version of Airflow. I developed a taste for pipelines-as-code while w…

[full disclosure, Airflow committer here] I've never heard of "HP Operation Orchestration", but that looks like a drag and drop enterprise tool from a different Windows-GUI era. Ai…

[full disclosure, I'm the creator of Airflow] Many environments run tens of thousands of concurrent tasks, and hundreds of thousands of tasks daily. The list of companies using Air…

Here's the actual talk: https://www.youtube.com/watch?v=23_1WlxGGM4

[author] Airflow is not a data flow engine, though you can use it to do some of that, but we typically defer on doing data transformations using/coordinating external engines (Spar…

[auhtor] Oh cool, I didn't know you guys released your solution yet, I remember demoing Airflow to you guys early on. Looks like it turned out great, congrats on the release!

At a high level [for Airflow specifically] the scheduler or workflow engine cares most about the tasks and their dependencies, and is somewhat agnostic about the units of work it t…

[Airflow author] The task is centric to the workflow engine. It's an important entity, and it's complementary to the data lineage graph (not necessarily a DAG btw). At Airbnb we ha…

[Bias Alert: author of Airflow] can confirm that Airflow allows you to incorporate all of the seven steps, and more as an open platform. At Airbnb Airflow is far from being limited…

In a modern data team, Spark is just one of the type of job you may want to orchestrate. Typically as your company gets more tangled in data processing, you'll have many storage an…

[author] sorry you feel that way. I understand that the section on other similar tools is controversial, especially if you work on one of those. I'm repeating what I hear being ver…

[Airflow author here] in general, and when thinking in terms of best practices, we like to think of a DAG's shape as slowly changing, in a similar way that a database's tables defi…

I also hear that Uber forked Airflow early on [twice]. I'm curious to hear where they're at at this time.

[Airflow author here] one of the main differences between Airflow and Luigi is the fact that in Airflow you instantiate operators to create tasks, where with Luigi you derive class…

[author of Airflow here] as I wrote in another comment, I'd argue for a programmatic approach to workflows/dataflows as opposed to drag and drop. It turns out that code is a better…

I'd argue for something programmatic over drag and drop for reasons described in this other article I wrote: https://medium.freecodecamp.com/the-rise-of-the-data-enginee...

Many [most] of the companies using Airflow are small-ish scale, maybe less than a half a dozen people writing jobs that need to be scheduled. Airflow will bring clarity even to mod…