Viewing profile — dimberman

We built an RL environment for credit card reward optimization and trained Qwen 32B with GRPO against it. The trained model scores ~0.51 on held-out tasks vs. Opus 4 at ~0.41 and G…

Astronomer.io | (multiple roles) Fully remote-first compnay. We maintain and offer Apache Airflow as a service to customers ranging from early stage start ups to Fortune 500s. We'r…

Hi @theptip, I'm an Airflow PMC and would love to know a bit more about your comparison :). 1. Have you tried Airflow 2.0? We made some pretty big overhauls both in terms of UI and…

I'd say it's simpler than that. Airflow is a backend project built by backend engineers. Most UI people don't use Airflow or know what it is. @ryanhamilton is the first front-end d…

We're hiring multiple UI experts to rebuild the Airflow UI from scratch in react, so this is just the beginning :).

Hi didip! could you make an airflow issue related to that or start a thread in the dev list? That could be interesting! (though you might want to wait until after the holidays as w…

On the OSS side we have a helm chart that is heavily based on the one we use at Astronomer. That should hopefully get you started (or you can reach out to astro and someone will he…

I'm sure they will eventually, but Astronomer and (I believe) Cloud Composer have day 1 support.

Airflow 2.0 :)

WAIT. Sorry I misspoke. The only thing that is no longer allowed is using a bitshift operator between a DAG and a task. task_1 >> task_2 is totally fine my_dag >> task_1 Is no long…

I'm sure they will, I'm not sure on their timeline though.

So there are a few options for that if you're interested! 1. If you're using the KubernetesExecutor, you can point to custom images for individual tasks, this will primarily work i…

Unfortunately that is one feature that we had to take out, but you should check out the TaskFlow API, it's a very worthwhile trade-off! Edit: Sorry I misspoke here The only thing t…

Thank you for the feedback! I'm gonna pass that on to some AWS experts in the community. One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released …

A sensor would also work here. Especially with the new SmartSensor feature, sensors are basically free so you can set them up for event-based DAG executions.

We didn't separate task instances from timestamps YET purely because there was already so much to release that we didn't want to add more potential for bugs/upgrade difficulties. I…

I would say at this point Airflow is leaning pretty heavily on being a data tool. I wouldn't recommend it for something like CI/CD for example. Do you have a use-case in mind?

Thank you Holden! :D I seriously love that you're an Airflow user since your spark talks first got me into OSS.

Yeah we've been benchmarking for a while, it is VISIBLY noticeable. You're gonna love it :)

If you're switching to python operators you should check out the TaskFlow API. You can basically build python operators with just python functions and decorators.

Hi y'all! Airflow PMC here! Feel free to AMA about Airflow's new features/the roadmap going forward!

Hi y'all! Airflow PMC here! Feel free to AMA about Airflow's new features/the roadmap going forward!

I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at …