11–20 of 95 posts
+1 for Airflow. I use it every day to handle tasks with many components and dependencies. I love that everything is code & version-controlled. I do wish it had a REST API though.
+1 for Airflow. I use it every day to handle tasks with many components and dependencies. I love that everything is code & version-controlled. I do wish it had a REST API though.
Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.
Airflow is just the workflow management layer on top of your data pipeline. The flexibility to generate custom graphs based on user-specific parameters should be handled within a pipeline task. Based on your example, I would have a single dag that would 1. get user data and 2. generate a graph. All the flexibility should be defined in whatever function, script or program you define to generate the graph.
Earlier quoted context omitted.
Airflow is just the workflow management layer on top of your data pipeline. The flexibility to generate custom graphs based on user-specific parameters should be handled within a pipeline task. Based on your example, I would have a single dag that would 1. get user data and 2. generate a graph. All the flexibility should be defined in whatever function, script or program you define to generate the graph.
I think they are referring to "event-driven" DAGs which would be both shaped and triggered dynamically. You can accomplish this now but it feels a bit hacky and is pretty clear that it goes against the Airflow paradigm of static, slowly-changing workflows
Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.
What our tool does is allow users to organize the flow of their processing jobs on an infinite 2D layout, have some jobs run at the beginning of the flow while they organize another part to run later.
Unfortunately it's a big pile of messy code that depends too much on other "internal" systems so we can't open source it... I'd like to add "yet" because I try to gradually clean it up, simplify and make it more generic, but I'm not sure I'll see that day myself.
In the meantime... maybe Node-RED ? https://nodered.org/
I'm used to running R scripts with cron to handle some batch jobs and I'm no python dev. Would it be easy to start using Airflow?
Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale. Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow. Edit: Now I see in the interview there is this: About Luigi, it is simpler in scope than Airflow, and…
A lot of name dropping and unprovable statements. I'm really interested by the domain and progress, but Airflow needs to be more generous in real information and less in marketing bs. Can someone share a more introductory article about what makes Airflow different from the current state of the art?
* Scheduler that knows how to handle retries, skipped tasks, failing tasks
* Great UI
* Horizontal scaleable
* Great community
* Extensible; we could make it work in an enterprise context (kerberos, ldap etc)
* No XML
* Testable and debug-able workflows