Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow and the Future of Data Engineering: A Q&A

medium.com

Post reply on HN
Print view View original

91–95 of 95 posts

Re: Airflow and the Future of Data Engineering: A Q&A

#91

Post by somewhatoff »

caravel wrote: post #88
somewhatoff wrote: post #85

How would you see Airflow in relation to Apache Beam / GC Dataflow?

[author] Airflow is not a data flow engine, though you can use it to do some of that, but we typically defer on doing data transformations using/coordinating external engines (Spark, Hive, Cascading, Sqoop, PIG, ...). We operate at a higher level: orchestration. If we were to start using Apache Beam at Airbnb (and we very well may soon!), we' use Airflow to schedule and trigger batch beam jobs alongside the rest of o…

Thanks, that's really interesting. The usage of 'pipeline' to describe both sorts of system made me think there was a lot of overlap, but I'm understanding now how they are complementary.

Re: Airflow and the Future of Data Engineering: A Q&A

#92

Post by detroitcoder »

jaz46 wrote: post #68

One of the important paradigms that I think Luigi and Airflow miss is that they treat pipelines as a DAG of tasks, when it really should be thought of as a DAG of Data. It's a subtle difference, but has huge impacts when you're trying to dynamically scale tasks based on cluster resources and track data lineage throughout your system.(Disclosure: I'm the founder of Pachyderm[0], a containerized data pipeline framework…

Apache Nifi. I always make the comparison between the two when introducing Airflow as it often times takes awhile to under stand the nuanced (but critical) difference

Re: Airflow and the Future of Data Engineering: A Q&A

#93

Post by pas »

gregn610 wrote: post #83
frugalmail wrote: post #73

Earlier quoted context omitted.

Maybe we have drastically different use cases, but dynamic and weak typing are a disaster in the data space. Not sure why people build production systems in such languages. If it matters anywhere it matters in the data space. We don't see the disconnect between decimal and int, but when you're expecting a character and you get varchar, (not sure about the apostrophe case, but I suspect your talking about quotes and e…

If you enjoy making sure your peas don't touch the carrots, then sure, strong typing is great in the data space. But when you get woken up because a spreadsheet comes through as 11.0 rather than 11 or you have to type Double.parseDouble(x) == Double.parseDouble(y) /* instead of pythons */ x == y 27 times to get a feed file parsed, then I'd say that the tools are lacking basic useability. And especially primitive in l…

That means you don't have a clean "domain model" (or business logic layer) and a data filtering layer that creates the domain objects. You should apply business logic on the pure objects/entities, and you should make sure that the filtering handles the representational problems (parsing, data integrity, partial data, and so on).

And ideally if something makes it through the data filtering layer into the logic layer and does not make sense there, then that should be handled. And that's where strong types help. It forces you to handle these cases, even if that means logging/alerting/ignoring, but at least you'll have to make a decision when you write the logic, instead of 3AM in the morning.

Re: Airflow and the Future of Data Engineering: A Q&A

#94

Post by dataops »

batbomb wrote: post #69
dataops wrote: post #32

Data engineering is converging under the umbrella of DataOps. For those interested, there's a DataOps Summit in Boston this June https://www.dataopssummit.com/

This is just Data Management, a term which predates "DataOps" by more than a decade in both research and enterprise. I don't really think it needs a rebranding.

That's not really the case... There's a nice, short podcast that was just posted that defines DataOps more broadly - spend 20 minutes listening to it, and see what you think. Interested to hear your thoughts.

https://thenewstack.io/delving-dataops-matters/

Re: Airflow and the Future of Data Engineering: A Q&A

#95

Post by ktamura »

caravel wrote: post #87
ktamura wrote: post #25

Earlier quoted context omitted.

If simplicity and non-Python-centricity matter, I encourage folks to look into Digdag [1][2]. It's Ansible for Workflow Management. While both Luigi and Airflow (somewhat rightfully) assume the user to know/have affinity for Python, Digdag focuses on ease of use and helping enterprises move data around many systems. If we learned one thing from today's S3 outage, it's not enough to use multiple cloud infrastructure p…

[auhtor] Oh cool, I didn't know you guys released your solution yet, I remember demoing Airflow to you guys early on. Looks like it turned out great, congrats on the release!

Thanks! Airflow was/has been a great source of inspiration =)
Post reply on HN