Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Guide to data tools landscape for developers

sinja.io

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Guide to data tools landscape for developers

#31

Post by vishnuharidas »

otter-in-a-suit wrote: post #25

This is a fantastic primer. A few suggestions: - Add a "last updated" note, since this space changes often (see the prefect/dagster situation that just happened) - Add a note about MCPs and other LLM-driven tools and features are becoming more and more important (e.g. hex.ai or the various MCPs shipped with some of the tools you mention, such as OpenMetadata) - Maybe organize the various tools you mention briefly by…

+ Also a few diagrams showing the end-to-end data flow and where all these tools fit in.

Re: Guide to data tools landscape for developers

#32

Post by gwerbin »

sdpy wrote: post #20
aleda145 wrote: post #18

As a data engineering person I can say that this is a great write up! Some thoughts: A "bubbling" topic right now is conversational analytics (i.e. talk to your data). There has been an explosion of tools in the last 6 months. YC is backing one too: https://getnao.io/ I feel like pandas is also somewhat frowned upon, the industry has moved on from that. Most SQL tools can now do everything that we could only do with…

Just to add, people in my network have been talking about polars (as an alternative to pandas) and other dataframe libraries. They're much easier to use now thanks to the Narwhals compatibility layer (for example, Narwhals was recently added as a dependency to scikit-learn).

What's different between Narwhals and Ibis? Why does the former exist when the latter has already (struggled along) existing for a while? Narrower scope / benefit of hindsight? No support for "cloud" data frames like Ray and Spark?

Also Pandas is very much still a great tool and it's only getting better. It has some fundamental limitations that are relevant for processing bigger datasets or running things with higher performance. But it's still my preferred data frame for interactive day-to-day work. I only switch to Polars (or DuckDB) when I want to maximize performance.

Re: Guide to data tools landscape for developers

#35

Post by sdpy »

gwerbin wrote: post #32
sdpy wrote: post #20

Earlier quoted context omitted.

Just to add, people in my network have been talking about polars (as an alternative to pandas) and other dataframe libraries. They're much easier to use now thanks to the Narwhals compatibility layer (for example, Narwhals was recently added as a dependency to scikit-learn).

What's different between Narwhals and Ibis? Why does the former exist when the latter has already (struggled along) existing for a while? Narrower scope / benefit of hindsight? No support for "cloud" data frames like Ray and Spark? Also Pandas is very much still a great tool and it's only getting better. It has some fundamental limitations that are relevant for processing bigger datasets or running things with higher…

There is a slide about Narwhals and Ibis here (a recording of a talk from PyData Berlin 2025): https://youtu.be/qOJiQtqI348?t=2496 TL;DR: It doesn't have to be Narwhals vs. Ibis because Narwhals supports Ibis.

Update: Here's a benchmark as well, but it may be outdated: https://www.kaggle.com/code/marcogorelli/narwhals-vs-ibis-ov...

Re: Guide to data tools landscape for developers

#36

Post by di1eep »

This is a great article covering the ecosystem but has not covered one of the key advancements in this area in the past 40 to 50 years., which is IVM (incremental view maintenance).

The topic could start from database “views” and how analytical queries are basically different views of the same underlying data and how IVM solves the batch+stream unification and real-time queries over data elegantly. Tying change data capture into the picture as well.

The company “Feldera” and its founding folks are the leading researchers/experts on IVM and it’s becoming increasingly relevant for all kinds of workloads.

Re: Guide to data tools landscape for developers

#37

Post by porksoda »

datadrivenangel wrote: post #19

"Popular metadata catalog solutions are Hive Metastore, AWS Glue Data Catalog, and Unity Catalog from Databricks. For query engine you can use, for example, Apache Spark, Trino, or Amazon Athena." DuckDB is eating the query engines and catalogs. Really could use more coverage on how DuckDB is changing the data tools landscape.

It is a POS. Try connect a read-only while another proc is connected, you cannot. Try have separate processes cooperate, db corrupts if open too close on heels of close. I hate ddb with the power of a thousand sharp wet farts. I bitched to my new boss and no sympathy, imagine the schadenfreude when a week later I heard the whole team move to postgres - the boss hit his own head on the thing.
Post reply on HN