Good to finally know the difference between a lake and a warehouse
Guide to data tools landscape for developers
11–20 of 38 posts
Re: Guide to data tools landscape for developers
#12Well, I find this post looks good, but a like lot of 'data for developers' posts it's just a list of tools. As if a collection of tools banded together actually makes your customer successful.
What's missing?
1. There's nothing about deployment. How do I take this collection of tools and code and actually deploy it into production, or actually regression test it functionally? How do I make a small change in a database table and not have a massive regression? How do you do that automatically? How do you do it quickly?
2. It's cursory on testing. One of the biggest differences from a software developer to a data engineer is that your data providers give you crap data all the time. It could break. How do you test data? How do you get adequate test coverage? These things are essential for software developers and are actually doubly essential for data engineers and building analytics systems.
3. It's what success looks like. It's not just about a collection of tech; it's about making your customers successful. What does it mean to deliver good insight? How do you do it? How do you measure customer success, and measure your success? As a team, you wouldn't talk about software engineering without mentioning DevOps or DORA metrics. There's nothing here about that.
sigh
Re: Guide to data tools landscape for developers
#13excluding Denodo from the list sows this is more of a non-enterprise guide to data management and tools. There is only one real semantic layer that can cover operational and historical data and thats Denodo. If you use snowflake horizon or unity, all the data needs to be loaded first and not real time.
Re: Guide to data tools landscape for developers
#14Nice guide. One addition in the ingestion section: sling (https://slingdata.io/). it's a single go binary, cli-first.
Re: Guide to data tools landscape for developers
#15> A data warehouse on the other hand is an OLAP database and is optimized to work on columns A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together…
You are pedantically correct but technically wrong, as even optimized postgres is going to suffer on analytical patterns without extensions. With extensions (DuckDB or Citus) you can do large aggregations, but regular postgres at medium/large scale (billions of rows, 100s of GB) starts having a lot of foot guns and complex babying to do analytics. A bunch of indexes and you'll be fine though.
Re: Guide to data tools landscape for developers
#16What about PI?
Re: Guide to data tools landscape for developers
#17excluding Denodo from the list sows this is more of a non-enterprise guide to data management and tools. There is only one real semantic layer that can cover operational and historical data and thats Denodo. If you use snowflake horizon or unity, all the data needs to be loaded first and not real time.
I have never heard of Denodo before. "The only real semantic layer" is a bold claim. From the website it just looks like another data governance tool?
Re: Guide to data tools landscape for developers
#18As a data engineering person I can say that this is a great write up!
Some thoughts:
A "bubbling" topic right now is conversational analytics (i.e. talk to your data). There has been an explosion of tools in the last 6 months. YC is backing one too: https://getnao.io/
I feel like pandas is also somewhat frowned upon, the industry has moved on from that. Most SQL tools can now do everything that we could only do with pandas.
In my network everyone is talking about DuckDB. As long as you are under a 1TB it will have everything you need. I think most people should start with that vs locking themselves into something like Snowflake
Re: Guide to data tools landscape for developers
#19"Popular metadata catalog solutions are Hive Metastore, AWS Glue Data Catalog, and Unity Catalog from Databricks.
For query engine you can use, for example, Apache Spark, Trino, or Amazon Athena."
DuckDB is eating the query engines and catalogs. Really could use more coverage on how DuckDB is changing the data tools landscape.
Re: Guide to data tools landscape for developers
#20As a data engineering person I can say that this is a great write up! Some thoughts: A "bubbling" topic right now is conversational analytics (i.e. talk to your data). There has been an explosion of tools in the last 6 months. YC is backing one too: https://getnao.io/ I feel like pandas is also somewhat frowned upon, the industry has moved on from that. Most SQL tools can now do everything that we could only do with…
Just to add, people in my network have been talking about polars (as an alternative to pandas) and other dataframe libraries. They're much easier to use now thanks to the Narwhals compatibility layer (for example, Narwhals was recently added as a dependency to scikit-learn).