81–90 of 111 posts
I really hope this is not the case again.
(yes, I understand my sarcasm is unneeded, I couldn't help myself)
[1]: https://www.ververica.com/blog/curious-case-broken-benchmark...
Also proof that lakehouse and spot compute price performance economics are here to stay, that's good for customers.
Otherwise, as a vendor blog post with nothing but self-reported performance, this is worthless.
Disclaimer: I work at Databricks but I admire Snowflake's product for what it is - iron sharpens iron.
Earlier quoted context omitted.
> I'd hope so if they're selling a product based on FOSS code, there has to be a value add to justify it They have some proprietary features like DBIO [1]. They also have some cloud-specific features like storage autoscaling [2] that would not be available in OSS Spark. Even Delta Lake [3] used to be proprietary, but I suspect the rise of open-source frameworks like Iceberg led them to open-source it. Shameless plug…
The open source Delta is not a replacement for the real thing - they did not include features like optimizing small files (small file problem is well known in big data, and much more of a problem once streaming gets involved) and others. It is more of a demo of the real thing. Which does not stop them from repeating everywhere how open they are, of course. EDIT: the delta also still keeps partitioning information in…
Check out https://github.com/delta-io/delta/blob/3ffb30d86c6acda9b59b9... when you get a chance. You don't need hive metastore to query delta tables since all metadata for a Delta table is stored alongside the data
>they did not include features like optimizing small files
For optimizing small files, you could run https://docs.delta.io/latest/best-practices.html#compact-fil...
Earlier quoted context omitted.
It’s probably just me but the distinction between datalake and data warehouse seems like splitting hairs. Unstructured data can always be stored on structure databases. What’s the main reason for both to coexist?
It used to be that way. Old data warehouses (built on relational dbs) couldn't handle large scale data, and old data lakes used to be hard to use (write a map-reduce job to query data). It is barely true nowadays.
i didn't enjoy working w/either the datastore directly, or the DBA team that ran it either. an early, more old-white-dude "i just want to serve 5T"
Can someone ELI5 what Snowflake and Databricks are? I spent a few minutes on the Databricks website once and couldn't really penetrate the marketing jargon. There are also some technical terms I don't know at all, and when I've searched for them, the top results are all more Azure stuff. Like wtf is a datalake?
They are a data warehouse with analytics? So data warehouse as a service in the cloud? So they can collect data from different places like sql, images, etc. I think a better question would be what type of data can't they ingest? Once you have your data i guess you can run some analytics to find out what your data tells you
This really all ties back to the "old" Hadoop days, and is an evolution of compute over data not in a fixed and managed format/schema.
Earlier quoted context omitted.
They are a data warehouse with analytics? So data warehouse as a service in the cloud? So they can collect data from different places like sql, images, etc. I think a better question would be what type of data can't they ingest? Once you have your data i guess you can run some analytics to find out what your data tells you
I'd like to add some points: Ive used Snowflake for several years. Snowflake works with structured and semi-structured data (think spreadsheets and JSON). I've never tried working with pics or videos - and I'm not sure it would make sense to do that. I've evaluated Databricks. It works with the above mentioned structured and semi-structured data. I also suspect it could process unstructured data. My understanding is…
Earlier quoted context omitted.
Geometric mean is commonly used in benchmarks when the workloads consists of queries that have large (often orders of magnitude) differences in runtime. Consider 4 queries. Two run for 1sec, and the other two 1000sec. If we look at arithmetic mean, then we are really only taking into account the large queries. But improving geometric mean would require improving all queries. Note that I'm on the opposite side (Databr…
> But improving geometric mean would require improving all queries. No. Improving the geometric mean only requires reducing the product of their execution times. So if you can make the two 1 ms queries execute in 0.5 ms at the expense of the two 1000 ms queries taking 1800 ms each then that’s an improvement in terms of geometric mean. So… kind of QED. The geometric mean is not easy to reason about.
One of the benefits of geometric mean is that all queries have "equal" weight in the metric, this keeps vendors from focusing on the long running queries and ignoring the short running ones. It is one way to balance between long and short query performance.
A similar concept is applied to TPC-DS for data load, single user run (Power), multi user run (Throughput) and data maintenance (Concurrent Delete and Inserts).
Check clause 7.6.3.1 in the TPC-Ds spec in http://tpc.org/tpc_documents_current_versions/pdf/tpc-ds_v3....
Snowflake conceding they have a 700% markup between Standard and Premium editons which has zero impact on query performance is ... well, it's something. I'd start squeezing my sales engineers about that, definitely not sustainable... Also proof that lakehouse and spot compute price performance economics are here to stay, that's good for customers. Otherwise, as a vendor blog post with nothing but self-reported perfor…
Earlier quoted context omitted.
People who downvoted this, please take a minute and reflect that your world is not the whole world. There is a serious question in this comment and there are myriads of topics _you_ have no clue about.
sure, but if I see the term 'data lake' I'm gonna Bing it, with the first result being https://aws.amazon.com/big-data/datalakes-and-analytics/what... which explains it nicely. ELI5 is for reddit, generally here we expect you can google it to get the ELI5 explanation before giving us your hot take in a comment
> A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.
This may be self-explanatory for you, but what it means in practice is not as self-evident as you believe. For all it describes, it could be an FTP upload directory that loads things into an sqlite database. It's not until the scale is invoked (multi-terabyte/day) that the inadequacies of a naive solution become apparent. For those in that area of the industry, Snowflake is already known. (Seriously, if you're running into issues with limitations of RedShift, it behooves you to take a look at Snowflake.) For those that aren't, data warehousing is unfamiliar, never mind data lake. For those outside the ML sphere, the finer points of training runs are also non-obvious.
Earlier quoted context omitted.
If people have never heard of Databricks, now is the time because a 100 billion company just started a war against them. Great marketing win Databricks.
To be fair, I've been equating Databricks for a month or so. Databricks is coming after Snowflake. Snowflake doesn't care. Snowflake has a pretty solid moat with: EASY SQL, data sharing (they have a marketplace), simple scaling