Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

51–60 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#51

Post by zmgsabst »

ayhanfuat wrote: post #49
zmgsabst wrote: post #41

Earlier quoted context omitted.

But reality doesn’t care: If half your requests are 2x as long and half are 2x as fast, you don’t take the same wall time to run — you take longer. Let’s say you have 20 requests, 10 of type A and 10 of type B. They originally both take 10 seconds, for 200 seconds total. You halve A and double B. Now it takes 50 + 200 = 250 seconds, or 12.5 on average. This is a case where geometric mean deceives you - because the tw…

There is definitely no single magical number that can perfectly represent an entire set of numbers. There will always be some cases they are not representative enough. In the request example you are mostly interested in the total processing times so it does make sense you use a metric based on addition. But you could also frame a similar scenario where halving the processing time lets you handle twice as many items i…

Sure — but the arithmetic mean also captures that case: if you only halve the time, it also will report that change accurately.

What we’re handling is the case where you have split outcomes — and there the arithmetic and geometric mean disagree, so we can ask which better reflects reality.

I’m not saying the geometric mean is always wrong — but it is in this case.

A case where it makes sense is what happens when your stock halves in value then doubles in value?

In general, geometric mean is appropriate where effects are compounding (eg, two price changes to the same stock) but not when we’re combining (requests are handled differently). Two benchmarks is more combining (do task A then task B), rather than compounding.

Re: A lost decade chasing distributed architectures for data analytics?

#52

Post by mehulashah »

For those of you from the AI world, this is the equivalent of the bitter lesson and DeWitts argument about database machines from the early 80s. That is, if you wait a bit with the exponential pace of Moores law (or modern equivalents), improvements in “general purpose” hardware will obviate DB specific improvements. The problem is that back in 2012, we had customers that wanted to query terabytes of logs for observability, or analyze adtech streams, etc. So, I feel like this is a pointless argument. If your data fit on an old MacBook Pro, sure you should’ve built for that.

Re: A lost decade chasing distributed architectures for data analytics?

#53

Post by benterix »

roenxi wrote: post #47

> As recently shown, the median scan in Amazon Redshift and Snowflake reads a doable 100 MB of data, and the 99.9-percentile reads less than 300 GB. So the singularity might be closer than we think. This isn't really saying much. It is a bit like saying the 1:1000 year storm levy is overbuilt for 99.9% of storms. They aren't the storms the levy was built for, y'know. It wasn't set up with them close to the top of min…

> This isn't really saying much.

On the contrary, it's saying a lot about sheer data size, that's all. The things you mention may be crucial why Redshift and co. have been chosen (or not - in my org Redshift was used as standard so even small dataset were put into it as the management want to standardize, for better or worse), but the fact remains that if you deal with smaller datasets all of the time, you may want to reconsider the solutions you use.

Re: A lost decade chasing distributed architectures for data analytics?

#54

Post by tomrod »

threeseed wrote: post #18
rr808 wrote: post #12

Ugh I have joined a big data team. 99% of the feeds are less than a few GB yet we have to use Scala and Spark. Its so slow to develop and slow to run.

a) Scala being a JVM language is one of the fastest around. Much faster than say Python. b) How large are the 1% of the feeds and the size of the total joined datasets. Because ultimately that is what you build platforms for. Not the simple use cases.

PySpark is a wrapper, so Scala is unnecessary and boggy.

Re: A lost decade chasing distributed architectures for data analytics?

#55

Post by PaulHoule »

roenxi wrote: post #47

> As recently shown, the median scan in Amazon Redshift and Snowflake reads a doable 100 MB of data, and the 99.9-percentile reads less than 300 GB. So the singularity might be closer than we think. This isn't really saying much. It is a bit like saying the 1:1000 year storm levy is overbuilt for 99.9% of storms. They aren't the storms the levy was built for, y'know. It wasn't set up with them close to the top of min…

You can take a different approach to the 1-in-1000 jobs. Like don't do them, or approximate them. I remember the time I wrote a program that would have taken a century to finish and then developed an approximation that got it done in about 20 minutes.

Re: A lost decade chasing distributed architectures for data analytics?

#57

Post by capitol_ »

roenxi wrote: post #47

> As recently shown, the median scan in Amazon Redshift and Snowflake reads a doable 100 MB of data, and the 99.9-percentile reads less than 300 GB. So the singularity might be closer than we think. This isn't really saying much. It is a bit like saying the 1:1000 year storm levy is overbuilt for 99.9% of storms. They aren't the storms the levy was built for, y'know. It wasn't set up with them close to the top of min…

If you only have 1tb of data then you can have it in ram on a modern server.

Re: A lost decade chasing distributed architectures for data analytics?

#58

Post by szarnyasg »

mehulashah wrote: post #52

For those of you from the AI world, this is the equivalent of the bitter lesson and DeWitts argument about database machines from the early 80s. That is, if you wait a bit with the exponential pace of Moores law (or modern equivalents), improvements in “general purpose” hardware will obviate DB specific improvements. The problem is that back in 2012, we had customers that wanted to query terabytes of logs for observa…

AWS started offering local SSD storage up to 2 TB in 2012 (HI1 instance type) and in late 2013 this went up to 6.4 TB (I2 instance type). While these amounts don't cover all customers, plenty of data fits on these machines. But the software stack to analyze it efficiently was lacking, especially in the open-source space.

Re: A lost decade chasing distributed architectures for data analytics?

#59

Post by braza »

> History is full of “what if”s, what if something like DuckDB had existed in 2012? The main ingredients were there, vectorized query processing had already been invented in 2005. Would the now somewhat-silly-looking move to distributed systems for data analysis have ever happened?

I like the gist of the article, but the conclusion sounds like 20/20 hindsight.

All the elements were there, and the author nails it, but maybe the right incentive structure wasn't there to create the conditions to make it able to be done.

Between 2010 and 2015, there was a genuine feeling from almost all industry that we would converge to massive amounts of data, because until this time, the industry had never faced a time with so much abundance of data in terms of data capture and ease of placing sensors everywhere.

The natural step in this scenario won't be, most of the time, something like "let's find efficient ways to do it with the same capacity" but instead "let's invest to be able to process this in a distributed manner independent of the volume that we can have."

It's the same thing between OpenAI/ChatGPT and DeepSeek, where one can say that the math was always there, but the first runner was OpenAI with something less efficient but with a different set of incentive structures.

Re: A lost decade chasing distributed architectures for data analytics?

#60

Post by steveBK123 »

Maybe it was all VC funded solutions looking for problems?

It's a lot easier to monetize data analytics solutions if users code & data are captive in your hosted infra/cloud environment than it is to sell people a binary they can run on their own kit...

All the better if its an entire ecosystem of .. stuff.. living in "the cloud", leaving end users writing checks to 6 different portfolio companies.

Post reply on HN