Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

41–50 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#41

Post by zmgsabst »

ayhanfuat wrote: post #11
querez wrote: post #7

> The geometric mean of the timings improved from 218 to 12, a ca. 20× improvement. Why do they use the geometric mean to average execution times?

It's a way of saying twice as fast and twice as slow have equal effect on opposite sides. If your baseline is 10 seconds, one benchmark takes 5 seconds, and another one takes 20 seconds then the geometric mean gives you 10 seconds as the result because they cancel each other. The arithmetic mean would treat it differently because in absolute terms 10 seconds slow down is bigger than 5 seconds speedup. But that is not…

But reality doesn’t care:

If half your requests are 2x as long and half are 2x as fast, you don’t take the same wall time to run — you take longer.

Let’s say you have 20 requests, 10 of type A and 10 of type B. They originally both take 10 seconds, for 200 seconds total. You halve A and double B. Now it takes 50 + 200 = 250 seconds, or 12.5 on average.

This is a case where geometric mean deceives you - because the two really are asymmetric and “twice as fast” is worth less than “twice as slow”.

Re: A lost decade chasing distributed architectures for data analytics?

#42

Post by mangecoeur »

Did my phd around that time and did a project “scaling” my work on a spark cluster. Huge pita and no better than my local setup which was an MBP15 with pandas a postgres (actually I wrote+contributed a big chunk of pandas read_sql at that time to make is postgres compatible using sqlalchemy)

Re: A lost decade chasing distributed architectures for data analytics?

#43

Post by jgalt212 »

IanCal wrote: post #21
wodenokoto wrote: post #20

Earlier quoted context omitted.

A really big part of a in-memory dataframe centric workflow is how easy it is to do one step at a time and inspect the result. With a database it is difficult to run a query, look at the result and then run a query on the result. To me, that is what is missing in replacing pandas/dplyr/polars with DuckDB.

I'm not sure I really follow, you can create new tables for any step if you want to do it entirely within the db, but you can also just run duckdb against your dataframes in memory.

In R, data sources, intermediate results, and final results are all dataframes (slight simplification). With DuckDB, to have the same consistency you need every layer and step to be a database table, not a data frame, which is awkward for the standard R user and use case.

Re: A lost decade chasing distributed architectures for data analytics?

#44

Post by simlevesque »

I'm working on a big research project that uses duckdb, I need a lot of compute resources to develop my idea but I don't have a lot of money.

I'm throwing a bottle into the ocean: if anyone has spare compute with good specs they could lend me for a non-commercial project it would help me a lot.

My email is in my profile. Thank you.

Re: A lost decade chasing distributed architectures for data analytics?

#45

Post by hobs »

I have worked for a half dozen companies all swearing up and down they had big data and meaningfully one customer had 100TB of logs and another 10TB of stuff, everyone else when actually thought of properly and had just utter trash removed was really under 10TB.

Also - sqlite would have been totally fine for these queries a decade ago or more (just slower) - I messed with 10GB+ datasets with it more than 10 years ago.

Re: A lost decade chasing distributed architectures for data analytics?

#46

Post by hobs »

threeseed wrote: post #19
willvarfar wrote: post #10

Earlier quoted context omitted.

And its very much the tech culture at large that influences the company's tech choices. Those techies chasing shiny things and trying to shoehorn it into their job - perhaps cynically to pad their cvs or perhaps generously thinking it will actually be the right thing to do - have an outsized say in how tech teams think about tech and what they imagine their job is. Back in 2012 we were just recovering from the everyt…

Why you don't just quietly use SQL instead of condescending lecturing others about how compromised their tech choices are. NoSQL e.g. Cassandra, MongoDB and Microservices were invented to solve real-world problems which is why they are still so heavily used today. And the criticism of them is exactly the same that was levelled at SQL back in the day. It's all just tools at the end of the day and there isn't one that…

Every person I know who has ever used Cassandra in prod has cursed its name. Mongo lost data for close to a decade, and Microservices mostly are NOT used to solve real world problems but instead used either as an organizational or technical hammer for which everything is a nail. Hell there's entire books written how you should cut people off from each other so they can "naturally" write microservices and hyperscale your company!!

Re: A lost decade chasing distributed architectures for data analytics?

#47

Post by roenxi »

> As recently shown, the median scan in Amazon Redshift and Snowflake reads a doable 100 MB of data, and the 99.9-percentile reads less than 300 GB. So the singularity might be closer than we think.

This isn't really saying much. It is a bit like saying the 1:1000 year storm levy is overbuilt for 99.9% of storms. They aren't the storms the levy was built for, y'know. It wasn't set up with them close to the top of mind. The database might do 1,000 queries in a day.

The focus for design purposes is really to queries that live out on the tail - can they be done on a smaller database? How much value do they add? What capabilities does the database need to handle them? Etc. That is what should justify a Redshift database. Or you can provision one to hold your 1Tb of data because red things go fast and we all know it :/

Re: A lost decade chasing distributed architectures for data analytics?

#49

Post by ayhanfuat »

zmgsabst wrote: post #41
ayhanfuat wrote: post #11

Earlier quoted context omitted.

It's a way of saying twice as fast and twice as slow have equal effect on opposite sides. If your baseline is 10 seconds, one benchmark takes 5 seconds, and another one takes 20 seconds then the geometric mean gives you 10 seconds as the result because they cancel each other. The arithmetic mean would treat it differently because in absolute terms 10 seconds slow down is bigger than 5 seconds speedup. But that is not…

But reality doesn’t care: If half your requests are 2x as long and half are 2x as fast, you don’t take the same wall time to run — you take longer. Let’s say you have 20 requests, 10 of type A and 10 of type B. They originally both take 10 seconds, for 200 seconds total. You halve A and double B. Now it takes 50 + 200 = 250 seconds, or 12.5 on average. This is a case where geometric mean deceives you - because the tw…

There is definitely no single magical number that can perfectly represent an entire set of numbers. There will always be some cases they are not representative enough. In the request example you are mostly interested in the total processing times so it does make sense you use a metric based on addition. But you could also frame a similar scenario where halving the processing time lets you handle twice as many items in the same duration. In that case a ratio-based or multiplicative view might be more appropriate.

Re: A lost decade chasing distributed architectures for data analytics?

#50

Post by wodenokoto »

IanCal wrote: post #21
wodenokoto wrote: post #20

Earlier quoted context omitted.

A really big part of a in-memory dataframe centric workflow is how easy it is to do one step at a time and inspect the result. With a database it is difficult to run a query, look at the result and then run a query on the result. To me, that is what is missing in replacing pandas/dplyr/polars with DuckDB.

I'm not sure I really follow, you can create new tables for any step if you want to do it entirely within the db, but you can also just run duckdb against your dataframes in memory.

You can, but then every step starts with a drop table if exists; insert into …
Post reply on HN