Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

81–90 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#81

Post by jandrewrogers »

> As recently shown, the median scan in Amazon Redshift and Snowflake reads a doable 100 MB of data, and the 99.9-percentile reads less than 300 GB. So the singularity might be closer than we think.

There is some circular reasoning embedded here. I've seen many, many cases of people finding ways to cut up their workloads into small chunks because the performance and efficiency of these platforms is far from optimal if you actually tried to run your workload at its native scale. To some extent, these "small reads" reflect the inadequacy of the platform, not the desire of a user to run a particular workload.

A better interpretation may be that the existing distributed architectures for data analytics don't scale well except for relatively trivial workloads. There has been an awareness of this for over a decade but a dearth of platform architectures that address it.

Re: A lost decade chasing distributed architectures for data analytics?

#82

Post by znpy »

threeseed wrote: post #26
kukkeliskuu wrote: post #23

Earlier quoted context omitted.

Around 20 years ago I was working for a database company. During that time, I attended SIGMOD, which is the top conference for databases. The keynote speaker for the conference Stonebraker, who started Postgres, among other things. He talked about the history of relational databases. At that time, XML databases were all the rage -- now nobody remembers them. Stonebraker explained that there is nothing new in the hier…

MongoDB is a $2b/year revenue company growing at 20% y/y. JSON stores are not going anywhere and it's an essential tool for dealing in data where you have no control over the schema or you want to do it in the application layer. And the only "battle" is one you've invented in your head. People who deal in data for a living just pick the right data store for the right data schema.

Ah yes MongoDB, it's web-scale!

Re: A lost decade chasing distributed architectures for data analytics?

#83

Post by jcheng »

cess11 wrote: post #63
wodenokoto wrote: post #50

Earlier quoted context omitted.

You can, but then every step starts with a drop table if exists; insert into …

Or you nest your queries: select second from (select 42 as first, (select 69) as second); Intermediate steps won't be stored but until queries take a while to execute it's a nice way to do step-wise extension of an analysis. Edit: It's a rather neat and underestimated property of query results that you can query them in the next scope.

Or better yet, use CTEs: https://duckdb.org/docs/stable/sql/query_syntax/with.html

Re: A lost decade chasing distributed architectures for data analytics?

#84

Post by znpy »

braza wrote: post #64
steveBK123 wrote: post #60

Maybe it was all VC funded solutions looking for problems? It's a lot easier to monetize data analytics solutions if users code & data are captive in your hosted infra/cloud environment than it is to sell people a binary they can run on their own kit... All the better if its an entire ecosystem of .. stuff.. living in "the cloud", leaving end users writing checks to 6 different portfolio companies.

> Maybe it was all VC funded solutions looking for problems? Remember, from 2020-2023 we had an entire movement to push a thing called "Modern data stack (MDS)" with big actors like a16z lecturing the market about it [1]. I am originally from Data. Never worked with anything out of the Data: DS, MLE, DE, MLOps and so on. One thing that I envy from other developer careers is to have bosses/leaders that had battle-test…

> and in a space that wasn't easy for guys from other parts of tech to join easily and call it out (e.g., SRE, Backend, Design, Front-end, Systems Engineering, etc.).

As an SRE/SysEng/Devops/SysAdmin (depending on the company that hires me): most people in the same job as me could easily call it out.

You don't have to be such a big nerds to know that you can fit 6TB of memory in a single (physical) server. That's been true for a few years. Heck, AWS had 1TB+ memory instances for a few years now.

The thing is... Upper management wanted "big data" and the marketing people wanted to put the fancy buzzword on the company website and on linkedin. The data people wanted to be able to put the fancy buzzword on their CV (and on their Linkedin profile -- and command higher salaries due to that - can you blame them?).

> In other words, it's quite simple to sell complexity or obscure technology for most of these people

The unspoken secret is that this kind of BS wasn't/isn't only going on in the data fields (in my opinion).

Re: A lost decade chasing distributed architectures for data analytics?

#85

Post by tetromino_ »

bobchadwick wrote: post #66

It's not the point of the blog post, but I love the fact that the author's 2012 MacBook Pro is still useable. I can't imagine there are too many Dell laptops from that era still alive and kicking.

The machine from the article - a 2012 MBP Retina with 16 GB memory and 2.6 GHz i7 - had cost $2999 in the US (and significantly more in most of the rest of the world) at release. That's around $4200 today adjusting for inflation. You won't see many Dell laptops with that sort of price tag.

Re: A lost decade chasing distributed architectures for data analytics?

#86

Post by hodgesrm »

> If we look at the time a bit closer, we see the queries take anywhere between a minute and half an hour. Those are not unreasonable waiting times for analytical queries on that sort of data in any way.

I'm really skeptical arguments that say it's OK to be slow. Even on the modern laptop example queries still take up to 47 seconds.

Granted, I'm not looking at the queries but the fact is that there are a lot of applications where users need results back in less than a second.[0] If the results are feeding automated processes like page rendering they need it back in 10s of millisecond at most. That takes hardware to accomplish consistently. Especially if the datasets are large.

The small data argument becomes even weaker when you consider that analytic databases don't just do queries on static datasets. Large datasets got that way by absorbing a lot of data very quickly. They therefore do ingest, compaction, and transformations. These require resources, especially if they run in parallel with query on the same data. Scaling them independently requires distributed systems. There isn't another solution.

[0] SIEM, log management, trace management, monitoring dashboards, ... All potentially large datasets where people sift through data very quickly and repeatedly. Nobody wants to wait more than a couple seconds for results to come back.

Re: A lost decade chasing distributed architectures for data analytics?

#87

Post by spark1377485 »

tomrod wrote: post #54
threeseed wrote: post #18

Earlier quoted context omitted.

a) Scala being a JVM language is one of the fastest around. Much faster than say Python. b) How large are the 1% of the feeds and the size of the total joined datasets. Because ultimately that is what you build platforms for. Not the simple use cases.

PySpark is a wrapper, so Scala is unnecessary and boggy.

PySpark is great, except for UDF performance. This gap means that Scala is helpful for some Spark edge cases like column-level encryption/decryption with UDF

Re: A lost decade chasing distributed architectures for data analytics?

#88

Post by steveBK123 »

znpy wrote: post #84
braza wrote: post #64

Earlier quoted context omitted.

> Maybe it was all VC funded solutions looking for problems? Remember, from 2020-2023 we had an entire movement to push a thing called "Modern data stack (MDS)" with big actors like a16z lecturing the market about it [1]. I am originally from Data. Never worked with anything out of the Data: DS, MLE, DE, MLOps and so on. One thing that I envy from other developer careers is to have bosses/leaders that had battle-test…

> and in a space that wasn't easy for guys from other parts of tech to join easily and call it out (e.g., SRE, Backend, Design, Front-end, Systems Engineering, etc.). As an SRE/SysEng/Devops/SysAdmin (depending on the company that hires me): most people in the same job as me could easily call it out. You don't have to be such a big nerds to know that you can fit 6TB of memory in a single (physical) server. That's bee…

> The unspoken secret is that this kind of BS wasn't/isn't only going on in the data fields (in my opinion).

Yes, once you see it in one area you notice if everywhere.

A lot of IT spend is CEOs chasing something they half heard/misunderstanding a competitor doing, or a CTO taking Gartner a little too seriously, or engineering leads doing resume driven architecture. My last shop did a lot of this kind of this stuff "we need a head of [observability|AI|$buzzword].

The ONE thing that gives me the most pause about DuckDB is that some people in my industry who are guilty of the above are VERY interested in DuckDB. I like to wait for the serial tech evangelists to calm down a bit and see where the dust settles.

Re: A lost decade chasing distributed architectures for data analytics?

#89

Post by npalli »

DuckDB works well if

* you have a small datasets (total, not just what a single user is scanning)

* no real-time updates, just a static dataset that you can analyze at leisure

* only few users and only one doing any writes

* several seconds is an OK response time, get's worse if you have to load your scanned segment into DuckDB node.

* generally read-only workloads

So yeah, not convinced we lost a decade.

Re: A lost decade chasing distributed architectures for data analytics?

#90

Post by gopher_space »

mamcx wrote: post #75
braza wrote: post #59

> History is full of “what if”s, what if something like DuckDB had existed in 2012? The main ingredients were there, vectorized query processing had already been invented in 2005. Would the now somewhat-silly-looking move to distributed systems for data analysis have ever happened? I like the gist of the article, but the conclusion sounds like 20/20 hindsight. All the elements were there, and the author nails it, but…

It will not happened. The problem is that people believe theirs app will be web-scale pretty-soon so need to solve the problem ASAP. Is only after being burned many many times that arise the need for simplicity. Is the same of NoSql. Only after suffer it you appreciate going back. ie: Tools like this circle back only after the pain of a bubble. It can't be done inside it

> The problem is that people believe theirs app will be web-scale pretty-soon so need to solve the problem ASAP.

Investors really wanted to hear about your scaling capabilities, even when it didn't make sense. But the burn rate at places that didn't let a spreadsheet determine scale was insane.

Years working on microservices, and now I start planning/discovery with "why isn't this running on a box in the closet" and only accept numerical explanations. Putting a dollar value on excess capacity and labeling it "ad spend" changes perspectives.

Post reply on HN