Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

61–70 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#61

Post by steveBK123 »

capitol_ wrote: post #57
roenxi wrote: post #47

> As recently shown, the median scan in Amazon Redshift and Snowflake reads a doable 100 MB of data, and the 99.9-percentile reads less than 300 GB. So the singularity might be closer than we think. This isn't really saying much. It is a bit like saying the 1:1000 year storm levy is overbuilt for 99.9% of storms. They aren't the storms the levy was built for, y'know. It wasn't set up with them close to the top of min…

If you only have 1tb of data then you can have it in ram on a modern server.

AND even if you have 10TB of data, NVMe storage is ridiculously fast compared to what disk used to look like (or s3...)

Re: A lost decade chasing distributed architectures for data analytics?

#62

Post by xyzzy_plugh »

capitol_ wrote: post #57
roenxi wrote: post #47

> As recently shown, the median scan in Amazon Redshift and Snowflake reads a doable 100 MB of data, and the 99.9-percentile reads less than 300 GB. So the singularity might be closer than we think. This isn't really saying much. It is a bit like saying the 1:1000 year storm levy is overbuilt for 99.9% of storms. They aren't the storms the levy was built for, y'know. It wasn't set up with them close to the top of min…

If you only have 1tb of data then you can have it in ram on a modern server.

In the last few years, sure, but certainly not in 2012.

Re: A lost decade chasing distributed architectures for data analytics?

#63

Post by cess11 »

wodenokoto wrote: post #50
IanCal wrote: post #21

Earlier quoted context omitted.

I'm not sure I really follow, you can create new tables for any step if you want to do it entirely within the db, but you can also just run duckdb against your dataframes in memory.

You can, but then every step starts with a drop table if exists; insert into …

Or you nest your queries:

    select second from (select 42 as first, (select 69) as second);
Intermediate steps won't be stored but until queries take a while to execute it's a nice way to do step-wise extension of an analysis.

Edit: It's a rather neat and underestimated property of query results that you can query them in the next scope.

Re: A lost decade chasing distributed architectures for data analytics?

#64

Post by braza »

steveBK123 wrote: post #60

Maybe it was all VC funded solutions looking for problems? It's a lot easier to monetize data analytics solutions if users code & data are captive in your hosted infra/cloud environment than it is to sell people a binary they can run on their own kit... All the better if its an entire ecosystem of .. stuff.. living in "the cloud", leaving end users writing checks to 6 different portfolio companies.

> Maybe it was all VC funded solutions looking for problems?

Remember, from 2020-2023 we had an entire movement to push a thing called "Modern data stack (MDS)" with big actors like a16z lecturing the market about it [1].

I am originally from Data. Never worked with anything out of the Data: DS, MLE, DE, MLOps and so on. One thing that I envy from other developer careers is to have bosses/leaders that had battle-tested knowledge around delivering things using pragmatic technologies.

Most of the "AI/Data Leaders" have at maximum 15-17 years of career dealing with those tools (and I am talking about some dinosaurs in a good sense that saw the DWH or Data Mining).

After 2018 we had an explosion of people working in PoCs or small projects at best, trying to mimic what the latest blog post from some big tech company pushed.

A lot of those guys are the bosses/leaders today, and worse, they were formed during a 0% interest environment, tons of hype around the technology, little to no scrutiny or business necessity for impact, upper management that did not understand really what those guys were doing, and in a space that wasn't easy for guys from other parts of tech to join easily and call it out (e.g., SRE, Backend, Design, Front-end, Systems Engineering, etc.).

In other words, it's quite simple to sell complexity or obscure technology for most of these people, and the current moment in tech is great because we have more guys from other disciplines chime in and share their knowledge on how to assess and implement technology.

[1] - https://a16z.com/emerging-architectures-for-modern-data-infr...

Re: A lost decade chasing distributed architectures for data analytics?

#65

Post by jononor »

mangecoeur wrote: post #42

Did my phd around that time and did a project “scaling” my work on a spark cluster. Huge pita and no better than my local setup which was an MBP15 with pandas a postgres (actually I wrote+contributed a big chunk of pandas read_sql at that time to make is postgres compatible using sqlalchemy)

Thank you for read_sql with SQLalchemy/postgres! We use it all the time at our company:)

Re: A lost decade chasing distributed architectures for data analytics?

#69

Post by dagw »

godber wrote: post #68

This makes a completely valid point when you constrain the meaning of Big Data to “the largest dataset one can fit on a single computer”.

At companies I've worked at "Big Data" was often used to mean "too big to open in Excel" or in the extreme case "too big to fit in RAM on my laptop"

Re: A lost decade chasing distributed architectures for data analytics?

#70

Post by steveBK123 »

braza wrote: post #64
steveBK123 wrote: post #60

Maybe it was all VC funded solutions looking for problems? It's a lot easier to monetize data analytics solutions if users code & data are captive in your hosted infra/cloud environment than it is to sell people a binary they can run on their own kit... All the better if its an entire ecosystem of .. stuff.. living in "the cloud", leaving end users writing checks to 6 different portfolio companies.

> Maybe it was all VC funded solutions looking for problems? Remember, from 2020-2023 we had an entire movement to push a thing called "Modern data stack (MDS)" with big actors like a16z lecturing the market about it [1]. I am originally from Data. Never worked with anything out of the Data: DS, MLE, DE, MLOps and so on. One thing that I envy from other developer careers is to have bosses/leaders that had battle-test…

Right.. shove your data in our data platform.

OK now you need PortCo1's company analytics platform, PortCo2's orchestration platform, PortCo3's SRE platform, PortCo4's Auth platform, PortCo5's IaC platform, PortCo6's Secrets Mgmt Platform, PortoCo7's infosec platform, etc.

I am sure I forgot another 10 things. Even if some of these things were open source or "open source", there was the upsell to the managed/supported/business license/etc version for many of these tools.

Post reply on HN