Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

91–100 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#91

Post by mehulashah »

szarnyasg wrote: post #58
mehulashah wrote: post #52

For those of you from the AI world, this is the equivalent of the bitter lesson and DeWitts argument about database machines from the early 80s. That is, if you wait a bit with the exponential pace of Moores law (or modern equivalents), improvements in “general purpose” hardware will obviate DB specific improvements. The problem is that back in 2012, we had customers that wanted to query terabytes of logs for observa…

AWS started offering local SSD storage up to 2 TB in 2012 (HI1 instance type) and in late 2013 this went up to 6.4 TB (I2 instance type). While these amounts don't cover all customers, plenty of data fits on these machines. But the software stack to analyze it efficiently was lacking, especially in the open-source space.

AWS also had customers that had petabytes of data in Redshift for analysis. The conversation is missing a key point: DuckDB is optimizing for a different class of use cases. They’re optimizing for data science and not traditional data warehousing use cases. It’s masquerading as size. Even for small sizes, there are other considerations: access control, concurrency control, reliability, availability, and so on. The requirements are different for those different use cases. Data science tends to be single user, local, and lower availability requirements than warehouses that serve production pipelines, data sharing, and so on. I also think that DuckDB can be used for those, but not optimized for those.

Data size is a red herring in the conversation.

Re: A lost decade chasing distributed architectures for data analytics?

#92

Post by threeseed »

pragmatic wrote: post #37
threeseed wrote: post #26

Earlier quoted context omitted.

MongoDB is a $2b/year revenue company growing at 20% y/y. JSON stores are not going anywhere and it's an essential tool for dealing in data where you have no control over the schema or you want to do it in the application layer. And the only "battle" is one you've invented in your head. People who deal in data for a living just pick the right data store for the right data schema.

And sql server alone is like 5 billion/yr.

Almost like there is room in the market for more than just SQL databases.

Re: A lost decade chasing distributed architectures for data analytics?

#93

Post by threeseed »

hobs wrote: post #46
threeseed wrote: post #19

Earlier quoted context omitted.

Why you don't just quietly use SQL instead of condescending lecturing others about how compromised their tech choices are. NoSQL e.g. Cassandra, MongoDB and Microservices were invented to solve real-world problems which is why they are still so heavily used today. And the criticism of them is exactly the same that was levelled at SQL back in the day. It's all just tools at the end of the day and there isn't one that…

Every person I know who has ever used Cassandra in prod has cursed its name. Mongo lost data for close to a decade, and Microservices mostly are NOT used to solve real world problems but instead used either as an organizational or technical hammer for which everything is a nail. Hell there's entire books written how you should cut people off from each other so they can "naturally" write microservices and hyperscale y…

So all of this is just meaningless anecdotes.

Whereas the fact is that Datastax and MongoDB are highly successful companies indicating that in fact those databases are solving a real world problem.

Re: A lost decade chasing distributed architectures for data analytics?

#94

Post by threeseed »

Larrikin wrote: post #38
threeseed wrote: post #18

Earlier quoted context omitted.

a) Scala being a JVM language is one of the fastest around. Much faster than say Python. b) How large are the 1% of the feeds and the size of the total joined datasets. Because ultimately that is what you build platforms for. Not the simple use cases.

But can you justify Scala existing at all in 2025. I think it pushed boundaries but ultimately failed as a language worth adoption.l anymore.

Absolutely.

a) One of the only languages you can write your entire app in Scala i.e. it supports compiling to Javascript, JVM and LLVM.

b) It has the only formally proven type system of any language.

c) It is the innovation language. Many of the concepts that are now standard in other languages had their implementation borrowed from Scala. And it is continuing to innovate with libraries like Gears (https://github.com/lampepfl/gears) which does async without colouring and compiler additions like resource capabilities.

Re: A lost decade chasing distributed architectures for data analytics?

#95

Post by kwillets »

braza wrote: post #64
steveBK123 wrote: post #60

Maybe it was all VC funded solutions looking for problems? It's a lot easier to monetize data analytics solutions if users code & data are captive in your hosted infra/cloud environment than it is to sell people a binary they can run on their own kit... All the better if its an entire ecosystem of .. stuff.. living in "the cloud", leaving end users writing checks to 6 different portfolio companies.

> Maybe it was all VC funded solutions looking for problems? Remember, from 2020-2023 we had an entire movement to push a thing called "Modern data stack (MDS)" with big actors like a16z lecturing the market about it [1]. I am originally from Data. Never worked with anything out of the Data: DS, MLE, DE, MLOps and so on. One thing that I envy from other developer careers is to have bosses/leaders that had battle-test…

Cloud and SaaS were good for a while because they took away the old sales-CTO pipeline that often saw a whole org suffering from one person's signature. But they also took away the benefits of a more formal evaluation process, and nowadays nobody knows how to do one.

Re: A lost decade chasing distributed architectures for data analytics?

#96

Post by nojito »

mehulashah wrote: post #91
szarnyasg wrote: post #58

Earlier quoted context omitted.

AWS started offering local SSD storage up to 2 TB in 2012 (HI1 instance type) and in late 2013 this went up to 6.4 TB (I2 instance type). While these amounts don't cover all customers, plenty of data fits on these machines. But the software stack to analyze it efficiently was lacking, especially in the open-source space.

AWS also had customers that had petabytes of data in Redshift for analysis. The conversation is missing a key point: DuckDB is optimizing for a different class of use cases. They’re optimizing for data science and not traditional data warehousing use cases. It’s masquerading as size. Even for small sizes, there are other considerations: access control, concurrency control, reliability, availability, and so on. The re…

>Data size is a red herring in the conversation.

Not really. A Redshift paper just shared that.

>..here is a small number of tables in Redshift with trillions of rows, while the majority is much more reasonably sized with only millions of rows. In fact, most tables have less than a million rows and the vast majority (98 %) has less than a billion rows.

The argument can be made that 98% of people using redshift can potentially get by with DuckDB.

https://assets.amazon.science/24/3b/04b31ef64c83acf98fe3fdca...

Re: A lost decade chasing distributed architectures for data analytics?

#97

Post by mountainriver »

threeseed wrote: post #94
Larrikin wrote: post #38

Earlier quoted context omitted.

But can you justify Scala existing at all in 2025. I think it pushed boundaries but ultimately failed as a language worth adoption.l anymore.

Absolutely. a) One of the only languages you can write your entire app in Scala i.e. it supports compiling to Javascript, JVM and LLVM. b) It has the only formally proven type system of any language. c) It is the innovation language. Many of the concepts that are now standard in other languages had their implementation borrowed from Scala. And it is continuing to innovate with libraries like Gears ( https://github.co…

I’m sorry but these are extremely weak arguments, and I would contend scale caused more harm than good in all

Re: A lost decade chasing distributed architectures for data analytics?

#99

Post by icedchai »

mountainriver wrote: post #98
rr808 wrote: post #12

Ugh I have joined a big data team. 99% of the feeds are less than a few GB yet we have to use Scala and Spark. Its so slow to develop and slow to run.

I can’t believe anyone would write scala at this point

I worked at a Scala shop about 15 years ago. It was terrible. Everyone had their own "dialect" of features they used (kinda like C++.). The tooling was the worst (Eclipse's Scala plugin was especially awful, IntelliJ's was okay.) The compiler was slow.

I'm assuming it's better now?

Re: A lost decade chasing distributed architectures for data analytics?

#100

Post by icedchai »

threeseed wrote: post #26
kukkeliskuu wrote: post #23

Earlier quoted context omitted.

Around 20 years ago I was working for a database company. During that time, I attended SIGMOD, which is the top conference for databases. The keynote speaker for the conference Stonebraker, who started Postgres, among other things. He talked about the history of relational databases. At that time, XML databases were all the rage -- now nobody remembers them. Stonebraker explained that there is nothing new in the hier…

MongoDB is a $2b/year revenue company growing at 20% y/y. JSON stores are not going anywhere and it's an essential tool for dealing in data where you have no control over the schema or you want to do it in the application layer. And the only "battle" is one you've invented in your head. People who deal in data for a living just pick the right data store for the right data schema.

I find using Postgres and JSONB often gets me the best of both worlds.
Post reply on HN