Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

101–110 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#101

Post by steveBK123 »

kwillets wrote: post #95
braza wrote: post #64

Earlier quoted context omitted.

> Maybe it was all VC funded solutions looking for problems? Remember, from 2020-2023 we had an entire movement to push a thing called "Modern data stack (MDS)" with big actors like a16z lecturing the market about it [1]. I am originally from Data. Never worked with anything out of the Data: DS, MLE, DE, MLOps and so on. One thing that I envy from other developer careers is to have bosses/leaders that had battle-test…

Cloud and SaaS were good for a while because they took away the old sales-CTO pipeline that often saw a whole org suffering from one person's signature. But they also took away the benefits of a more formal evaluation process, and nowadays nobody knows how to do one.

I'm not sure how cloud/saas made the CTO behavior and its consequences any better. At least on-prem if they picked the "wrong" DB / message bus / etc, you could quietly replicate to another stack internally as needed for your analytics needs.

If your data is lodged in some SaaS product in AWS, good luck replicating that to GCP, Azure, or heaven forbid on-prem, without extortion level costs.

Re: A lost decade chasing distributed architectures for data analytics?

#102

Post by hobs »

threeseed wrote: post #93
hobs wrote: post #46

Earlier quoted context omitted.

Every person I know who has ever used Cassandra in prod has cursed its name. Mongo lost data for close to a decade, and Microservices mostly are NOT used to solve real world problems but instead used either as an organizational or technical hammer for which everything is a nail. Hell there's entire books written how you should cut people off from each other so they can "naturally" write microservices and hyperscale y…

So all of this is just meaningless anecdotes. Whereas the fact is that Datastax and MongoDB are highly successful companies indicating that in fact those databases are solving a real world problem.

So is Phillip Morris - feel free to smoke all you want.

Re: A lost decade chasing distributed architectures for data analytics?

#103

Post by cess11 »

jcheng wrote: post #83
cess11 wrote: post #63

Earlier quoted context omitted.

Or you nest your queries: select second from (select 42 as first, (select 69) as second); Intermediate steps won't be stored but until queries take a while to execute it's a nice way to do step-wise extension of an analysis. Edit: It's a rather neat and underestimated property of query results that you can query them in the next scope.

Or better yet, use CTEs: https://duckdb.org/docs/stable/sql/query_syntax/with.html

Absolutely, if the engine has them and they're not wonky somehow.

Re: A lost decade chasing distributed architectures for data analytics?

#104

Post by wodenokoto »

cess11 wrote: post #63
wodenokoto wrote: post #50

Earlier quoted context omitted.

You can, but then every step starts with a drop table if exists; insert into …

Or you nest your queries: select second from (select 42 as first, (select 69) as second); Intermediate steps won't be stored but until queries take a while to execute it's a nice way to do step-wise extension of an analysis. Edit: It's a rather neat and underestimated property of query results that you can query them in the next scope.

We all have different definitions on what is difficult. Maybe annoying or bothersome had been better words, but below beats nesting things:

    df |> select(..) |>
        filter(...) |>
        mutate(...) |>
        ...
And every time I've learned something about the intermediate result I can add another line, or save the result in a new variable and branch my exploration. And I can easily just highlight and run and number of of steps from step 1 onwards.

Even oldschool

    df2
Gives me the same benefit.

Re: A lost decade chasing distributed architectures for data analytics?

#105

Post by tsss »

Larrikin wrote: post #38
threeseed wrote: post #18

Earlier quoted context omitted.

a) Scala being a JVM language is one of the fastest around. Much faster than say Python. b) How large are the 1% of the feeds and the size of the total joined datasets. Because ultimately that is what you build platforms for. Not the simple use cases.

But can you justify Scala existing at all in 2025. I think it pushed boundaries but ultimately failed as a language worth adoption.l anymore.

Scala is still one of the most powerful languages out there.

Re: A lost decade chasing distributed architectures for data analytics?

#106

Post by cess11 »

wodenokoto wrote: post #104
cess11 wrote: post #63

Earlier quoted context omitted.

Or you nest your queries: select second from (select 42 as first, (select 69) as second); Intermediate steps won't be stored but until queries take a while to execute it's a nice way to do step-wise extension of an analysis. Edit: It's a rather neat and underestimated property of query results that you can query them in the next scope.

We all have different definitions on what is difficult. Maybe annoying or bothersome had been better words, but below beats nesting things: df |> select(..) |> filter(...) |> mutate(...) |> ... And every time I've learned something about the intermediate result I can add another line, or save the result in a new variable and branch my exploration. And I can easily just highlight and run and number of of steps from st…

Yeah, sure, I do a lot of such things in RAM in Elixir, some Lisp, PHP or, if I must, Python.

But sometimes I just happen to have just imported a data set in a SQL client or I'm hooked into a remote database where I don't have anything but the SQL client. When developing an involved analysis query nesting also comes in handy sometimes, e.g. to mock away a part of the full query.

Re: A lost decade chasing distributed architectures for data analytics?

#107

Post by specialist »

rr808 wrote: post #12

Ugh I have joined a big data team. 99% of the feeds are less than a few GB yet we have to use Scala and Spark. Its so slow to develop and slow to run.

After a year, I managed to convince a team to migrate our hottest data set (~1.5m records) from DynamoDB to Redis.

Hot damn, we collectively spent so much time mitigating our misuse & abuse of DynamoDB.

Re: A lost decade chasing distributed architectures for data analytics?

#108

Post by kwillets »

kukkeliskuu wrote: post #23
threeseed wrote: post #19

Earlier quoted context omitted.

Why you don't just quietly use SQL instead of condescending lecturing others about how compromised their tech choices are. NoSQL e.g. Cassandra, MongoDB and Microservices were invented to solve real-world problems which is why they are still so heavily used today. And the criticism of them is exactly the same that was levelled at SQL back in the day. It's all just tools at the end of the day and there isn't one that…

Around 20 years ago I was working for a database company. During that time, I attended SIGMOD, which is the top conference for databases. The keynote speaker for the conference Stonebraker, who started Postgres, among other things. He talked about the history of relational databases. At that time, XML databases were all the rage -- now nobody remembers them. Stonebraker explained that there is nothing new in the hier…

Stonebraker is one of the few whose criticism is listened to. He recently updated his "What goes around comes around" paper; it's worth a read:

https://db.cs.cmu.edu/papers/2024/whatgoesaround-sigmodrec20...

Re: A lost decade chasing distributed architectures for data analytics?

#109

Post by mountainriver »

icedchai wrote: post #99
mountainriver wrote: post #98

Earlier quoted context omitted.

I can’t believe anyone would write scala at this point

I worked at a Scala shop about 15 years ago. It was terrible. Everyone had their own "dialect" of features they used (kinda like C++.). The tooling was the worst (Eclipse's Scala plugin was especially awful, IntelliJ's was okay.) The compiler was slow. I'm assuming it's better now?

Yeah I had a similar experience, it was the worst dev shop I've ever been a part of. I tried it again 5 or so years back, it was a bit better but still a lot of SBT strangeness and unneeded abstractions

Re: A lost decade chasing distributed architectures for data analytics?

#110

Post by threeseed »

hobs wrote: post #102
threeseed wrote: post #93

Earlier quoted context omitted.

So all of this is just meaningless anecdotes. Whereas the fact is that Datastax and MongoDB are highly successful companies indicating that in fact those databases are solving a real world problem.

So is Phillip Morris - feel free to smoke all you want.

What an idiotic and childish take.

Yes using JSON to store your data will kill millions of people.

Post reply on HN