Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

31–40 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#31

Post by cgio »

rr808 wrote: post #29
threeseed wrote: post #28

Earlier quoted context omitted.

a) You can easily run Spark jobs on a single box. Just set executors = 1. b) The reason centralised clusters exist is because you can't have dozens/hundreds of data engineers/scientists all copying company data onto their laptop, causing support headaches because they can't install X library and making productionising impossible. There are bigger concerns than your personal productivity.

> a) You can easily run Spark jobs on a single box. Just set executors = 1. Sure but why would you do this? Just using pandas or duckdb or even bash scripts makes your life is much easier than having to deal with Spark.

For when you need more executors without rewriting your logic.

Re: A lost decade chasing distributed architectures for data analytics?

#32

Post by HPsquared »

drewm1980 wrote: post #2

I mean, not everyone spent their decade on distributed computing. Some devs with a retrogrouch inclination kept writing single threaded code in native languages on a single node. Single core clock speed stagnated, but it was still worth buying new CPU's with more cores because they also had more cache, and all the extra cores are useful for running ~other peoples' bloated code.

High-frequency trading, gaming, audio/DSP, embedded, etc. There's a lot of room for that kind of developer.

Re: A lost decade chasing distributed architectures for data analytics?

#33

Post by tbillington »

ayhanfuat wrote: post #11
querez wrote: post #7

> The geometric mean of the timings improved from 218 to 12, a ca. 20× improvement. Why do they use the geometric mean to average execution times?

It's a way of saying twice as fast and twice as slow have equal effect on opposite sides. If your baseline is 10 seconds, one benchmark takes 5 seconds, and another one takes 20 seconds then the geometric mean gives you 10 seconds as the result because they cancel each other. The arithmetic mean would treat it differently because in absolute terms 10 seconds slow down is bigger than 5 seconds speedup. But that is not…

This is the best explain-like-im-5 I've heard for geo mean and helped it click in my head, thank you :)

Re: A lost decade chasing distributed architectures for data analytics?

#34

Post by compiler-devel »

Cthulhu_ wrote: post #16
willvarfar wrote: post #4

I only retired my 2014 MBP ... last week! It started transiently not booting and then, after just a few weeks, it switched to be only transiently booting. Figured it was time. My new laptop is actually a very budget buy, and not a mac, and in many things a bit slower than the old MBP. Anyway, the old laptop is about par with the 'big' VMs that I use for work to analyse really big BQ datasets. My current flow is to do…

I still have mine, but it's languishing, I don't know what to do with it / how to get rid of it, it doesn't feel like trash. The Apple stores do returns but for this one you get nothing, they're just like "yeah we'll take care of it". The screen started to delaminate on the edges, and its follow-up (a MBP with the touch bar)'s screen is completely broken (probably just the connector cable). I don't have a use for it,…

I have the same machine and installed Fedora 41 on it. Everything works out of the box, including WiFi and sound.

Re: A lost decade chasing distributed architectures for data analytics?

#35

Post by rpier001 »

threeseed wrote: post #28
rr808 wrote: post #22

Earlier quoted context omitted.

1) Yes Scala and JVM is fast. If we could just use that to clean up a feed on a single box that would be great. The problem is calling the Spark API creates a lot of complexity for developers and runtime platform which is super slow. 2) Yes for the few feeds that are a TB we need spark. The platform really just loads from hadoop transforms then saves back again.

a) You can easily run Spark jobs on a single box. Just set executors = 1. b) The reason centralised clusters exist is because you can't have dozens/hundreds of data engineers/scientists all copying company data onto their laptop, causing support headaches because they can't install X library and making productionising impossible. There are bigger concerns than your personal productivity.

Re: b. This is a place where remote standard dev environments are a boon. I'm not going to give each dev a terabyte of RAM, but a terabyte to share with a reservation mechanism understanding that contention for the full resource is low? Yes, please.

Re: A lost decade chasing distributed architectures for data analytics?

#36

Post by culebron21 »

A tangential story. I remember, back in 2010, contemplating the idea of completely distributed DBs inspired by then popular torrent technology. In this one, a client would not be different from a server, except by the amount of data it holds. And it would probably receive the data in torrents manner.

What puzzled me was that a client would want others to execute its queries, but not want to load all the data and make queries for the others. And how to prevent conflicting update queries sent to different seeds.

I also thought that Crockford's distributed web idea (where every page is hosted like on torrents) was a good one, even though I didn't think deep of this one.

Until I saw the discussion on web3, where someone pointed out that uploading any data on one server would make a lot of hosts to do the job of hosting a part of it, and every small movement would cause tremendous amounts of work for the entire web.

Re: A lost decade chasing distributed architectures for data analytics?

#37

Post by pragmatic »

threeseed wrote: post #26
kukkeliskuu wrote: post #23

Earlier quoted context omitted.

Around 20 years ago I was working for a database company. During that time, I attended SIGMOD, which is the top conference for databases. The keynote speaker for the conference Stonebraker, who started Postgres, among other things. He talked about the history of relational databases. At that time, XML databases were all the rage -- now nobody remembers them. Stonebraker explained that there is nothing new in the hier…

MongoDB is a $2b/year revenue company growing at 20% y/y. JSON stores are not going anywhere and it's an essential tool for dealing in data where you have no control over the schema or you want to do it in the application layer. And the only "battle" is one you've invented in your head. People who deal in data for a living just pick the right data store for the right data schema.

And sql server alone is like 5 billion/yr.

Re: A lost decade chasing distributed architectures for data analytics?

#38

Post by Larrikin »

threeseed wrote: post #18
rr808 wrote: post #12

Ugh I have joined a big data team. 99% of the feeds are less than a few GB yet we have to use Scala and Spark. Its so slow to develop and slow to run.

a) Scala being a JVM language is one of the fastest around. Much faster than say Python. b) How large are the 1% of the feeds and the size of the total joined datasets. Because ultimately that is what you build platforms for. Not the simple use cases.

But can you justify Scala existing at all in 2025. I think it pushed boundaries but ultimately failed as a language worth adoption.l anymore.

Re: A lost decade chasing distributed architectures for data analytics?

#39

Post by this_user »

cgio wrote: post #31
rr808 wrote: post #29

Earlier quoted context omitted.

> a) You can easily run Spark jobs on a single box. Just set executors = 1. Sure but why would you do this? Just using pandas or duckdb or even bash scripts makes your life is much easier than having to deal with Spark.

For when you need more executors without rewriting your logic.

Using a Python solution like Dask might actually be better, because you can work with all of the Python data frameworks and tools, but you can also easily scale it if you need it without having to step into the Spark world.

Re: A lost decade chasing distributed architectures for data analytics?

#40

Post by nyanpasu64 »

drewm1980 wrote: post #2

I mean, not everyone spent their decade on distributed computing. Some devs with a retrogrouch inclination kept writing single threaded code in native languages on a single node. Single core clock speed stagnated, but it was still worth buying new CPU's with more cores because they also had more cache, and all the extra cores are useful for running ~other peoples' bloated code.

I find that good multithreading can speed up parallelizable workloads by 5-10 times depending on CPU core count, if you don't have tight latency constraints (and even games with millisecond-level latency deadlines are multithreaded these days, though real-time code may look different than general code).
Post reply on HN