Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

11–20 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#11

Post by ayhanfuat »

querez wrote: post #7

> The geometric mean of the timings improved from 218 to 12, a ca. 20× improvement. Why do they use the geometric mean to average execution times?

It's a way of saying twice as fast and twice as slow have equal effect on opposite sides. If your baseline is 10 seconds, one benchmark takes 5 seconds, and another one takes 20 seconds then the geometric mean gives you 10 seconds as the result because they cancel each other. The arithmetic mean would treat it differently because in absolute terms 10 seconds slow down is bigger than 5 seconds speedup. But that is not fair for speedups because the absolute speedup you can reach is at most 10 seconds but slow down has no limits.

Re: A lost decade chasing distributed architectures for data analytics?

#13

Post by Mortiffer »

The R community has been hard at work on small data. I still highly prefer working on on memory data in R dplyr DataTable are elegant and fast.

The CRan packages are all high quality if the maintainer stops responding to emails for 2 months your package is automatically removed. Most packages come from university Prof's that have been doing this their whole career.

Re: A lost decade chasing distributed architectures for data analytics?

#14

Post by DonHopkins »

zkmon wrote: post #6

A database is not only about disk size and query performance. Database reflects the company's culture, processes, workflows, collaboration etc. It has an entire ecosystem around it - master data, business processes, transactions, distributed applications, regulatory requirements, resiliency, Ops, reports, tooling etc, The role of a database is not just to deliver query performance. It needs to fit into the ecosystem,…

You can always make your data bigger without increasing disk space or decreasing performance by making the font size larger!

Re: A lost decade chasing distributed architectures for data analytics?

#16

Post by Cthulhu_ »

willvarfar wrote: post #4

I only retired my 2014 MBP ... last week! It started transiently not booting and then, after just a few weeks, it switched to be only transiently booting. Figured it was time. My new laptop is actually a very budget buy, and not a mac, and in many things a bit slower than the old MBP. Anyway, the old laptop is about par with the 'big' VMs that I use for work to analyse really big BQ datasets. My current flow is to do…

I still have mine, but it's languishing, I don't know what to do with it / how to get rid of it, it doesn't feel like trash. The Apple stores do returns but for this one you get nothing, they're just like "yeah we'll take care of it".

The screen started to delaminate on the edges, and its follow-up (a MBP with the touch bar)'s screen is completely broken (probably just the connector cable).

I don't have a use for it, but it feels wasteful just to throw it away.

Re: A lost decade chasing distributed architectures for data analytics?

#17

Post by HPsquared »

Cthulhu_ wrote: post #16
willvarfar wrote: post #4

I only retired my 2014 MBP ... last week! It started transiently not booting and then, after just a few weeks, it switched to be only transiently booting. Figured it was time. My new laptop is actually a very budget buy, and not a mac, and in many things a bit slower than the old MBP. Anyway, the old laptop is about par with the 'big' VMs that I use for work to analyse really big BQ datasets. My current flow is to do…

I still have mine, but it's languishing, I don't know what to do with it / how to get rid of it, it doesn't feel like trash. The Apple stores do returns but for this one you get nothing, they're just like "yeah we'll take care of it". The screen started to delaminate on the edges, and its follow-up (a MBP with the touch bar)'s screen is completely broken (probably just the connector cable). I don't have a use for it,…

eBay is pretty active for that kind of thing. Spares/repair.

Re: A lost decade chasing distributed architectures for data analytics?

#18

Post by threeseed »

rr808 wrote: post #12

Ugh I have joined a big data team. 99% of the feeds are less than a few GB yet we have to use Scala and Spark. Its so slow to develop and slow to run.

a) Scala being a JVM language is one of the fastest around. Much faster than say Python.

b) How large are the 1% of the feeds and the size of the total joined datasets. Because ultimately that is what you build platforms for. Not the simple use cases.

Re: A lost decade chasing distributed architectures for data analytics?

#19

Post by threeseed »

willvarfar wrote: post #10
zkmon wrote: post #6

A database is not only about disk size and query performance. Database reflects the company's culture, processes, workflows, collaboration etc. It has an entire ecosystem around it - master data, business processes, transactions, distributed applications, regulatory requirements, resiliency, Ops, reports, tooling etc, The role of a database is not just to deliver query performance. It needs to fit into the ecosystem,…

And its very much the tech culture at large that influences the company's tech choices. Those techies chasing shiny things and trying to shoehorn it into their job - perhaps cynically to pad their cvs or perhaps generously thinking it will actually be the right thing to do - have an outsized say in how tech teams think about tech and what they imagine their job is. Back in 2012 we were just recovering from the everyt…

Why you don't just quietly use SQL instead of condescending lecturing others about how compromised their tech choices are.

NoSQL e.g. Cassandra, MongoDB and Microservices were invented to solve real-world problems which is why they are still so heavily used today. And the criticism of them is exactly the same that was levelled at SQL back in the day.

It's all just tools at the end of the day and there isn't one that works for all use cases.

Re: A lost decade chasing distributed architectures for data analytics?

#20

Post by wodenokoto »

Mortiffer wrote: post #13

The R community has been hard at work on small data. I still highly prefer working on on memory data in R dplyr DataTable are elegant and fast. The CRan packages are all high quality if the maintainer stops responding to emails for 2 months your package is automatically removed. Most packages come from university Prof's that have been doing this their whole career.

A really big part of a in-memory dataframe centric workflow is how easy it is to do one step at a time and inspect the result.

With a database it is difficult to run a query, look at the result and then run a query on the result. To me, that is what is missing in replacing pandas/dplyr/polars with DuckDB.

Post reply on HN