Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A lost decade chasing distributed architectures for data analytics?

duckdb.org

Post reply on HN
Print view View original

21–30 of 117 posts

Re: A lost decade chasing distributed architectures for data analytics?

#21

Post by IanCal »

wodenokoto wrote: post #20
Mortiffer wrote: post #13

The R community has been hard at work on small data. I still highly prefer working on on memory data in R dplyr DataTable are elegant and fast. The CRan packages are all high quality if the maintainer stops responding to emails for 2 months your package is automatically removed. Most packages come from university Prof's that have been doing this their whole career.

A really big part of a in-memory dataframe centric workflow is how easy it is to do one step at a time and inspect the result. With a database it is difficult to run a query, look at the result and then run a query on the result. To me, that is what is missing in replacing pandas/dplyr/polars with DuckDB.

I'm not sure I really follow, you can create new tables for any step if you want to do it entirely within the db, but you can also just run duckdb against your dataframes in memory.

Re: A lost decade chasing distributed architectures for data analytics?

#22

Post by rr808 »

threeseed wrote: post #18
rr808 wrote: post #12

Ugh I have joined a big data team. 99% of the feeds are less than a few GB yet we have to use Scala and Spark. Its so slow to develop and slow to run.

a) Scala being a JVM language is one of the fastest around. Much faster than say Python. b) How large are the 1% of the feeds and the size of the total joined datasets. Because ultimately that is what you build platforms for. Not the simple use cases.

1) Yes Scala and JVM is fast. If we could just use that to clean up a feed on a single box that would be great. The problem is calling the Spark API creates a lot of complexity for developers and runtime platform which is super slow. 2) Yes for the few feeds that are a TB we need spark. The platform really just loads from hadoop transforms then saves back again.

Re: A lost decade chasing distributed architectures for data analytics?

#23

Post by kukkeliskuu »

threeseed wrote: post #19
willvarfar wrote: post #10

Earlier quoted context omitted.

And its very much the tech culture at large that influences the company's tech choices. Those techies chasing shiny things and trying to shoehorn it into their job - perhaps cynically to pad their cvs or perhaps generously thinking it will actually be the right thing to do - have an outsized say in how tech teams think about tech and what they imagine their job is. Back in 2012 we were just recovering from the everyt…

Why you don't just quietly use SQL instead of condescending lecturing others about how compromised their tech choices are. NoSQL e.g. Cassandra, MongoDB and Microservices were invented to solve real-world problems which is why they are still so heavily used today. And the criticism of them is exactly the same that was levelled at SQL back in the day. It's all just tools at the end of the day and there isn't one that…

Around 20 years ago I was working for a database company. During that time, I attended SIGMOD, which is the top conference for databases.

The keynote speaker for the conference Stonebraker, who started Postgres, among other things. He talked about the history of relational databases.

At that time, XML databases were all the rage -- now nobody remembers them. Stonebraker explained that there is nothing new in the hierarchical databases. There was a significant battle in SIGMOD, I think somewhere in the 1980s (I forget the exact time frame) between network databases and relational databases.

The relational databases won that battle, as they have won against each competing hierarchical database technology since.

The reason is that relational databases are based on relational algebra. This has very practical consequences, for example you can query the data more flexibly.

When you use JSON storage such as MongoDB, when you decide your root entities you are stuck with that decision. I see very often in practice that there will always come new requirements that you did not foresee that you then need to work around.

I don't care what other people use, however.

Re: A lost decade chasing distributed architectures for data analytics?

#25

Post by yorwba »

willvarfar wrote: post #9
querez wrote: post #7

> The geometric mean of the timings improved from 218 to 12, a ca. 20× improvement. Why do they use the geometric mean to average execution times?

Squaring is a really good way to make the common-but-small numbers have bigger representation than the outlying-but-large numbers. I just did a quick google and first real result was this blog post with a good explanation with some good illustrations https://jlmc.medium.com/understanding-three-simple-statistic... Its the very first illustration at the top of that blog post that 'clicks' for me. Hope it helps! The inv…

The geometric mean of n numbers is the n-th root of the product of all numbers. The mean square error is the sum of the squares of all numbers, divided by n. (I.e. the arithmetic mean of the squares.) They're not the same.

Re: A lost decade chasing distributed architectures for data analytics?

#26

Post by threeseed »

kukkeliskuu wrote: post #23
threeseed wrote: post #19

Earlier quoted context omitted.

Why you don't just quietly use SQL instead of condescending lecturing others about how compromised their tech choices are. NoSQL e.g. Cassandra, MongoDB and Microservices were invented to solve real-world problems which is why they are still so heavily used today. And the criticism of them is exactly the same that was levelled at SQL back in the day. It's all just tools at the end of the day and there isn't one that…

Around 20 years ago I was working for a database company. During that time, I attended SIGMOD, which is the top conference for databases. The keynote speaker for the conference Stonebraker, who started Postgres, among other things. He talked about the history of relational databases. At that time, XML databases were all the rage -- now nobody remembers them. Stonebraker explained that there is nothing new in the hier…

MongoDB is a $2b/year revenue company growing at 20% y/y. JSON stores are not going anywhere and it's an essential tool for dealing in data where you have no control over the schema or you want to do it in the application layer.

And the only "battle" is one you've invented in your head. People who deal in data for a living just pick the right data store for the right data schema.

Re: A lost decade chasing distributed architectures for data analytics?

#27

Post by willvarfar »

yorwba wrote: post #25
willvarfar wrote: post #9

Earlier quoted context omitted.

Squaring is a really good way to make the common-but-small numbers have bigger representation than the outlying-but-large numbers. I just did a quick google and first real result was this blog post with a good explanation with some good illustrations https://jlmc.medium.com/understanding-three-simple-statistic... Its the very first illustration at the top of that blog post that 'clicks' for me. Hope it helps! The inv…

The geometric mean of n numbers is the n -th root of the product of all numbers. The mean square error is the sum of the squares of all numbers, divided by n . (I.e. the arithmetic mean of the squares.) They're not the same.

I'm not gonna edit what I wrote but you are interpreting it too way too literally. I was not describing the implementation of anything, I was just giving a link that explains why thinking about things in terms of area (geometry) is popular in stats. Its a bit like the epiphany that histograms don't need to be bars of equal width.

Re: A lost decade chasing distributed architectures for data analytics?

#28

Post by threeseed »

rr808 wrote: post #22
threeseed wrote: post #18

Earlier quoted context omitted.

a) Scala being a JVM language is one of the fastest around. Much faster than say Python. b) How large are the 1% of the feeds and the size of the total joined datasets. Because ultimately that is what you build platforms for. Not the simple use cases.

1) Yes Scala and JVM is fast. If we could just use that to clean up a feed on a single box that would be great. The problem is calling the Spark API creates a lot of complexity for developers and runtime platform which is super slow. 2) Yes for the few feeds that are a TB we need spark. The platform really just loads from hadoop transforms then saves back again.

a) You can easily run Spark jobs on a single box. Just set executors = 1.

b) The reason centralised clusters exist is because you can't have dozens/hundreds of data engineers/scientists all copying company data onto their laptop, causing support headaches because they can't install X library and making productionising impossible. There are bigger concerns than your personal productivity.

Re: A lost decade chasing distributed architectures for data analytics?

#29

Post by rr808 »

threeseed wrote: post #28
rr808 wrote: post #22

Earlier quoted context omitted.

1) Yes Scala and JVM is fast. If we could just use that to clean up a feed on a single box that would be great. The problem is calling the Spark API creates a lot of complexity for developers and runtime platform which is super slow. 2) Yes for the few feeds that are a TB we need spark. The platform really just loads from hadoop transforms then saves back again.

a) You can easily run Spark jobs on a single box. Just set executors = 1. b) The reason centralised clusters exist is because you can't have dozens/hundreds of data engineers/scientists all copying company data onto their laptop, causing support headaches because they can't install X library and making productionising impossible. There are bigger concerns than your personal productivity.

> a) You can easily run Spark jobs on a single box. Just set executors = 1.

Sure but why would you do this? Just using pandas or duckdb or even bash scripts makes your life is much easier than having to deal with Spark.

Re: A lost decade chasing distributed architectures for data analytics?

#30

Post by lazide »

threeseed wrote: post #26
kukkeliskuu wrote: post #23

Earlier quoted context omitted.

Around 20 years ago I was working for a database company. During that time, I attended SIGMOD, which is the top conference for databases. The keynote speaker for the conference Stonebraker, who started Postgres, among other things. He talked about the history of relational databases. At that time, XML databases were all the rage -- now nobody remembers them. Stonebraker explained that there is nothing new in the hier…

MongoDB is a $2b/year revenue company growing at 20% y/y. JSON stores are not going anywhere and it's an essential tool for dealing in data where you have no control over the schema or you want to do it in the application layer. And the only "battle" is one you've invented in your head. People who deal in data for a living just pick the right data store for the right data schema.

Sensitive much?
Post reply on HN