Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Joining a billion rows 20x faster than Apache Spark

snappydata.io

Post reply on HN
Print view View original

21–30 of 86 posts

Re: Joining a billion rows 20x faster than Apache Spark

#21

Post by plamb »

Bedon292 wrote: post #19

I know its just a benchmark for comparison, and it is awesome. I love seeing cool comparisons like this, but why do I care that this particular benchmark is faster than Spark? What sort of analytics will be affected by this improvement, and will it actually be saving me time on real world use cases?

Our impression was that when Databricks released the billion-rows-in-one-second-on-a-laptop benchmark, readers were pretty awed by that result. We wanted to show that when you combine an in-memory database with Spark so it shares the same JVM/block manager, you can squeeze even more performance out of Spark workloads (over and above Spark 's internal columnar storage). Any analytics that require multiple trips to a database will be impacted by this design. E.g. workloads on a Spark + Cassandra analytics cluster will be significantly slower, barring some fundamental changes to Cassandra.

Re: Joining a billion rows 20x faster than Apache Spark

#22

Post by quantumhobbit »

dajohnson89 wrote: post #15
Loic wrote: post #10

Earlier quoted context omitted.

The question is more, what do you want as result? Suppose you search in your 8TB database of molecules the 1000 molecules most similar to a given one, you have 16 cores, you cut the 8TB in 500GB skunks, preload continuously 1GB of molecules per core and accumulate 16*1000 molecules and merge at the end. You can do it on a single system and you work with a TB size dataset. It means that the size of the dataset is not…

I see your point -- size alone doesn't matter, it's how you use it :-). But how do things change when the dataset grows to 9GB? Now we need more than one HD. Hadoop + Spark is built for this exact use case...

I think the conundrum comes up from the "donut hole" of medium sized data. For 1TB use a script and a laptop; for 100TB use Spark running on dozens to hundreds of machines.

The problem is exactly that 8-9TB range because running spark on just two or three machines will be slower than on a laptop with an extra external drive. You need to scale up into potentially dozens of machines just to get the same performance you were getting on a laptop. You were ok with a laptop, add more data and now you have a not insignificant AWS bill, unless you are ok puttering around on a few machines much more slowly than on the laptop.

There is no middle ground solution, so everyone starts with a overkill solution that scales out of fear of getting stuck on one machine when the dataset grows. But most of these systems never grow enough to need to scale this way. So we are wasting resources running toy clusters on problems that would fit on a laptop.

Maybe I am becoming a cranky old man who yells at clouds, but I miss MPI. It had no frills but it runs with next to no overhead and scales up to super computers with no donut hole in between.

Re: Joining a billion rows 20x faster than Apache Spark

#23

Post by user5994461 »

quantumhobbit wrote: post #22
dajohnson89 wrote: post #15

Earlier quoted context omitted.

I see your point -- size alone doesn't matter, it's how you use it :-). But how do things change when the dataset grows to 9GB? Now we need more than one HD. Hadoop + Spark is built for this exact use case...

I think the conundrum comes up from the "donut hole" of medium sized data. For 1TB use a script and a laptop; for 100TB use Spark running on dozens to hundreds of machines. The problem is exactly that 8-9TB range because running spark on just two or three machines will be slower than on a laptop with an extra external drive. You need to scale up into potentially dozens of machines just to get the same performance you…

If a distributed software running on multiple high end expensive servers cannot beat another solution running on a single laptop with a cheap external hard drives, the issue is not distributed systems, the issue is that that specific software is crap.

Re: Joining a billion rows 20x faster than Apache Spark

#24

Post by user5994461 »

marknadal wrote: post #18
banachtarski wrote: post #13

Earlier quoted context omitted.

The problem is what "baseline" means. For example, a multithreaded program will always run slower than a single threaded one on one thread by definition. It has to do work in order to coordinate the threads. Obviously, this doesn't mean we avoid multithreaded code. In this case, the software being tested was explicitly written to manage the coordination of data on many nodes, so why is the definition of "baseline" a…

Yes, but that is exactly why I think these types of articles and discussions are useful - people who understand what is going on often times assume that so do others, but for many people it all looks like magic. How many people out there (genuine question here) assume the opposite of what you know / that they are ignorant of it? How many people do you think that when they hear "multithreaded" that they associate that…

You guys should realize that this is a commercial company promoting its own product. They're just doing the test that makes them look good.

Re: Joining a billion rows 20x faster than Apache Spark

#25

Post by maaaats »

minimaxir wrote: post #4
plamb wrote: post #3

Earlier quoted context omitted.

The font in the embedded gists or the font on the page?

Likely the font on the page. A web design QA note for all: thin fonts (e.g 300-400 weight) as a body font but work fine on macOS due to better font rendering, but do not work well on Windows.

Is it better on Mac? Whenever I boot into Win10 Im struck by how crisp text looks compared to mac.

Re: Joining a billion rows 20x faster than Apache Spark

#27

Post by Bedon292 »

luckydata wrote: post #20
Bedon292 wrote: post #19

I know its just a benchmark for comparison, and it is awesome. I love seeing cool comparisons like this, but why do I care that this particular benchmark is faster than Spark? What sort of analytics will be affected by this improvement, and will it actually be saving me time on real world use cases?

it's literally impossible to test every possible workload for a tool like Spark so... what's the point of asking? Stand up a testing cluster, run some of your jobs and you'll get the answer.

I am not asking about every workload. I was just curious about an example workload where this benchmark matters.

Re: Joining a billion rows 20x faster than Apache Spark

#28

Post by makapuf »

usgroup wrote: post #12

Lol was hoping it was a combination of awk and paste :) That always makes me chuckle. Honestly though ... Jenkins + bash + cloud storage and you'll be surprised at how many big data problems you can solve with a fraction of the complexity.

Pardon my ignorance but what would you use Jenkins for ? Scheduling ?

Re: Joining a billion rows 20x faster than Apache Spark

#29

Post by quantumhobbit »

user5994461 wrote: post #23
quantumhobbit wrote: post #22

Earlier quoted context omitted.

I think the conundrum comes up from the "donut hole" of medium sized data. For 1TB use a script and a laptop; for 100TB use Spark running on dozens to hundreds of machines. The problem is exactly that 8-9TB range because running spark on just two or three machines will be slower than on a laptop with an extra external drive. You need to scale up into potentially dozens of machines just to get the same performance you…

If a distributed software running on multiple high end expensive servers cannot beat another solution running on a single laptop with a cheap external hard drives, the issue is not distributed systems, the issue is that that specific software is crap.

There will always be some overhead, but yes it seems like some of these frameworks are pretty bloated.

Re: Joining a billion rows 20x faster than Apache Spark

#30

Post by franciscop »

EGreg wrote: post #26

This seems like impressive stats about a relational database technology. But the scrolling on their website doesn't work on mobile. So in grand HN tradition, I left and now tell you all about it here, instead of the main point of their invention :)

It worked for me but the nav of the browser didn't hide, which I recognize as messing around with absolute/fixed positioning and/or overflows. I'd recommend to use media queries to show a simple site on mobile and leave all the fancy stuff they are surely doing in the desktop for the desktop only.

Edit: on a second check, it might have to do with that nav that moves the whole page down.

Post reply on HN