Joining a billion rows 20x faster than Apache Spark • Hacker News Board

Re: Joining a billion rows 20x faster than Apache Spark

#31

Post by edw » Thu, Feb 09, 2017, 9:54 PM UTC

quantumhobbit wrote: post #8

banachtarski wrote: post #7

Am I reading this correctly? The testbed was a single laptop? A big part of spark is the distributed in-memory aspect so I'm not sure I understand why any of these numbers mean anything.

This paper is a must read: https://pdfs.semanticscholar.org/6753/959eed800e9fad9e330daa... People keep stumbling upon the same thing over and over which is that the ability to scale has significant overhead.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading.

Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `Sed`, `awk`, `grep`, `sort`, `join`, and so forth are some of the least appreciated tools in the Unix toolbox.

People want to think they have Big Data problems but they probably just have plain old normal-data problems. I have had to unwind the ridiculous, heavy-weight, Big Data solutions to normal-data problems that "kids today" love.

If you don't work for Netflix or Google or Facebook or insert maybe a hundred other companies here, you probably do not have a Big Data problem.

Re: Joining a billion rows 20x faster than Apache Spark

#32

Post by makapuf » Thu, Feb 09, 2017, 9:56 PM UTC

quantumhobbit wrote: post #22

dajohnson89 wrote: post #15

Earlier quoted context omitted.

I see your point -- size alone doesn't matter, it's how you use it :-). But how do things change when the dataset grows to 9GB? Now we need more than one HD. Hadoop + Spark is built for this exact use case...

I think the conundrum comes up from the "donut hole" of medium sized data. For 1TB use a script and a laptop; for 100TB use Spark running on dozens to hundreds of machines. The problem is exactly that 8-9TB range because running spark on just two or three machines will be slower than on a laptop with an extra external drive. You need to scale up into potentially dozens of machines just to get the same performance you…

Processing data wouldn't be the problem with 2 socket xeons neither would it be putting 3 or 5 Hdd on a raid5. Getting the 32TB in, however, would take at least 8 hours at 10Gbps saturated, if your disks can write that fast.

Re: Joining a billion rows 20x faster than Apache Spark

#33

Post by usgroup » Thu, Feb 09, 2017, 9:58 PM UTC

makapuf wrote: post #28

usgroup wrote: post #12

Lol was hoping it was a combination of awk and paste :) That always makes me chuckle. Honestly though ... Jenkins + bash + cloud storage and you'll be surprised at how many big data problems you can solve with a fraction of the complexity.

Pardon my ignorance but what would you use Jenkins for ? Scheduling ?

Triggering jobs more generally (schedule or push notification) and splitting things into jobs and/or pipelines.

Re: Joining a billion rows 20x faster than Apache Spark

#34

Post by tetha » Thu, Feb 09, 2017, 10:05 PM UTC

makapuf wrote: post #28

usgroup wrote: post #12

Lol was hoping it was a combination of awk and paste :) That always makes me chuckle. Honestly though ... Jenkins + bash + cloud storage and you'll be surprised at how many big data problems you can solve with a fraction of the complexity.

Pardon my ignorance but what would you use Jenkins for ? Scheduling ?

Jenkins in such a setting gives you two good things: i) scheduling. and ii) access control. The ability to give random dude X the ability to trigger computation Y, Z, and A, without the ability to change said computations.

Re: Joining a billion rows 20x faster than Apache Spark

#35

Post by makapuf » Thu, Feb 09, 2017, 10:07 PM UTC

edw wrote: post #31

quantumhobbit wrote: post #8

Earlier quoted context omitted.

This paper is a must read: https://pdfs.semanticscholar.org/6753/959eed800e9fad9e330daa... People keep stumbling upon the same thing over and over which is that the ability to scale has significant overhead.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

Amen. Also said as: too big for excel is not big data. See also https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

Re: Joining a billion rows 20x faster than Apache Spark

#36

Post by filereaper » Thu, Feb 09, 2017, 10:12 PM UTC

I apologize in advance, but whenever people claim to use a in-memory big-data system, how exactly does this end up working?

You can only stuff so much into memory, so you can scale up vertically in-terms of memory, unless you buy a massive big-iron POWER box, you scale out horizontally. But with each of these in-memory appliances, what happens when you need to spill out to disk?

In essence why should one bother with these in-memory appliances as opposed to buying boxes with fast SSD's instead? Sure you spill out to disk, but do you take that big of a hit compared to the enormous cost of keeping everything in memory?

Re: Joining a billion rows 20x faster than Apache Spark

#37

Post by alexchamberlain » Thu, Feb 09, 2017, 10:13 PM UTC

Python 2.7 can do it in 0.0867 usec (Intel i7);

    $ python2.7 -m timeit 'n=10**9; (n*n + n) / 2'
    10000000 loops, best of 3: 0.0867 usec per loop

(Admittedly, I killed `n=109; sum(range(1,n+1))`.)

Re: Joining a billion rows 20x faster than Apache Spark

#38

Post by alexchamberlain » Thu, Feb 09, 2017, 10:14 PM UTC

filereaper wrote: post #36

I apologize in advance, but whenever people claim to use a in-memory big-data system, how exactly does this end up working? You can only stuff so much into memory, so you can scale up vertically in-terms of memory, unless you buy a massive big-iron POWER box, you scale out horizontally. But with each of these in-memory appliances, what happens when you need to spill out to disk? In essence why should one bother with…

It's not big data if it fits in memory... This article is demonstrating an architecture that may scale well with big data.

Re: Joining a billion rows 20x faster than Apache Spark

#39

Post by placeybordeaux » Thu, Feb 09, 2017, 10:17 PM UTC

makapuf wrote: post #35

edw wrote: post #31

Earlier quoted context omitted.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

Amen. Also said as: too big for excel is not big data. See also https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

The picture has now gotten a little fuzzier as this blog post conflates map reduce and YARN and calls them both hadoop. The scala pseudo code is just about exactly what you'd use with spark which runs on YARN.

Re: Joining a billion rows 20x faster than Apache Spark

#40

Post by edw » Thu, Feb 09, 2017, 10:21 PM UTC

makapuf wrote: post #35

edw wrote: post #31

Earlier quoted context omitted.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

Amen. Also said as: too big for excel is not big data. See also https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

Amen right back at ya! (I love the O'Reilly book cover.) I highly recommend people read your blog post. And there's also this classic:

https://aadrake.com/command-line-tools-can-be-235x-faster-th...

I'll also take this opportunity to plug Make and Drake for manipulating data in a replicable way:

https://bost.ocks.org/mike/make/

https://github.com/Factual/drake

If you're processing data using tools that cannot trace their ancestry directly to some time before 1985, you're probably wasting your own and your colleagues' time.