Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Joining a billion rows 20x faster than Apache Spark

snappydata.io

Post reply on HN
Print view View original

41–50 of 86 posts

Re: Joining a billion rows 20x faster than Apache Spark

#41

Post by edw »

placeybordeaux wrote: post #39
makapuf wrote: post #35

Earlier quoted context omitted.

Amen. Also said as: too big for excel is not big data. See also https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

The picture has now gotten a little fuzzier as this blog post conflates map reduce and YARN and calls them both hadoop. The scala pseudo code is just about exactly what you'd use with spark which runs on YARN.

I think his point is that bloated, over-engineered Big Data systems—whether batch or streaming—are overkill for the vast majority of problems.

Re: Joining a billion rows 20x faster than Apache Spark

#42

Post by blhack »

filereaper wrote: post #36

I apologize in advance, but whenever people claim to use a in-memory big-data system, how exactly does this end up working? You can only stuff so much into memory, so you can scale up vertically in-terms of memory, unless you buy a massive big-iron POWER box, you scale out horizontally. But with each of these in-memory appliances, what happens when you need to spill out to disk? In essence why should one bother with…

Maybe I'm misunderstanding the problem, but why can't you scale out horizontally?

If the problem is that queries or sets of data might have to jump nodes, couldn't the data be designed in such a way where an assumption is made about what sorts of queries will happen at write?

Optimize so that node spanning is rare, eat the cost when it does happen, and let those 1/n queries disappear into the average.

Re: Joining a billion rows 20x faster than Apache Spark

#43

Post by plamb »

franciscop wrote: post #30
EGreg wrote: post #26

This seems like impressive stats about a relational database technology. But the scrolling on their website doesn't work on mobile. So in grand HN tradition, I left and now tell you all about it here, instead of the main point of their invention :)

It worked for me but the nav of the browser didn't hide, which I recognize as messing around with absolute/fixed positioning and/or overflows. I'd recommend to use media queries to show a simple site on mobile and leave all the fancy stuff they are surely doing in the desktop for the desktop only. Edit: on a second check, it might have to do with that nav that moves the whole page down.

Appreciate these comments, the site did not go through much testing before being deployed. Overflowing was modified to eliminate horizontal scroll on mobile but it looks like there were some vertical issues as well. We will get this fixed

Re: Joining a billion rows 20x faster than Apache Spark

#44

Post by stingraycharles »

filereaper wrote: post #36

I apologize in advance, but whenever people claim to use a in-memory big-data system, how exactly does this end up working? You can only stuff so much into memory, so you can scale up vertically in-terms of memory, unless you buy a massive big-iron POWER box, you scale out horizontally. But with each of these in-memory appliances, what happens when you need to spill out to disk? In essence why should one bother with…

I think there are many use cases. Fraud detection, risk analysis in finance, weather simulations, etc. These don't need to spill out to disk and are a perfect use case for these systems.

A friend of mine works for a company that does high speed weather analysis to make predictions for energy brokers, to predict prices of wind / solar energy on the market. They use these kind of systems extensively, because of the speed and volatility of the data. Fascinating stuff.

Re: Joining a billion rows 20x faster than Apache Spark

#45

Post by placeybordeaux »

edw wrote: post #41
placeybordeaux wrote: post #39

Earlier quoted context omitted.

The picture has now gotten a little fuzzier as this blog post conflates map reduce and YARN and calls them both hadoop. The scala pseudo code is just about exactly what you'd use with spark which runs on YARN.

I think his point is that bloated, over-engineered Big Data systems—whether batch or streaming—are overkill for the vast majority of problems.

There are just many points that don't really apply to stuff like spark or tez that runs on YARN:

ex: Hadoop I completely agree with

Mapreduce I do a lot of my processing on sparkSQL and through RDD transformations as opposed to Mapreduce limiting, slow KV style processing.

Re: Joining a billion rows 20x faster than Apache Spark

#46

Post by devonkim »

edw wrote: post #31
quantumhobbit wrote: post #8

Earlier quoted context omitted.

This paper is a must read: https://pdfs.semanticscholar.org/6753/959eed800e9fad9e330daa... People keep stumbling upon the same thing over and over which is that the ability to scale has significant overhead.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

I'm mostly convinced that most companies interested in Big Data stuff are not as interested in the scale of the problem but that they want to create "data lakes" to unite thousands of different forms of data that exist in their organization under a federated, centralized database of some sort. But most of us experienced in either enterprise companies or machine learning is that data quality is the primary problem that almost nobody actually can solve without brute force human eyeballs, which simply won't scale with the amount of data pouring in. So now there's interest in machine learning primarily to try to do that instead of people.

Re: Joining a billion rows 20x faster than Apache Spark

#47

Post by mtanski »

makapuf wrote: post #32
quantumhobbit wrote: post #22

Earlier quoted context omitted.

I think the conundrum comes up from the "donut hole" of medium sized data. For 1TB use a script and a laptop; for 100TB use Spark running on dozens to hundreds of machines. The problem is exactly that 8-9TB range because running spark on just two or three machines will be slower than on a laptop with an extra external drive. You need to scale up into potentially dozens of machines just to get the same performance you…

Processing data wouldn't be the problem with 2 socket xeons neither would it be putting 3 or 5 Hdd on a raid5. Getting the 32TB in, however, would take at least 8 hours at 10Gbps saturated, if your disks can write that fast.

10gigE seams fast, but in reality it's only 1.25GB/s in an ideal case. One enterprise PCIe SSD drive will saturate that. Or 5x of the old style 3.5 inch 7.2k RPM drives (you can fit 12 of these in a dense 1U case).

That why you see 40gigE or 56gigE used in HPC.

Re: Joining a billion rows 20x faster than Apache Spark

#48

Post by mtanski »

maaaats wrote: post #25
minimaxir wrote: post #4

Earlier quoted context omitted.

Likely the font on the page. A web design QA note for all: thin fonts (e.g 300-400 weight) as a body font but work fine on macOS due to better font rendering, but do not work well on Windows.

Is it better on Mac? Whenever I boot into Win10 Im struck by how crisp text looks compared to mac.

Prob the retina display the high PPI makes fonts much easier to read.

Re: Joining a billion rows 20x faster than Apache Spark

#49

Post by hinkley »

edw wrote: post #31
quantumhobbit wrote: post #8

Earlier quoted context omitted.

This paper is a must read: https://pdfs.semanticscholar.org/6753/959eed800e9fad9e330daa... People keep stumbling upon the same thing over and over which is that the ability to scale has significant overhead.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

There used to be a little web site where you'd fill out a little form that asked you how much data you had, and it would provide a list of commercially available hardware that could be bought or configured to handle it on a single machine. I think it would even give you a link to someplace you could order that piece of hardware.

This reminds me of the time, way back when, that a coworker told me about how our customer was filling a rack with a terabyte of hard drives. My eyes bulged a little bit to think of it. Now I chuckle to think that the laptop I had two laptops ago had a terabyte drive in it.

Re: Joining a billion rows 20x faster than Apache Spark

#50

Post by cle »

edw wrote: post #31
quantumhobbit wrote: post #8

Earlier quoted context omitted.

This paper is a must read: https://pdfs.semanticscholar.org/6753/959eed800e9fad9e330daa... People keep stumbling upon the same thing over and over which is that the ability to scale has significant overhead.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

I think this is only half the story.

There are other use cases other than mere size that can necessitate "big data" solutions. E.g. timeliness, resiliency, maintainability...

If you are building production data processing systems that have constraints on data size, latency, resiliency, scheduling, dependency management, etc., you might be better off with a "big data" system. Even if the data could all fit on a beefy box. This was a painful lesson for me to learn.

Post reply on HN