Joining a billion rows 20x faster than Apache Spark • Hacker News Board

Re: Joining a billion rows 20x faster than Apache Spark

#51

Post by rodionos » Thu, Feb 09, 2017, 11:42 PM UTC

stingraycharles wrote: post #44

filereaper wrote: post #36

I apologize in advance, but whenever people claim to use a in-memory big-data system, how exactly does this end up working? You can only stuff so much into memory, so you can scale up vertically in-terms of memory, unless you buy a massive big-iron POWER box, you scale out horizontally. But with each of these in-memory appliances, what happens when you need to spill out to disk? In essence why should one bother with…

I think there are many use cases. Fraud detection, risk analysis in finance, weather simulations, etc. These don't need to spill out to disk and are a perfect use case for these systems. A friend of mine works for a company that does high speed weather analysis to make predictions for energy brokers, to predict prices of wind / solar energy on the market. They use these kind of systems extensively, because of the spe…

You can also measure cloud oktas from satellite imagery of you want to get fancy in terms of solar energy supply side forecasting: https://axibase.com/calculating-cloud-oktas/

Re: Joining a billion rows 20x faster than Apache Spark

#52

Post by olavgg » Thu, Feb 09, 2017, 11:49 PM UTC

mtanski wrote: post #47

makapuf wrote: post #32

Earlier quoted context omitted.

Processing data wouldn't be the problem with 2 socket xeons neither would it be putting 3 or 5 Hdd on a raid5. Getting the 32TB in, however, would take at least 8 hours at 10Gbps saturated, if your disks can write that fast.

10gigE seams fast, but in reality it's only 1.25GB/s in an ideal case. One enterprise PCIe SSD drive will saturate that. Or 5x of the old style 3.5 inch 7.2k RPM drives (you can fit 12 of these in a dense 1U case). That why you see 40gigE or 56gigE used in HPC.

Its 25G, 50G or 100G today.

Re: Joining a billion rows 20x faster than Apache Spark

#53

Post by jy2947 » Thu, Feb 09, 2017, 11:56 PM UTC

devonkim wrote: post #46

edw wrote: post #31

Earlier quoted context omitted.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

I'm mostly convinced that most companies interested in Big Data stuff are not as interested in the scale of the problem but that they want to create "data lakes" to unite thousands of different forms of data that exist in their organization under a federated, centralized database of some sort. But most of us experienced in either enterprise companies or machine learning is that data quality is the primary problem tha…

+1

Re: Joining a billion rows 20x faster than Apache Spark

#54

Post by edw » Fri, Feb 10, 2017, 12:04 AM UTC

devonkim wrote: post #46

edw wrote: post #31

Earlier quoted context omitted.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

I'm mostly convinced that most companies interested in Big Data stuff are not as interested in the scale of the problem but that they want to create "data lakes" to unite thousands of different forms of data that exist in their organization under a federated, centralized database of some sort. But most of us experienced in either enterprise companies or machine learning is that data quality is the primary problem tha…

At the risk of sounding cynical, there are also companies out there that want to _appear_ to be interested in all the things you just mentioned, so they'll hire a few people to do their [wave hands] data science, machine learning "thing" and those few people then go down the rabbit hole, untethered from reality. The C-suite people will then have their message they can deliver externally—and internally—about their commitment to [wave hands again] all that stuff. The tragedy when this happens is that sometimes that small team of people is not aware what their function at the company is—being the human props for the sales and marketing teams. The alternative, I suppose, is far more depressing—that the "data scientists" are fully aware of their role in the grand scheme of things.

Re: Joining a billion rows 20x faster than Apache Spark

#55

Post by nickpsecurity » Fri, Feb 10, 2017, 12:41 AM UTC

makapuf wrote: post #35

edw wrote: post #31

Earlier quoted context omitted.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

Amen. Also said as: too big for excel is not big data. See also https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

Thanks for the link. The replies in it were hilariously obvious.

Re: Joining a billion rows 20x faster than Apache Spark

#56

Post by sumwale » Fri, Feb 10, 2017, 12:43 AM UTC

banachtarski wrote: post #7

Am I reading this correctly? The testbed was a single laptop? A big part of spark is the distributed in-memory aspect so I'm not sure I understand why any of these numbers mean anything.

The test uses 4 partitions (4 threads) and is not single-threaded. Of course, distributing over network will have network costs which can be significant for sub-second queries but will become insignificant for larger data and more involved queries. The Spark execution plan will be exactly same on laptop or in cluster, and these specific queries will scale about linearly.

Re: Joining a billion rows 20x faster than Apache Spark

#57

Post by jupiter90000 » Fri, Feb 10, 2017, 12:57 AM UTC

banachtarski wrote: post #7

Am I reading this correctly? The testbed was a single laptop? A big part of spark is the distributed in-memory aspect so I'm not sure I understand why any of these numbers mean anything.

Different example, doing a simple 'group by' sparksql query on only about 20 million rows on a distributed phoenix/hbase table couldn't even be completed because of spark dumbly shuffling all the data around the cluster. Spark/phoenix RDD drivers apparently had no 'group by' push down support for phoenix so shuffled all the data amazingly inefficiently. Running the same query directly on phoenix took all of about a minute to finish.

My point is, these 'on a laptop/single machine memory' examples don't really give me an indicator of scenarios where I might actually want to use spark/etc.

Re: Joining a billion rows 20x faster than Apache Spark

#58

Post by FractalNerve » Fri, Feb 10, 2017, 1:43 AM UTC

edw wrote: post #54

devonkim wrote: post #46

Earlier quoted context omitted.

I'm mostly convinced that most companies interested in Big Data stuff are not as interested in the scale of the problem but that they want to create "data lakes" to unite thousands of different forms of data that exist in their organization under a federated, centralized database of some sort. But most of us experienced in either enterprise companies or machine learning is that data quality is the primary problem tha…

At the risk of sounding cynical, there are also companies out there that want to _appear_ to be interested in all the things you just mentioned, so they'll hire a few people to do their [wave hands] data science, machine learning "thing" and those few people then go down the rabbit hole, untethered from reality. The C-suite people will then have their message they can deliver externally—and internally—about their com…

You really just described me. I've to do such a show-case project to complete my MSc Thesis for a minimum pay. And make all the proofs, so that the C-Suite guys can use during their presentations to sell what I made for a lots of money. (Even unaware if they'll charge their customers in the hundred thausands or millions range).

However, I'm really glad to find out about SnappyData.io, that's gonna save me a lot of time waiting. It would truly be my perfect dream, if they allowed running any programming language inside an environment like Jupyter.org or BeakerNotebook.com, but with Pandoc.org Markdown. So that I can essentially work fulltime programming, while I can also document it and also be able to export my documentation to a good looking latex thesis.

Re: Joining a billion rows 20x faster than Apache Spark

#59

Post by jagsr123 » Fri, Feb 10, 2017, 2:03 AM UTC

Bedon292 wrote: post #19

I know its just a benchmark for comparison, and it is awesome. I love seeing cool comparisons like this, but why do I care that this particular benchmark is faster than Spark? What sort of analytics will be affected by this improvement, and will it actually be saving me time on real world use cases?

Good questions. One answer - Speed in analytics when working with large data sets matters. A lot. Think about this - several vendors seem to be claiming support for interactive analytics. i.e. i can ask an adhoc question on large volume of data and get some sort of insight in "interactive" times. Really? maybe with a thousand cores? In a competitive industry, say like in investment banking, if one can discern a pattern before the competition it simply provides an edge. Ask the question to folks involved in detecting fraud, or online portals trying to place an appropriate ad, or manufacturing plant trying to anticipate/predict faults. It isn't so much about trying to prove we are better than Spark (well other than grabbing some attention :-) ) but rather the potential to live in a world where batch processing is a thing of the past - like working with mainframes. The hope is that we can gain insight instantly. fwiw, we love Spark and expect some of these optimizations simply become part of Apache Spark.

Re: Joining a billion rows 20x faster than Apache Spark

#60

Post by brians » Fri, Feb 10, 2017, 2:14 AM UTC

cle wrote: post #50

edw wrote: post #31

Earlier quoted context omitted.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

I think this is only half the story. There are other use cases other than mere size that can necessitate "big data" solutions. E.g. timeliness, resiliency, maintainability... If you are building production data processing systems that have constraints on data size, latency, resiliency, scheduling, dependency management, etc., you might be better off with a "big data" system. Even if the data could all fit on a beefy…

Hm. Size is mere size. Latency will never improve with a "big data" solution over one machine with in-RAM data. Dependency management? You're going to declare it once and impose it everywhere anyway. Scheduling? Again, one machine with in-RAM data will always win.

That leaves resiliency and etc. I can't answer etc., but—how is resilience helped with a big data solution? That seems like Lampson's distributed system: more machines, but you need k-of-n, k>1. Better to just mirror to two machines with the data in RAM.