Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Joining a billion rows 20x faster than Apache Spark

snappydata.io

Post reply on HN
Print view View original

71–80 of 86 posts

Re: Joining a billion rows 20x faster than Apache Spark

#71

Post by SapphireSun »

blhack wrote: post #42
filereaper wrote: post #36

I apologize in advance, but whenever people claim to use a in-memory big-data system, how exactly does this end up working? You can only stuff so much into memory, so you can scale up vertically in-terms of memory, unless you buy a massive big-iron POWER box, you scale out horizontally. But with each of these in-memory appliances, what happens when you need to spill out to disk? In essence why should one bother with…

Maybe I'm misunderstanding the problem, but why can't you scale out horizontally? If the problem is that queries or sets of data might have to jump nodes, couldn't the data be designed in such a way where an assumption is made about what sorts of queries will happen at write ? Optimize so that node spanning is rare, eat the cost when it does happen, and let those 1/n queries disappear into the average.

My lab works with multi-terabyte datasets on a regular basis. We have big machines to do machine learning on, but when they're not in used, I can tell you that it's way easier to provision and write a single or multi-threaded script that just loads everything into memory rather than deal with networking and partitioned data.

Imagine the difference between setting up a spark cluster and writing a for loop. For instance, for reasons someone created a 1TB hdf5 file. Luckily, we had a computer with 500GB+ of ram and lots of swap, so instead of having to hack the file apart and figure out how to chunk or parallelize it, we loaded it into memory for a one time batch job and did other useful things in the mean time.

Re: Joining a billion rows 20x faster than Apache Spark

#72

Post by sudsmenon »

FractalNerve wrote: post #58
edw wrote: post #54

Earlier quoted context omitted.

At the risk of sounding cynical, there are also companies out there that want to _appear_ to be interested in all the things you just mentioned, so they'll hire a few people to do their [wave hands] data science, machine learning "thing" and those few people then go down the rabbit hole, untethered from reality. The C-suite people will then have their message they can deliver externally—and internally—about their com…

You really just described me. I've to do such a show-case project to complete my MSc Thesis for a minimum pay. And make all the proofs, so that the C-Suite guys can use during their presentations to sell what I made for a lots of money. (Even unaware if they'll charge their customers in the hundred thausands or millions range). However, I'm really glad to find out about SnappyData.io, that's gonna save me a lot of ti…

SnappyData has a Zeppellin interpreter and the code is open source. So if adding a interpreter for jupyter is something that can be easily added, I am sure someone from the community would find it a interesting project to undertake. Agree that it would be useful

Re: Joining a billion rows 20x faster than Apache Spark

#73

Post by brangalina4eva »

emmanuel_1234 wrote: post #69
edw wrote: post #54

Earlier quoted context omitted.

At the risk of sounding cynical, there are also companies out there that want to _appear_ to be interested in all the things you just mentioned, so they'll hire a few people to do their [wave hands] data science, machine learning "thing" and those few people then go down the rabbit hole, untethered from reality. The C-suite people will then have their message they can deliver externally—and internally—about their com…

Woah, that was exactly my previous position. Upside: I had one of the most paying position among the technical people. I also got to play with expensive stuff. Downside: it was soul crushing, I was delivering no value whatsoever and had a really hard time looking at my colleagues in the eye, as they were making a third of my salary (at best). I got out, joined a new company with that in mind and now have a very excit…

Did I write this in my sleep? At least we process a few gigs a day that no one uses.

Stay in a job that pays stupidly well but crushes your soul or get out..?

Re: Joining a billion rows 20x faster than Apache Spark

#74

Post by sudsmenon »

ganeshkrishnan wrote: post #65
edw wrote: post #31

Earlier quoted context omitted.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

>If you don't work for Netflix or Google or Facebook or insert maybe a hundred other companies here, you probably do not have a Big Data problem. I disagree over here. I have worked across multiple scenarios which warranted big data solutions and such solutions were not feasible before Apache Spark and such were available. Even our current startup (www.aihello.com) has 8.7 million products and calculating LDA + Cosin…

Being able to do things like churn prediction and net promoter score in real time was one of the motivations for creating SnappyData. You get the ability to mutate data (think KPI maintenance in memory without having to jump across products) , and do joins etc. on streams, which makes things a lot simpler

Re: Joining a billion rows 20x faster than Apache Spark

#75

Post by errantblaze »

bagels wrote: post #63
brians wrote: post #60

Earlier quoted context omitted.

Hm. Size is mere size. Latency will never improve with a "big data" solution over one machine with in-RAM data. Dependency management? You're going to declare it once and impose it everywhere anyway. Scheduling? Again, one machine with in-RAM data will always win. That leaves resiliency and etc. I can't answer etc., but—how is resilience helped with a big data solution? That seems like Lampson's distributed system: m…

Is it even possible to get even 10TB of ram on a single commodity server?

The Super Micro SuperServer 8048B-TR4FT lists that it supports up to 12TB DDR4 ECC RAM (which could have 4xE7-8890v4 for 96 cores / 192 threads). Close to a commodity server, but probably doesn't quite count. Taking a wild guess on the price - $250k-$350k?

The SuperServer 7088B-TR4FT lists that it supports 24TB DDR4 ECC RAM (with 8xE78890v4 for 192 cores / 384 threads).

Re: Joining a billion rows 20x faster than Apache Spark

#76

Post by chx »

edw wrote: post #31
quantumhobbit wrote: post #8

Earlier quoted context omitted.

This paper is a must read: https://pdfs.semanticscholar.org/6753/959eed800e9fad9e330daa... People keep stumbling upon the same thing over and over which is that the ability to scale has significant overhead.

Back in '10, I needed a three or four node Hadoop cluster just to match the performance I was getting using a spare Mac mini in development mode when I was doing a lot of work in Cascalog, which is based on Cascading. Most problems are not Big Data problems. The size a problem must be before it qualifies as a Big-Data problem grows larger every day with the availability of machines with ever-more cores and memory. `S…

https://twitter.com/garybernhardt/status/600783770925420546

> Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.

Considering https://www.supermicro.com/products/system/4U/8048/SYS-8048B... which is a plain old 4U server not some fancy, super expensive NUMA machine can eat up 12TB memory, this quip and parent has quite some merits.

6TB is not even https://memory.net/product/s26361-f3843-e618-fujitsu-1x-64gb... horrible at 57 504 dollars. That's about 48 engineering days if your engineer related expenses are 150 an hour (and it's likely they are more).

Re: Joining a billion rows 20x faster than Apache Spark

#77

Post by makapuf »

edw wrote: post #40
makapuf wrote: post #35

Earlier quoted context omitted.

Amen. Also said as: too big for excel is not big data. See also https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

Amen right back at ya! (I love the O'Reilly book cover.) I highly recommend people read your blog post. And there's also this classic: https://aadrake.com/command-line-tools-can-be-235x-faster-th... I'll also take this opportunity to plug Make and Drake for manipulating data in a replicable way: https://bost.ocks.org/mike/make/ https://github.com/Factual/drake If you're processing data using tools that cannot trace t…

Just for clarification, I'm not the original blogger. +10 for the other link and using make ! I don't know Drake however.

Re: Joining a billion rows 20x faster than Apache Spark

#78

Post by Loic »

jagsr123 wrote: post #17
Loic wrote: post #5

What is the algorithm used to join the tables? Is it a hash join on `id` and `k` or using the fact that the ids are sorted and using a kind of galloping approach?

Yes, it is a hash join.

I will need to dig into the implementation of the hash function, it must be a nice read as the speed shows that it is definitely well optimized! Thank you.

Re: Joining a billion rows 20x faster than Apache Spark

#79

Post by pmlnr »

usgroup wrote: post #12

Lol was hoping it was a combination of awk and paste :) That always makes me chuckle. Honestly though ... Jenkins + bash + cloud storage and you'll be surprised at how many big data problems you can solve with a fraction of the complexity.

https://aadrake.com/command-line-tools-can-be-235x-faster-th...

Discussion is here: https://news.ycombinator.com/item?id=8908462

Re: Joining a billion rows 20x faster than Apache Spark

#80

Post by dandermotj »

edw wrote: post #54
devonkim wrote: post #46

Earlier quoted context omitted.

I'm mostly convinced that most companies interested in Big Data stuff are not as interested in the scale of the problem but that they want to create "data lakes" to unite thousands of different forms of data that exist in their organization under a federated, centralized database of some sort. But most of us experienced in either enterprise companies or machine learning is that data quality is the primary problem tha…

At the risk of sounding cynical, there are also companies out there that want to _appear_ to be interested in all the things you just mentioned, so they'll hire a few people to do their [wave hands] data science, machine learning "thing" and those few people then go down the rabbit hole, untethered from reality. The C-suite people will then have their message they can deliver externally—and internally—about their com…

This is excruciatingly accurate.
Post reply on HN