Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

151–160 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#151

Post by AitchEmArsey »

treeman79 wrote: post #84

25 years ago my company Needed to match lists against a collection numbers not to call. Probably ran lists of 20k daily or more. So I wrote up a simple python program to match against a special format txt file. A few seconds for big lists at most. Higher up IT people were horrified, and commissioned a proper oracle soliton. It ran around a 1-2 records per second. So all day for a typical list. They were quite proud o…

> proper oracle soliton

I know this was a typo, but the idea of Oracle upselling someone on a quantum field theory based solution to a string matching problem amuses me greatly.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#152

Post by rstuart4133 »

charcircuit wrote: post #122
rstuart4133 wrote: post #40

Earlier quoted context omitted.

> How can 7 machines be that many times slower than a single machine? A mere 7 machines is almost certainly going to be slower than 1. There was a recent post here recently titled "Latency Numbers Every Programmer Should Know": https://news.ycombinator.com/item?id=30546995 In this case the dataset was small enough to fix in my laptop's DRAM without straining it. If we assume the 7 machines are in the data centre, tha…

I think you are missing my point. It's a difference between 12 seconds on a laptop and 26 minutes on 7 c1.medium instances. Yes there is overhead in shipping the data around, but there isn't 26 minutes worth of overhead. If a c1.medium can process at the same speed as his laptop it should take less than 3 seconds worst case. And yes 7 machines should be faster at this scale.

> If a c1.medium can process at the same speed as his laptop

As it happens, I got to perform that experiment. Sort of. I was moving stuff on physical Dell hardware to a virtualised environment, at the behest of MBA's. I was a bit concerned about it as we pushed the existing hardware hard - it had overnight stuff it had to get finished by morning. It had a 30% buffer.

It wasn't even close. The "virtual environment" was 15 times slower. It wasn't the CPU. It was slower but not by much and that could have been remedied by firing up more VM's on different physical machines. It was disc storage. They insisted their big beefy SAN they spent 100's of K on would be faster than a locally connected SSD. But a SAN operates over a network, and SSD's over 150mm gigabit links. The I/O was mostly random.

If they had of consulted "Latency Numbers Every Programmer Should Know" the could have predicted the outcome - just as I had.

BTW, I also benchmarked my the laptop. Despite what you apparently think it isn't much slower at doing a single task than the big Dell server, and I wouldn't expect it to be. The one difference is the Dell server comes with a factor of 10 more RAM, SSD, HDD and networking, and it seems to be able to drive all those devices at full speed in parallel. But that would not of helped in this instance as the task is too small. I know from experience for this sort of task my laptop will easily outperform a c1.medium.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#153

Post by tharne »

stingraycharles wrote: post #66
IanCal wrote: post #62

Earlier quoted context omitted.

The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.

I’d argue that for as long as Excel doesn’t support Parquet files, we haven’t seen the last of CSVs for a long time. Parquet is great, but it’s simply nowhere near as ubiquitous as CSV.

> I’d argue that for as long as Excel doesn’t support Parquet files, we haven’t seen the last of CSVs for a long time.

It's easy to forget just how much analytic "stuff" Excel still powers.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#154

Post by tharne »

markus_zhang wrote: post #112
bpodgursky wrote: post #6

Earlier quoted context omitted.

I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.

What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. Since I'm not a CTO who worries about not delivering, but a junior DE who actually wants to learn things, I really prefer that things were done in the old ways where we had to manage our own infrastructure and our own code for ETL. These kinds of things can not be learned "just for fun" because one has to work in a real…

> What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me.

This is by design. The less skill required to use a tool, the less some CTO has to pay the person using it.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#155

Post by burntsushi »

chakkepolja wrote: post #147
burntsushi wrote: post #120

Earlier quoted context omitted.

Do note that unlike GNU (or BSD) grep, ripgrep will automatically use multiple threads to execute a search on a directory.

Is there a reason classic grep or find utilities do not use multithreading? It takes 5-6 seconds to search for some files on my machine. Would'nt it be an easy win if `find` had spawned 4 threads?

I don't know. I'm not a maintainer of those projects. There's no obvious reason to me other than that it's hard to adapt a legacy code base that assumes a single thread to something that uses multiple threads. IIRC, there have been multiple attempts to make GNU grep multi-threaded. I believe folks have even submitted patches to the project. I don't follow it closely enough to know why they haven't been accepted.

Sometimes the answer to "why don't they do this" is just uninteresting. :-)

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#156

Post by ufmace »

I did a vaguely similar thing once. We once upon a time had a EMR / Hadoop based process that simply joined a some thousands of files of a few kb-ish from S3 each into one big file totaling 20GB or so. It worked, but took a while. After a bit of mucking with it, I noticed that the built-in tools for it used the wrong S3 API for the purpose. I wrote a simple dumb single-threaded Ruby script using the right API, and it ran like 4x faster IIRC.

Same lesson - before you reach for Big Data tools, make sure you've fully explored simple conventional solutions.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#157

Post by fivea »

chockchocschoir wrote: post #106
athenot wrote: post #100

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutio…

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can…

> Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data.

Doesn't that kind of system cost millions of dollars?

Hadoop and the like are popular because they run on COTS hardware that costs peanuts. It hardly makes any sense to argue that spending over a million dollars is enough to drop a solution that's primarily adopted because you do not have that kind of money to throw at a problem.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#158

Post by fivea »

louwrentius wrote: post #130
chockchocschoir wrote: post #106

Earlier quoted context omitted.

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can…

I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table. Even a 10+ year old DL380G8 - can hold 1.5TB+ RAM and 24/48 cores and that hardware is dirt cheap on the second hand market.

> I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table.

If all a server does is serve mostly static content from memory then it's not expected to handle a demanding workload beyond networking.

Therefore I fail to see the point of trying to downplay someone else's needs for TB of RAM just because you serve a site on a raspberry pi, as if that's the full extent of what anyone needs to do with a computer, particularly when the OP mentions big data.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#159

Post by BasDirks »

ed_elliott_asc wrote: post #46

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Lo…

I don't see why this is downvoted. Seems like a solid argument, even though I don't agree.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#160

Post by NavinF »

awild wrote: post #148
slavik81 wrote: post #141

Earlier quoted context omitted.

As someone in the process of setting up a pipeline and currently using pandas.to_csv as my output, I'm curious what makes you recommend parquet in particular? How does it compare to HDF or Feather?

Just a heads up, pandas to Csv is laughably slow. You're literally better off converting the pandas df to arrow and then serialise to Csv with arrow, though the format is quite limited with arrow.

Well at least it’s better than numpy’s csv functions. For my use case (~4 million images, one row per image, 2048 fp32 columns for features extracted with resnet), parsing/encoding csv with pandas was 10x faster than with numpy. Of course serializing the arrays to/from npy was 10x faster than any csv library.
Post reply on HN