Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

121–130 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#121

Post by markus_zhang »

manigandham wrote: post #118
markus_zhang wrote: post #112

Earlier quoted context omitted.

What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. Since I'm not a CTO who worries about not delivering, but a junior DE who actually wants to learn things, I really prefer that things were done in the old ways where we had to manage our own infrastructure and our own code for ETL. These kinds of things can not be learned "just for fun" because one has to work in a real…

What do you mean? Of course you can still learn them "just for fun" if you want. There are plenty of columnar data warehouses (memsql, greenplum, vertica, clickhouse, etc) and data processing frameworks (spark, flink, etc) that you can look at, implement and run yourself. It's all using the same principles underneath.

What I'm saying is that you can surely scratch the basics from personal use, but it's completely different from real usage and such can only be trained on job. Now that those jobs are fewer as everyone goes on cloud.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#122

Post by charcircuit »

rstuart4133 wrote: post #40
charcircuit wrote: post #29

Can someone explain to me why it took 26 minutes for hadoop? How can 7 machines be that many times slower than a single machine?

> How can 7 machines be that many times slower than a single machine? A mere 7 machines is almost certainly going to be slower than 1. There was a recent post here recently titled "Latency Numbers Every Programmer Should Know": https://news.ycombinator.com/item?id=30546995 In this case the dataset was small enough to fix in my laptop's DRAM without straining it. If we assume the 7 machines are in the data centre, tha…

I think you are missing my point. It's a difference between 12 seconds on a laptop and 26 minutes on 7 c1.medium instances. Yes there is overhead in shipping the data around, but there isn't 26 minutes worth of overhead.

If a c1.medium can process at the same speed as his laptop it should take less than 3 seconds worst case. And yes 7 machines should be faster at this scale.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#123

Post by zozbot234 »

chockchocschoir wrote: post #106
athenot wrote: post #100

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutio…

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can…

You could also use SSI (single system image) system architectures, including distributed shared memory, to seamlessly (at least wrt. software implementation) scale up from efficient single-node processing to an arbitrarily large, clustered system. AIUI this is how things are traditionally done in mainframe processing, and it is also regarded as a very meaningful possibility in HPC.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#124

Post by rovr138 »

Cthulhu_ wrote: post #65
rovr138 wrote: post #7

Earlier quoted context omitted.

But if you're going to keep adding destinations and going to them, an airport is a better idea.

Depends if it's just another store within walking distance. I think the gist of the GP comment was that they use a big tool for a small job. Many small jobs don't make the big tool any more useful.

Of course, there are uses.

But if you have many jobs that you have to maintain and run, standardizing the tool and workflow is really helpful.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#125

Post by rovr138 »

mellavora wrote: post #70
rovr138 wrote: post #7

Earlier quoted context omitted.

But if you're going to keep adding destinations and going to them, an airport is a better idea.

might depend on the distance between destinations

Fine. Maybe a bus or taxi service.

But standardizing helps and hadoop and other tools provides that.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#126

Post by pizza234 »

athenot wrote: post #100

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutio…

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice.

In general, this can be an effective approach, but at least fulltext search is another story.

Storing hundreds of MBs (actually, I think even tens of MBs can be problematic) in text files or a db like MySQL (whose FT engine is terrible) will result in slow fulltext searches.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#128

Post by tremon »

toto444 wrote: post #68
JimmyRuska wrote: post #22

If you don't have to do any complex sorting or grouping, yes a simple script works way better. It doesn't have the overhead of using the scheduler, or distributing the data into chunks on many servers. Also consider using sqlite, postgres, or your EDW if you have one. Tools like CSVkit and XSV are useful for preprocessing, exploration. I've seen many ETL scripts written where a simple SQL statement would have been be…

> I've seen many ETL scripts written where a simple SQL statement would have been better. SQL queries tend to work after a few queries have been verified to be correct, ETL jobs in languages like java can dump mysterious stack traces referencing many frameworks breaking due to data issues, memory issues, or unhandled cases. This is so true. I write data pipelines at work. I only use SQL to move data around for this v…

You sometimes need to adjust some queries to some new bad input but never ever will you see : 'index out of bound' or 'wrong argument' or whatever.

Just wait until you encounter the "String or binary data would be truncated" error. I guarantee you it will make you long even for Java stack traces.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#129

Post by commandlinefan »

trashtester wrote: post #79
mr_toad wrote: post #51

Earlier quoted context omitted.

Glue often uses EMR under the hood, which is often Spark. And Athena is PrestoDB, as far as I know it has nothing to do with Hadoop other than you can use it to query Hadoop data stores.

The way I see it, Hadoop is still in common use as the storage layer for Spark and related implementations, whether that is in the form of HDFS or something like EMRFS: Quote from AWS: "EMRFS is an implementation of the Hadoop file system ..." https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...

Ah - that's a good point. Usually when people say "Hadoop", I assume they're referring to HDFS, but there is (was...) the Hadoop MR system that ran on top of it that has been almost entirely replaced by Spark.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#130

Post by louwrentius »

chockchocschoir wrote: post #106
athenot wrote: post #100

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutio…

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can…

I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table.

Even a 10+ year old DL380G8 - can hold 1.5TB+ RAM and 24/48 cores and that hardware is dirt cheap on the second hand market.

Post reply on HN