Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

91–100 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#91

Post by belter »

secondcoming wrote: post #53
Foobar8568 wrote: post #28

Earlier quoted context omitted.

A lot of articles I read about snowflake involves data vault which is a massive turn off. And when their tech lead (Kent Graziano) is a prominent figure in the DV bullshit...

What is DV?

Count not knowing what it is, as a blessing. Run if you can.

https://en.wikipedia.org/wiki/Data_vault_modeling

Edit: As the Wikipedia article has no Criticism section I will add some references:

http://kejser.org/the-data-vault-vs-kimball-round-2/

https://timi.eu/blog/data-vaulting-from-a-bad-idea-to-ineffi...

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#92

Post by skocznymroczny »

usrbinbash wrote: post #57
ed_elliott_asc wrote: post #46

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Lo…

> That isn’t true with custom data pipelines written in shell scripts. Why not? Cron can schedule jobs, the documentation for standard shell tools is among the best, almost every developer can handle bash scripts, Logging can be done via syslog.

This is like the classic Dropbox hackernews comment:

> For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.

Sure, you can stitch several services together and it will work for your needs, but for most users there is a benefit to a centrally managed and complete solution.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#93

Post by kaba0 »

ricardobeat wrote: post #45
grenran wrote: post #5

Walking to the store is faster than catching a plane

I assume this is mocking the article as obvious, but it starts from someone using Hadoop to process 1.75GB of data. The default advice is that you should catch a plane to go anywhere, and the author is showing why you should walk.

1.75 GB of data is going to the supermarket down a street with a plane still. Hell, perhaps just to the fridge. It is nowhere near big data when a 10 years old low-end phone can hold it in memory.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#94

Post by morelisp »

wenc wrote: post #2

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.

The article still applies to the things which have replaced Hadoop. The operational, development, and computational overhead of distributing/re-aggregating work remains huge.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#95

Post by ImaCake »

Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of processes in parallel.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#96

Post by ricardobeat »

kaba0 wrote: post #93
ricardobeat wrote: post #45

Earlier quoted context omitted.

I assume this is mocking the article as obvious, but it starts from someone using Hadoop to process 1.75GB of data. The default advice is that you should catch a plane to go anywhere, and the author is showing why you should walk.

1.75 GB of data is going to the supermarket down a street with a plane still. Hell, perhaps just to the fridge. It is nowhere near big data when a 10 years old low-end phone can hold it in memory.

Yes, that's the point I was trying to make, could have phrased it better.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#97

Post by TacticalCoder »

ImaCake wrote: post #95

Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of…

Forking in the background is one way, you can also use GNU "parallel" ("apt-get install parallel" for those on Debian and the likes).

For example I've got all my CD ripped to FLAC files, but my car only takes mp3 or wav... So I did a batch convert of FLAC to mp3, making sure to put all cores at work by piping the output of "find" into "parallel".

Some commands also allows to directly parallelize (like, say, "make -j 16 ..." to build using 16 cores/hyperthreads).

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#98

Post by bsenftner »

riknox wrote: post #81
serial_dev wrote: post #78

Earlier quoted context omitted.

Whether a technology can replace Hadoop in an organization depends on many factors, but some technologies that solve at least in part similar problem are Apache Storm, Spark, Flink, Kafka Streams, and maybe BigQuery? Or, as the original article says, some companies just use some command line tools, shell scripts. It's been a couple of years since I was interested in Data Engineering, so my knowledge on this topic is…

I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience. I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.

I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#99

Post by usrbinbash »

skocznymroczny wrote: post #92
usrbinbash wrote: post #57

Earlier quoted context omitted.

> That isn’t true with custom data pipelines written in shell scripts. Why not? Cron can schedule jobs, the documentation for standard shell tools is among the best, almost every developer can handle bash scripts, Logging can be done via syslog.

This is like the classic Dropbox hackernews comment: > For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software. Sure, you can stitch several services together and it will work for your needs, but for…

> but for most users

Define "most users"

Most users who have to tackle actual big-data problems, meaning analysing things on the order of several TB or more?

Sure, they will absolutely benefit.

But there isn't just big data. There is also little data, where what is analysed is on the order of a few GiB or less, and everything in between.

I am not saying "use shell for everything!" I am saying "the right tool for the right job". A 15t excavator is probably not a good choice if I want to plant a small tree in my backyard, and a gardening shovel will probably not serve me well when I wanna start building a scyscraper.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#100

Post by athenot »

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do.

If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutions first, before entertaining something bigger.

Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice.

Post reply on HN