Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

101–110 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#101

Post by Thaxll »

chakkepolja wrote: post #72
Cthulhu_ wrote: post #64

Earlier quoted context omitted.

Things look to be quite cyclic; ten years ago, NoSQL was pushed as The Future, but these days companies still build their core data on top of a traditional relational database. I don't even know which NoSQL is even popular. It feels like it's been pushed to the fringes, e.g. key/value stores, or the Q in a CQRS setup where it acts more as a readonly cache.

MongoDB is pretty popular among newbie nodejs developers. (That's not a complement, btw).

"newbie", maybe get a look outside, MongoDB is used by very large and successful companies. Fortnite which makes multiple $B a year is heavily using MongoDB.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#102

Post by happymellon »

bsenftner wrote: post #98
riknox wrote: post #81

Earlier quoted context omitted.

I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience. I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.

I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?

Big data is a shifting concept as computers gain more storage and faster commodity processors.

My general rule of thumb is whether it is too big to put on my laptop. So greater than a couple of Tb's.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#103

Post by rurban »

auggierose wrote: post #69
wenc wrote: post #2

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.

So what is used instead of Hadoop currently?

GNU parallel. not moreutils parallel, and not xargs

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#105

Post by jsjohnst »

gehen88 wrote: post #80
stingraycharles wrote: post #66

Earlier quoted context omitted.

I’d argue that for as long as Excel doesn’t support Parquet files, we haven’t seen the last of CSVs for a long time. Parquet is great, but it’s simply nowhere near as ubiquitous as CSV.

I hadn't even heard of Parquet until now, and I'm sure this goes for lots of developers who don't do much data engineering.

https://xkcd.com/1053/

What’s the Parquet equivalent of going to the store to buy Mentos and Diet Coke now?

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#106

Post by chockchocschoir »

athenot wrote: post #100

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutio…

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice.

And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be!

On that note, there is this great website for seeing if there are servers that can handle fitting all your data in RAM: https://yourdatafitsinram.net/

Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Just like parent said here, do try to fit things on machines like this before even trying out large-scale storage clusters, because they are a hassle to deal with and generally not worth the cost.

Edit: For the Cloud-hosting crowd out there, I took a quick look at https://instances.vantage.sh/ and sorted it by memory. Largest instances you can get at AWS is "U-24TB1 Metal", which comes with 24TiB of RAM (and 448 vCPU so the computation itself gets as fast as possible too [granted you can parallelize it]), which should fit most cases of "big data" I've seen in the wild today. Unsure about running costs though, but I'm 90% sure it's cheaper to have that instance for a couple of hours than the cost for having to deal with storage clusters, which are also slower.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#107

Post by spaniard89277 »

renonce wrote: post #71

I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get something like MySQL, InfluxDB, TimescaleDB etc, but all of them are way too powerful and have their own query language and storage engine etc which are hard to learn and manipulate. In case you run out of storage/memory/CPU etc or when you want to query somet…

Mmm maybe DuckDB?

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#108

Post by chockchocschoir »

TacticalCoder wrote: post #97
ImaCake wrote: post #95

Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of…

Forking in the background is one way, you can also use GNU "parallel" ("apt-get install parallel" for those on Debian and the likes). For example I've got all my CD ripped to FLAC files, but my car only takes mp3 or wav... So I did a batch convert of FLAC to mp3, making sure to put all cores at work by piping the output of "find" into "parallel". Some commands also allows to directly parallelize (like, say, "make -j…

Another tip is using wget2 instead of wget if you're mirroring a site (but this is more I/O tip than computationally heavy) https://gitlab.com/gnuwget/wget2/-/wikis/home

Sadly, wget2 doesn't support WARC last time I checked, but wget2 comes with a `--max-threads` parameter that together with `--mirror` and `--tries` makes it trivial to mirror even the slowest websites out there.

Edit: your parallel to `parallel` made me think of wget2 as I often see scripts that use `parallel` together with `wget` when `wget2` can be much better to use alone instead of pairing the two. Just wanted to add some context.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#109

Post by NateEag »

Cthulhu_ wrote: post #63
kohlerm wrote: post #36

There is no point running something like Hadoop or Spark or Flink if your problem is not bottlenecked by the resources of of a single machine. E.g. how would this solution work if the data is so large or the problem requires more context (looking at more than one line at the time) and therefore more memory? Answer: it wouldn't .

That's it, for a long time, what a lot of companies that jumped on the big data bandwagon didn't realize is that they didn't actually have big data. I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine. Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volu…

My favorite litmus test for "Big Data" is https://yourdatafitsinram.net.

If you can fit it into RAM on a single machine, you probably shouldn't be using complex distributed systems for working with it. The developer time spent setting it up and fixing arcane bugs due to the distributed system is likely to cost you more than a single monster server will.

If your processes generate large amounts of data rapidly, you can get a decent idea when you should start working on a move to a "big data" solution by looking at your rate of growth.

...though in my experience, stuff like logs can be tarballed and tossed on S3 as insurance just in case you really do need to get an overview of some pattern over the past three years. Mostly only about the past six months of logs are really worth keeping on hand actively, IMO.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#110

Post by api »

The mis-use of "big data" tools like Hadoop to do with entire giant clusters of systems what can be done more quickly on a MacBook Air encapsulates quite a bit of what is wrong with all modern software development: overengineering, dick size based infrastructure decisions ("look at how big my cluster is!"), stacking up tools to pad one's resume, mindlessly copying what bigger companies do, needless layers of abstraction, etc.

Look up the largest hard drive of any type that you can find for sale. Now spec out a consumer or small business grade NAS with 2-4 of those drives. If your data will fit there, you do not have "big data." If the cost bothers you consider that the cloud footprint (or on-prem mini data center) required to use your big sexy "big data" approach will cost far more than one of those NAS systems, possibly every month.

The only real exception is if you need performance and the computations you are doing are CPU bound or highly parallelizable. If you need rapid turnaround you may want some kind of distributed replicated cluster approach that can do things in parallel. For the majority of jobs though these are periodic or internal facing analytics jobs and getting the results faster is not worth 10X-100X the hardware cost or cloud bill and 10X the developer time.

Post reply on HN