Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

71–80 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#71

Post by renonce »

I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get something like MySQL, InfluxDB, TimescaleDB etc, but all of them are way too powerful and have their own query language and storage engine etc which are hard to learn and manipulate. In case you run out of storage/memory/CPU etc or when you want to query something that the database doesn't support, there is no way out of it without waiting for the database vendor to provide some new feature.

Now I'm just writing all data to plain text files, one JSON object per line, and query and process them with cli tools like jq. Regular compression tools like zstd, pattern matchers like grep works. All the Unix philosophy applies and it's easy to do anything I want without being restricted to the features of a certain database.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#72

Post by chakkepolja »

Cthulhu_ wrote: post #64
donatj wrote: post #14

I went to a Hadoop workshop in 2016 where the speaker was insistent Hadoop would replace traditional relational databases in the next 5 years. It’s been six and I think the death of relational databases has still been greatly exaggerated. https://twitter.com/donatj/status/740210538320273408

Things look to be quite cyclic; ten years ago, NoSQL was pushed as The Future, but these days companies still build their core data on top of a traditional relational database. I don't even know which NoSQL is even popular. It feels like it's been pushed to the fringes, e.g. key/value stores, or the Q in a CQRS setup where it acts more as a readonly cache.

MongoDB is pretty popular among newbie nodejs developers. (That's not a complement, btw).

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#73

Post by Beltalowda »

renonce wrote: post #71

I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get something like MySQL, InfluxDB, TimescaleDB etc, but all of them are way too powerful and have their own query language and storage engine etc which are hard to learn and manipulate. In case you run out of storage/memory/CPU etc or when you want to query somet…

TimescaleDB "upgrades" fairly gracefully from regular PostgreSQL; there's a bunch of Timescale-specific functions, but it's not that much: mostly it's just regular SQL. I don't know how well it performs with thousand records per second though.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#74

Post by HelloNurse »

tus666 wrote: post #58
wenc wrote: post #2

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.

What about if you had 500 petabytes of data to analyze? Big data does exist in the wild.

Those 500 PB imply a competent staff, a world-class infrastructure, and worthwhile applications. These things don't materialize suddenly: they start maturing when the data set is 500 GB or even just planned and they usually mature enough to switch technology multiple times as the scale rises.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#75

Post by rizkeyz »

StreamBright wrote: post #38
wenc wrote: post #2

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.

> Outside of legacy systems, Hadoop isn't widely used anymore. This is not true at all. Almost all the cloud providers have their own Hadoop distributions that is used a lot in many companies.

Exactly. Basically most big data lives in hdfs and hdfs is part of hadoop. Even if you use Spark and Flink I would classify that as using Hadoop (under the hood).

> However, a very common setup is to use Flink to analyze data stored in the Hadoop Distributed File System (HDFS). -- https://wints.github.io/flink-web//faq.html

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#76

Post by rizkeyz »

Cthulhu_ wrote: post #63
kohlerm wrote: post #36

There is no point running something like Hadoop or Spark or Flink if your problem is not bottlenecked by the resources of of a single machine. E.g. how would this solution work if the data is so large or the problem requires more context (looking at more than one line at the time) and therefore more memory? Answer: it wouldn't .

That's it, for a long time, what a lot of companies that jumped on the big data bandwagon didn't realize is that they didn't actually have big data. I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine. Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volu…

At this point in time (2022) I consider everything below say 40TB not big (textual) data at all. It can be compressed 40TB -> 10TB (or less) and that fits fine on a single 16T drive.

For many questions, you won't need all the raw data, so you end up with some form of projection of the data that is maybe 1/10 in size, so 10TB -> 1TB. Heck, if you tune GNU sort a bit, it will blast through that TB quite quickly.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#77

Post by easytiger »

kohlerm wrote: post #35
jamal-kumar wrote: post #4

Earlier quoted context omitted.

Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?

Apache Flink is an alternative with "real" (not micro batching) streaming support.

I spent a while putting a POC of Flink for a use case together at client behest a couple of years ago. Really struggled with it. Seemed slow as heck too. I'd much rather write a few hundred extra lines of my own code to do stuff like that (ingest -> process -> output. Modern software fashion trends seems to create as much work as it tries to save; as extensive and magical as a lot of the features are.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#78

Post by serial_dev »

auggierose wrote: post #69
wenc wrote: post #2

This should be tagged (2014). This article has made the rounds many times. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Outside of legacy systems, Hadoop isn't widely used anymore.

So what is used instead of Hadoop currently?

Whether a technology can replace Hadoop in an organization depends on many factors, but some technologies that solve at least in part similar problem are Apache Storm, Spark, Flink, Kafka Streams, and maybe BigQuery?

Or, as the original article says, some companies just use some command line tools, shell scripts.

It's been a couple of years since I was interested in Data Engineering, so my knowledge on this topic is some years behind.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#79

Post by trashtester »

mr_toad wrote: post #51
ryapric wrote: post #20

Earlier quoted context omitted.

AWS: Glue, and by proxy Athena GCP: Dataproc Those are just the obvious ones though.

Glue often uses EMR under the hood, which is often Spark. And Athena is PrestoDB, as far as I know it has nothing to do with Hadoop other than you can use it to query Hadoop data stores.

The way I see it, Hadoop is still in common use as the storage layer for Spark and related implementations, whether that is in the form of HDFS or something like EMRFS:

Quote from AWS: "EMRFS is an implementation of the Hadoop file system ..."

https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-p...

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#80

Post by gehen88 »

stingraycharles wrote: post #66
IanCal wrote: post #62

Earlier quoted context omitted.

The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.

I’d argue that for as long as Excel doesn’t support Parquet files, we haven’t seen the last of CSVs for a long time. Parquet is great, but it’s simply nowhere near as ubiquitous as CSV.

I hadn't even heard of Parquet until now, and I'm sure this goes for lots of developers who don't do much data engineering.
Post reply on HN