Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

81–90 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#81

Post by riknox »

serial_dev wrote: post #78
auggierose wrote: post #69

Earlier quoted context omitted.

So what is used instead of Hadoop currently?

Whether a technology can replace Hadoop in an organization depends on many factors, but some technologies that solve at least in part similar problem are Apache Storm, Spark, Flink, Kafka Streams, and maybe BigQuery? Or, as the original article says, some companies just use some command line tools, shell scripts. It's been a couple of years since I was interested in Data Engineering, so my knowledge on this topic is…

I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience.

I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#82

Post by kenoph »

You can do a lot with just bash + pipes + unix tools. It can get messy as your pipeline grows though, and there are a lot of edge cases.

Relevant: "bashML: Why Spark when you can Bash?" (https://rev.ng/blog/bashml/post.html), aka how to deduplicate git repositories using `comm` + `awk`.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#83

Post by trashtester »

rizkeyz wrote: post #76
Cthulhu_ wrote: post #63

Earlier quoted context omitted.

That's it, for a long time, what a lot of companies that jumped on the big data bandwagon didn't realize is that they didn't actually have big data. I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine. Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volu…

At this point in time (2022) I consider everything below say 40TB not big (textual) data at all. It can be compressed 40TB -> 10TB (or less) and that fits fine on a single 16T drive. For many questions, you won't need all the raw data, so you end up with some form of projection of the data that is maybe 1/10 in size, so 10TB -> 1TB. Heck, if you tune GNU sort a bit, it will blast through that TB quite quickly.

If you just want cold storage, you can put 10TB of compressed textual data on a spinning hard drive. If want to run some processing of that data within a workday, you need multiple drives in parallel (still possible on a single server). However, if you want to process it in less than 30 mins you need a cluster.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#84

Post by treeman79 »

25 years ago my company Needed to match lists against a collection numbers not to call. Probably ran lists of 20k daily or more. So I wrote up a simple python program to match against a special format txt file. A few seconds for big lists at most.

Higher up IT people were horrified, and commissioned a proper oracle soliton. It ran around a 1-2 records per second. So all day for a typical list. They were quite proud of this. I think they spent 25-50 grand on IBM support to get it setup.

They were not happy when we said we would only consider using it if they could speed it up 5000 times faster.

We never heard back.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#85

Post by anonymousDan »

rizkeyz wrote: post #76
Cthulhu_ wrote: post #63

Earlier quoted context omitted.

That's it, for a long time, what a lot of companies that jumped on the big data bandwagon didn't realize is that they didn't actually have big data. I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine. Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volu…

At this point in time (2022) I consider everything below say 40TB not big (textual) data at all. It can be compressed 40TB -> 10TB (or less) and that fits fine on a single 16T drive. For many questions, you won't need all the raw data, so you end up with some form of projection of the data that is maybe 1/10 in size, so 10TB -> 1TB. Heck, if you tune GNU sort a bit, it will blast through that TB quite quickly.

The problem isn't necessarily (just) capacity. It's also the he I/O bandwidth needed to read the data.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#86

Post by bell-cot »

IanCal wrote: post #62
Puts wrote: post #23

Earlier quoted context omitted.

Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.

The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.

If you don't already have parquet deployed, there's a wide gulf (in skill set, overhead, etc.) between CSV and parquet.

If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters - specifically intended for such uses. Micro conceptual overhead, none of the CSV issues which screw up many *nix text-handling programs and little script files, and a decent modern filesystem can handle the compression separately.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#87

Post by Lucasoato »

rxin wrote: post #12
bpodgursky wrote: post #11

Earlier quoted context omitted.

You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is…

You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)

Databricks is amazing, the Delta Live Table technology is incredible. It's very hard to approach problems like Data Lineage and Data Quality, but that platform does it in the right way.

My only concern is that they offer just a managed cloud product. That's cool for startups, but large enterprises sometimes need more governance and ownership than that.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#89

Post by kenoph »

bell-cot wrote: post #86
IanCal wrote: post #62

Earlier quoted context omitted.

The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.

If you don't already have parquet deployed, there's a wide gulf (in skill set, overhead, etc.) between CSV and parquet. If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters - specifically intended for such uses. Micro conceptual overhead, none of the CSV issues which screw up many *nix text-handling program…

Tbh Unix programs don't handle non-ASCII text very well, in my experience.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#90

Post by derriz »

bell-cot wrote: post #86
IanCal wrote: post #62

Earlier quoted context omitted.

The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.

If you don't already have parquet deployed, there's a wide gulf (in skill set, overhead, etc.) between CSV and parquet. If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters - specifically intended for such uses. Micro conceptual overhead, none of the CSV issues which screw up many *nix text-handling program…

Yes - parquet tooling is non-existent compared to CSV particularly on the command line. And the cross-language/platform support is a mess - good luck reading Pandas generated parquet on .NET or in a (non-spark) JVM environment.

There are many reasons why CSV is flawed for the purposes of storing tabular data (e.g. loss of column type information) but the alternatives are just so unergonomic that CSV remains a viable choice in many situations.

Post reply on HN