Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

111–120 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#111

Post by kragen »

spullara wrote: post #48
Puts wrote: post #23

Earlier quoted context omitted.

Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.

I mean, it isn't like Hadoop wasn't used to parse text files. Also, it is all fun and games until you need a join.

comm(1) and join(1) can do joins on text files. Make sure they're sorted in the same locale you're joining them in; LANG=C tends to be the fastest.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#112

Post by markus_zhang »

bpodgursky wrote: post #6
jamal-kumar wrote: post #4

Earlier quoted context omitted.

Aside from Apache Spark, what's replaced it and does it still face the same speed of access limitations compared to just zipping through giant CSVs with awk or whatever streaming APIs you write with your own preferred language?

I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.

What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. Since I'm not a CTO who worries about not delivering, but a junior DE who actually wants to learn things, I really prefer that things were done in the old ways where we had to manage our own infrastructure and our own code for ETL. These kinds of things can not be learned "just for fun" because one has to work in a real environment.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#113

Post by ajoseps »

renonce wrote: post #71

I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get something like MySQL, InfluxDB, TimescaleDB etc, but all of them are way too powerful and have their own query language and storage engine etc which are hard to learn and manipulate. In case you run out of storage/memory/CPU etc or when you want to query somet…

I have a similar issue. I've been using feather files/parquet files for storage, and just using pandas to do analysis. There is an issue where the initial load/convert essentially doubles the memory usage of the file itself as it converts to a pandas dataframe. This can be avoided if you use a feather file and follow its recommendations for a zero-copy conversion (no NaNs/nulls).

I think it's a bit more flexible than using cli tools since you can set some sort of time index and query specific timeslices fairly easily

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#114

Post by pilotneko »

bsenftner wrote: post #98
riknox wrote: post #81

Earlier quoted context omitted.

I've not seen Storm being used anywhere sane for a few years at least now, and from a glance at job postings it looks unlikely. Spark, Kafka Streams etc. are definitely used in a modern data platform from my experience. I think we're seeing a big shift with Hadoop-like workloads being moved onto cloud providers, so BigQuery, Amazon EMR etc.

I'm curious what constitutes "big data" anymore. In an intermediate machine learning course, we train on nearly a petabyte of data using Google Colab and Jupyter Notebooks. Nobody discusses the size of the data requiring any special treatment due to its size... would not 95% of a petabyte be "big data"?

What course are you taking? Imagenet is only 150 GB, and Common Crawl is only 320 TB.

Big data is a moving target, but I’m comfortable defining it as data too large to fit in memory. Obviously, you can always get a bigger node, my rule is thumb is that if you need generators, you are working with big data.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#115

Post by tremon »

lmm wrote: post #54
988747 wrote: post #52

Earlier quoted context omitted.

No, that does not seem correct. SQL Datastores are not "map-reduce underneath", they have optimized datastructures for efficient querying (i.e. indices). Map-reduce is equivalent to those cases in SQL database where you have full table scan in your query plan - basically brute-forcing your way through the dataset.

You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cases you're doing some kind of aggregation over the whole table, so indices don't help any.

And if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#116

Post by charcircuit »

usrbinbash wrote: post #59
charcircuit wrote: post #33

Earlier quoted context omitted.

26 minutes worth? That can't be right. Sending 250 MB (uncompressed) to 7 machines shouldn't take that long.

How many sends are we talking about? Into how many messages is the data turned, how often does it get sent around? If I send 1MiB of data by packing it up into messages of 10 byte each, it will ikely be slower than sending 10MiB in a single message. Messages == Overhead. Envelopes, packing, unpacking, parsing, assembling, etc. all eat up cycles.

This is a distributed sum. You don't need extra messages being sent around. You send 7 messages giving each node 1/7th of all games. Then 1 message per node is sent with all the tallies to be merged together. And then maybe an extra message to give you the results if you weren't the one that merged it. 15 messages shouldn't take 26 minutes to send.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#117

Post by phkahler »

chockchocschoir wrote: post #106
athenot wrote: post #100

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutio…

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can…

>> And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be!

That's why the "big data" industry also encourages collection of absolutely every bit of data you can find. They want you to need their tools. You may not think there's a use for it, but vague promises of AI finding needles in the haystack are used to get you to keep it and bloat your system.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#118

Post by manigandham »

markus_zhang wrote: post #112
bpodgursky wrote: post #6

Earlier quoted context omitted.

I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.

What I really dislike modern cloud DWH such as Snowflake is that it hides a lot of things from me. Since I'm not a CTO who worries about not delivering, but a junior DE who actually wants to learn things, I really prefer that things were done in the old ways where we had to manage our own infrastructure and our own code for ETL. These kinds of things can not be learned "just for fun" because one has to work in a real…

What do you mean? Of course you can still learn them "just for fun" if you want. There are plenty of columnar data warehouses (memsql, greenplum, vertica, clickhouse, etc) and data processing frameworks (spark, flink, etc) that you can look at, implement and run yourself.

It's all using the same principles underneath.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#119

Post by manigandham »

deepstack wrote: post #25
bpodgursky wrote: post #6

Earlier quoted context omitted.

I use Spark for a number of jobs for language-specific features still but I think within 2 years all custom code will be trivially invoked as native UDFs in SQL data warehouses (ie Snowflake, which has essentially solved big-data performance as a going concern). I just write SQL in Snowflake and it replaces 95% of what I would otherwise have done in custom MapReduce or Spark code.

SQL will always be faster than Hadoop and MapReduce. The main reason to use those other slower services is developer are not use to SQL or declarative programming, and insist on having the code in Procedural way.

That makes no sense. SQL is a query language, commonly implemented by relational databases.

In the early 2000s, columnar relational data warehouses were not sophisticated and scalable enough to handle the scale of data encountered at Yahoo, Google and other internet companies. MapReduce (and the many evolutions of Hadoop ecosystem) was created to scale processing through low-level instructions and algorithms.

Eventually columnar data warehouses caught up and are now capable of handling petabyte scale, regardless of whatever language you use to query them. The fundamental storage and compute primitives haven't really changed that much, just offered in a much more user-friendly way now.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#120

Post by burntsushi »

ImaCake wrote: post #95

Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of…

Do note that unlike GNU (or BSD) grep, ripgrep will automatically use multiple threads to execute a search on a directory.
Post reply on HN