Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

161–169 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#161

Post by randomswede »

yung_steezy wrote: post #149
kenoph wrote: post #82

You can do a lot with just bash + pipes + unix tools. It can get messy as your pipeline grows though, and there are a lot of edge cases. Relevant: "bashML: Why Spark when you can Bash?" ( https://rev.ng/blog/bashml/post.html ), aka how to deduplicate git repositories using `comm` + `awk`.

Bash is a godsend for quick debugging and I can see the temptation to start writing production code using bash. It basically boils down to a few things IMO: - large bash scripts are hard to read/maintain - complex modelling chains need intermediary points in the processing On the latter point I can't count the amount of times where being able to query an athena database has saved a lot of headaches. The overhead from…

And bash is really painful once you're trying to do clever things with structured data.

Excellent glue, but there's also a skill in knowing when you should port your increasingly complicated shell script to another language.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#162

Post by randomswede »

rstuart4133 wrote: post #152
charcircuit wrote: post #122

Earlier quoted context omitted.

I think you are missing my point. It's a difference between 12 seconds on a laptop and 26 minutes on 7 c1.medium instances. Yes there is overhead in shipping the data around, but there isn't 26 minutes worth of overhead. If a c1.medium can process at the same speed as his laptop it should take less than 3 seconds worst case. And yes 7 machines should be faster at this scale.

> If a c1.medium can process at the same speed as his laptop As it happens, I got to perform that experiment. Sort of. I was moving stuff on physical Dell hardware to a virtualised environment, at the behest of MBA's. I was a bit concerned about it as we pushed the existing hardware hard - it had overnight stuff it had to get finished by morning. It had a 30% buffer. It wasn't even close. The "virtual environment" wa…

This is one of those "it depends" things. As usual, it's good to build an intuition for what you expect. But it is also worthwhile benchmarking it now and again.

At a previous job, we were pushing against going to "networked temp disk" instead of "local temp disk", on the assumption that local storage would be faster than remote storage. But, actual benchmarking showed that the "networked temp disk" was about two to three times faster. Mostly because almost all IOPS on each machine went to servicing network disk requests, so trying to squeeze in on one machine's IOPS caused IO stall times that trying to squeeze into N machines' IO queue didn't see.

It's also a "are you mostly doing read consecutive blocks" or "are you doing essentially random, scattered reads" (for read workloads, write workloads are a bit different, as it is approaching hard top speculatively write data that has not yet passed through a write(2) call).

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#163

Post by IanCal »

bell-cot wrote: post #132
IanCal wrote: post #131

Earlier quoted context omitted.

For many cases of passing data between systems, I'd say the gulf is dramatically lower than it has been until even fairly recently. Support for many standard packages is just there, swapping out "read_csv" for "read_parquet" if you're a pandas shop may even be enough. More and more tools read these directly, and it opens up a whole load of better options for processing data. > If you don't mind being old-school, the…

> I have used those before, and yet I still had those characters appear in data... Oh, yes - which is why I emphasized "is". But ASCII text is easy to test for, which lets you fast-track into exception handling - "Tell Sales that Customer data is not as represented", "Trouble-shoot internal data source", etc. (My experience is that substantial Customer data is never, ever as initially represented. Nor as represented…

> Oh, yes - which is why I emphasized "is". But ASCII text is easy to test for, which lets you fast-track into exception handling - "Tell Sales that Customer data is not as represented", "Trouble-shoot internal data source", etc.

Oh with that I mean the ASCII control characters appearing in inputs. So some columns would have record end markers in for example.

If I'm able to make everyone dealing with the reading and writing add specific characters to be used for start/end/etc I'd rather just tell them to swap to a parquet reader unless they've got a really good reason.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#164

Post by IanCal »

slavik81 wrote: post #141
IanCal wrote: post #131

Earlier quoted context omitted.

For many cases of passing data between systems, I'd say the gulf is dramatically lower than it has been until even fairly recently. Support for many standard packages is just there, swapping out "read_csv" for "read_parquet" if you're a pandas shop may even be enough. More and more tools read these directly, and it opens up a whole load of better options for processing data. > If you don't mind being old-school, the…

As someone in the process of setting up a pipeline and currently using pandas.to_csv as my output, I'm curious what makes you recommend parquet in particular? How does it compare to HDF or Feather?

It depends on what you're trying to optimise, parquet is a very good all round option. HDF I've never really gotten into as it always felt like a good solution only if I move everything over. It's great if your use case fits.

Feather is a layer on top of arrow and was a proof of concept (so I'm not sure how heavily it's used now), and arrow is fast becoming the interchange format. It's exactly laid out as things will be in memory - which means zero copy for shuttling it around from one place to another. I _think_ there is less support for feather but that is likely changing as everything converges.

Parquet should be

* Faster to write * Faster to read (even if you're reading the whole file, which actually isn't required, the format helps you read just sections of the columns you need) * Smaller * Better at handling actual floating points

than CSV, while having actual standards alongside it. Be a little wary of pandas guessing the right column types for you if you're creating partitioned files btw.

When you're working with pandas, etc (check out Dask) you can pretty much just swap out some reading and writing functions. You can also use pyarrow directly if you need to be very careful about column types.

For your use case you may want to explicitly use a single column for the features that is a list, I'm not sure if that's better/worse than having so many columns. If a reader may want to find just some images where a small subset of features are > X, you might benefit from multiple columns so that the reader only processes the data it needs.

Worth testing out, but I expect you should be able to try it out in an afternoon if you're already working with pandas/similar. Just install pyarrow and use a to_parquet. Things like dask (or straight pyarrow) give you partitioned files as output if you want too, if there's a useful column or columns to split on https://arrow.apache.org/docs/python/parquet.html#partitione...

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#166

Post by louwrentius »

fivea wrote: post #158
louwrentius wrote: post #130

Earlier quoted context omitted.

I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table. Even a 10+ year old DL380G8 - can hold 1.5TB+ RAM and 24/48 cores and that hardware is dirt cheap on the second hand market.

> I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table. If all a server does is serve mostly static content from memory then it's not expected to handle a demanding workload beyond networking. Therefore I fail to see the point of trying to downplay someone else's needs for TB of RAM…

You totally misinterpreted the point of my text.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#167

Post by mohanmcgeek »

bpodgursky wrote: post #11
mohanmcgeek wrote: post #9

Earlier quoted context omitted.

But I can write SQL in Spark just fine. Can't I?

You can write some SQL in Spark, but 1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc. 2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is…

> Snowflake is probably 10-50x as performant as Spark for data manipulation

Wow is this for a fact? I haven't used either in a while but I saw the blog post from databricks and Spark was more performant than snowflake.

I assumed that's what I'll also get when i run spark on kube

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#168

Post by LoriP »

renonce wrote: post #71

I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get something like MySQL, InfluxDB, TimescaleDB etc, but all of them are way too powerful and have their own query language and storage engine etc which are hard to learn and manipulate. In case you run out of storage/memory/CPU etc or when you want to query somet…

Just wanted to say that TimescaleDB is an extension of PostgreSQL and so it uses SQL. MySQL would use SQL too.

InfluxDB does, indeed, have its own query language though.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#169

Post by Breza »

athenot wrote: post #100

Large-scale storage clusters like Hadoop, Cassandra, ElasticSearch are generally slow, expensive, hard to set up properly and require a lot of monitoring to stay healthy. Use them only when other solutions won't do. If your data will fit in a set of text files or a cluster of relational databases, use those. Even if you plan on storing a gazillion TB of data, it's faster to iterate app logic on nimble storage solutio…

I highly recommend the book Designing Data-Intensive Applications. It talks at length about different complex data systems and then warns readers that they should only use them if absolutely necessary.
Post reply on HN