Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

141–150 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#141

Post by slavik81 »

IanCal wrote: post #131
bell-cot wrote: post #86

Earlier quoted context omitted.

If you don't already have parquet deployed, there's a wide gulf (in skill set, overhead, etc.) between CSV and parquet. If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters - specifically intended for such uses. Micro conceptual overhead, none of the CSV issues which screw up many *nix text-handling program…

For many cases of passing data between systems, I'd say the gulf is dramatically lower than it has been until even fairly recently. Support for many standard packages is just there, swapping out "read_csv" for "read_parquet" if you're a pandas shop may even be enough. More and more tools read these directly, and it opens up a whole load of better options for processing data. > If you don't mind being old-school, the…

As someone in the process of setting up a pipeline and currently using pandas.to_csv as my output, I'm curious what makes you recommend parquet in particular? How does it compare to HDF or Feather?

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#142

Post by efxhoy »

ImaCake wrote: post #95

Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of…

If you care about errors you can use

    function wait_all {
      for job in $(jobs -p); do wait "${job}"; done
    }
A lone wait wont tell you if any of the jobs fail.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#143

Post by 988747 »

tremon wrote: post #115
lmm wrote: post #54

Earlier quoted context omitted.

You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cases you're doing some kind of aggregation over the whole table, so indices don't help any.

And if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.

To be fair, Parquet, which is commonly used in Big Data solutions is a column store format. So, once you normalize your data and save it as Parquet you can have efficient column-wise aggregation - but that assumes some preprocessing step.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#144

Post by slowhand09 »

I've seen this article before and still consider it valid, especially for one-offs. Another similar and valuable resource is "https://datascienceatthecommandline.com/" Data Science at the Command Line, by Jeronen Janssens. Free to read at the link.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#145

Post by manigandham »

derriz wrote: post #90
bell-cot wrote: post #86

Earlier quoted context omitted.

If you don't already have parquet deployed, there's a wide gulf (in skill set, overhead, etc.) between CSV and parquet. If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters - specifically intended for such uses. Micro conceptual overhead, none of the CSV issues which screw up many *nix text-handling program…

Yes - parquet tooling is non-existent compared to CSV particularly on the command line. And the cross-language/platform support is a mess - good luck reading Pandas generated parquet on .NET or in a (non-spark) JVM environment. There are many reasons why CSV is flawed for the purposes of storing tabular data (e.g. loss of column type information) but the alternatives are just so unergonomic that CSV remains a viable…

Alternatives are available like Avro

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#146

Post by sitkack »

phkahler wrote: post #117
chockchocschoir wrote: post #106

Earlier quoted context omitted.

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can…

>> And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! That's why the "big data" industry also encourages collection of absolutely every bit of data you can find. They want you to need their tools. You may not think there's a use for it, but vague promises of AI finding needles…

This is why the first, possibly most important phase of a data analytics job is to filter down to the subset of the data that you actually need to compute the result.

Collecting it all on the front end can be very effective, as long as you can easily filter. If you can't, you are just causing yourself more problems.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#147

Post by chakkepolja »

burntsushi wrote: post #120
ImaCake wrote: post #95

Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of…

Do note that unlike GNU (or BSD) grep, ripgrep will automatically use multiple threads to execute a search on a directory.

Is there a reason classic grep or find utilities do not use multithreading?

It takes 5-6 seconds to search for some files on my machine. Would'nt it be an easy win if `find` had spawned 4 threads?

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#148

Post by awild »

slavik81 wrote: post #141
IanCal wrote: post #131

Earlier quoted context omitted.

For many cases of passing data between systems, I'd say the gulf is dramatically lower than it has been until even fairly recently. Support for many standard packages is just there, swapping out "read_csv" for "read_parquet" if you're a pandas shop may even be enough. More and more tools read these directly, and it opens up a whole load of better options for processing data. > If you don't mind being old-school, the…

As someone in the process of setting up a pipeline and currently using pandas.to_csv as my output, I'm curious what makes you recommend parquet in particular? How does it compare to HDF or Feather?

Just a heads up, pandas to Csv is laughably slow. You're literally better off converting the pandas df to arrow and then serialise to Csv with arrow, though the format is quite limited with arrow.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#149

Post by yung_steezy »

kenoph wrote: post #82

You can do a lot with just bash + pipes + unix tools. It can get messy as your pipeline grows though, and there are a lot of edge cases. Relevant: "bashML: Why Spark when you can Bash?" ( https://rev.ng/blog/bashml/post.html ), aka how to deduplicate git repositories using `comm` + `awk`.

Bash is a godsend for quick debugging and I can see the temptation to start writing production code using bash. It basically boils down to a few things IMO: - large bash scripts are hard to read/maintain - complex modelling chains need intermediary points in the processing On the latter point I can't count the amount of times where being able to query an athena database has saved a lot of headaches. The overhead from parquet and AWS bills pales in comparison. I'm sure almost everyone already agrees with me here but it's a classic case of the whole being more than the sum of its parts.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#150

Post by AtlasBarfed »

mathogre wrote: post #137

At my old company, Hadoop, JSON, Parquet were the solutions looking for a problem to solve. Yes we handled lots of data, but none of it was time critical. No matter, we had BIG DATA™. Bow when I say that. We weren't Google or Twitter. We were not surgically retrieving data, and while we had lots of data, we didn't have huge data centers; Dremel wasn't relevant to us. It's funny! It would take our Hadoop team three we…

A formal enterprise data group should probably

- be a clone/repo for disparate databases so you don't need to figure out access/security/location or impact production systems

- an interface to management types that aren't technical or don't have tech people to do these things

- should provide a "librarian" knowledge of the enterprise's data and data sources

- should have knowledge on how to analyze data using different tools

- be able to schedule movements/reports and manage that

If you don't need any of that, then ... yeah, don't use it. But those sets of requirements should be useful to anything that deems itself an "enterprise".

Post reply on HN