Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

adamdrake.com

Post reply on HN
Print view View original

131–140 of 169 posts

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#131

Post by IanCal »

bell-cot wrote: post #86
IanCal wrote: post #62

Earlier quoted context omitted.

The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.

If you don't already have parquet deployed, there's a wide gulf (in skill set, overhead, etc.) between CSV and parquet. If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters - specifically intended for such uses. Micro conceptual overhead, none of the CSV issues which screw up many *nix text-handling program…

For many cases of passing data between systems, I'd say the gulf is dramatically lower than it has been until even fairly recently. Support for many standard packages is just there, swapping out "read_csv" for "read_parquet" if you're a pandas shop may even be enough. More and more tools read these directly, and it opens up a whole load of better options for processing data.

> If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters

I have used those before, and yet I still had those characters appear in data. The only places I'd ever seen them were in the wiki page and in customer delivered data. Absolute pain to dig through and remove.

On top of that "if your data is ASCII" is something I'd be nervous about for many use cases even if it is right now.

Beyond that, then you need everyone to swap out their parsing to use those characters.

CSV is fine until it totally blows up in your face. All it takes is one "oh it's fine we'll use awk" stage somewhere or a CSV parser that isn't good enough and one person to put a newline where nobody had expected it before.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#132

Post by bell-cot »

IanCal wrote: post #131
bell-cot wrote: post #86

Earlier quoted context omitted.

If you don't already have parquet deployed, there's a wide gulf (in skill set, overhead, etc.) between CSV and parquet. If you don't mind being old-school, the data is ASCII text, and you're tired of some of CSV's little issue, then ASCII has the FS, GS, RS, and US control characters - specifically intended for such uses. Micro conceptual overhead, none of the CSV issues which screw up many *nix text-handling program…

For many cases of passing data between systems, I'd say the gulf is dramatically lower than it has been until even fairly recently. Support for many standard packages is just there, swapping out "read_csv" for "read_parquet" if you're a pandas shop may even be enough. More and more tools read these directly, and it opens up a whole load of better options for processing data. > If you don't mind being old-school, the…

> I have used those before, and yet I still had those characters appear in data...

Oh, yes - which is why I emphasized "is". But ASCII text is easy to test for, which lets you fast-track into exception handling - "Tell Sales that Customer data is not as represented", "Trouble-shoot internal data source", etc.

(My experience is that substantial Customer data is never, ever as initially represented. Nor as represented after you point out the first set of issues with it. Nor as represented after you point out the second set of issues. Nor as...)

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#133

Post by AitchEmArsey »

zozbot234 wrote: post #123
chockchocschoir wrote: post #106

Earlier quoted context omitted.

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice. And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be! On that note, there is this great website for seeing if there are servers that can…

You could also use SSI (single system image) system architectures, including distributed shared memory, to seamlessly (at least wrt. software implementation) scale up from efficient single-node processing to an arbitrarily large, clustered system. AIUI this is how things are traditionally done in mainframe processing, and it is also regarded as a very meaningful possibility in HPC.

That concept has fallen out of favour in HPC, but it'll be interesting to see if it makes a comeback.

As recently as 2016, an SGI UV3000 rack-scale SMP machine with 16TB of RAM was the sort of thing you'd see on a trade-show floor - now you can get that much memory in a 4U chassis, and things will only improve further if/when Optane DIMMs take off.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#135

Post by westurner »

wodenokoto wrote: post #50
ed_elliott_asc wrote: post #46

It often isn’t about the execution time, if it’s a daily job and takes 26 minutes who cares? Using something like databricks means it is easy to schedule and manage jobs, easy to write jobs that work in good enough time, easy to troubleshoot when things go wrong. It comes with a well documented security mode and a support contract when needed. Developers can be onboarded quickly and work code reviewed and managed. Lo…

Can't you just ask one of the machines in the databricks/spark cluster to run those shell commands? Or is that more of a kubernetes thing?

That's pretty much how Dask works; though Dask doesn't throw away data types for newline-delimited strings between each failed partitioned IPC pipeline process, and then the new hire didn't appropriately trap errors in their shell script, so the log messages are chronologically non-sequential and text-only.

https://github.com/dask/dask-labextension :

> This package provides a JupyterLab extension to manage Dask clusters, as well as embed Dask's dashboard plots directly into JupyterLab panes.

Something like ml-hub allows MLops teams to create resource-quota'd containers with k8s and IAM, though even signed code can DoS an unauditable system with no logs of which processes ran which signed archive of which code at what time, with bash and ssh. https://github.com/ml-tooling/ml-hub

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#136

Post by dhx »

IanCal wrote: post #62
Puts wrote: post #23

Earlier quoted context omitted.

Maybe we need a reminder every now and then that text-files are quite capable and except from being faster to parse in a lot of situation has the benefits of being "future safe" and easy to backup and compress as well.

The common text file for lots of this, CSVs, are absolutely awful. They're fine until they totally aren't, they were just the best option for a lot of use cases. I'd argue that's now been entirely replaced with parquet, significantly faster and broad support, proper types and more.

Column-oriented formats such as Parquet can be awful too. For example, if you have five columns of numbers you want to multiply together, Parquet is going to be the worst option available because of the extreme cache thrashing that the CPU will encounter.

Structure packing[1] and consideration of locality of reference[2] would need to be applied for high performance applications where a computer scientist has considered the algorithm needing to be implemented and the most efficient data format that the source data would need to be provided in.

[1] http://www.catb.org/esr/structure-packing/

[2] https://en.wikipedia.org/wiki/Locality_of_reference

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#137

Post by mathogre »

At my old company, Hadoop, JSON, Parquet were the solutions looking for a problem to solve. Yes we handled lots of data, but none of it was time critical. No matter, we had BIG DATA™. Bow when I say that. We weren't Google or Twitter. We were not surgically retrieving data, and while we had lots of data, we didn't have huge data centers; Dremel wasn't relevant to us.

It's funny! It would take our Hadoop team three weeks to get data together for our use. I often didn't have three weeks. In those times when I needed to use the data from the previous day, I'd just grab the raw data, organize it, process it, and be done with it in a few hours, using Unix/Linux tools and a bit of mathemagical wizardry.

"You're supposed to use Hadoop."

"You wanted to know what happened yesterday."

"I did!"

"If you want it from Hadoop, it will be ready in three weeks. Probably. That's if they have everything done."

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#138

Post by elmolino89 »

I had to ingest and explore enough dirty CSV/TSV data to use CLI tools only to get a first glimpse what's there. Whenever facing anything non trivial I go for CSV to parquet, and preferably write intermediate parquet data sets. Then DuckDB / SQL queries for slicing and dicing. I am yet to encounter some readable to a newcomers awk/uniq/sed combos for intersecting bunch of CSVs with 30+ columns.

On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it.

Re: Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

#139

Post by p33p »

elmolino89 wrote: post #138

I had to ingest and explore enough dirty CSV/TSV data to use CLI tools only to get a first glimpse what's there. Whenever facing anything non trivial I go for CSV to parquet, and preferably write intermediate parquet data sets. Then DuckDB / SQL queries for slicing and dicing. I am yet to encounter some readable to a newcomers awk/uniq/sed combos for intersecting bunch of CSVs with 30+ columns. On the other hand parq…

> On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it.

I thought parquet was columnar stored? Is this a fault of parquet or just the shear number of columns trying to get accessed?

I agree with your general premise though. I'd rather take a dirty dataset, throw it into S3, spin up a Redshift cluster, do what I need, spin down the cluster. You can work with billions of records fairly easily with plain old SQL and c-store databases.

Post reply on HN