Using AWK and R to parse 25TB • Hacker News Board

Re: Using AWK and R to parse 25TB

#61

Post by markus_zhang » Thu, Jun 27, 2019, 5:36 PM UTC

acomjean wrote: post #55

markus_zhang wrote: post #51

Earlier quoted context omitted.

I think that the optimal way to do these kind of things is: 1) Assuming there is no joins/merges requirement, read in chunks and output GB dumps. 2) If joins/merges are required, use external merge sort. Is this correct? Actually I'm wondering whether I could earn some bread and butter by focusing on the big data processing problems (e.g. sort/filter Terabytes+ dumps, do transformation for each line for Terabytes+ du…

Learning Unix tools is pretty good place to start. There are a lot of commands that can do a lot of processing. It’s been a while since I learned but the book “Unix power tools” from oreily is pretty good. It’s old, but honestly these commands haven’t changed much. http://shop.oreilly.com/product/9780596003302.do Python is slower compared to some of it’s compiled cousins, but it’s quick to write and a great skill to…

Thanks a lot! Time to fire up VirtualBox and learn some things.

Re: Using AWK and R to parse 25TB

#62

Post by 0x5002 » Thu, Jun 27, 2019, 5:44 PM UTC

devonkim wrote: post #48

scottlocklin wrote: post #19

You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

Furthermore, as computers get faster and cheaper in every dimension what makes economic sense to use “Big Data” tooling and efforts gets substantially larger with it. The limits of single nodes 15 years ago were pretty serious but most problems businesses have even in the so-called enterprise can currently easily fit on a workstation costing maybe $5k and be crunched through in a couple hours or maybe minutes - a lot…

That is precisely what the projects I'm usually involved in do. A client might want "buzzword technology", but at the heart of it, what they really need are stable, scalable, and consolidated data pipelines to e.g. Hadoop or AWS that gives "Data Scientists" a baseline to work with (and anyone needing information, really - it was just called "Business Intelligence" a couple of years ago).

In the end it doesn't matter if you wind up with a multi-TB copy of some large database or a handful of small XML files - it's all in one place, it gets updated, there are usable ACL in place, and it can be accessed and worked with. That's the point where you think about running a Spark job or the above AWK magic.

Re: Using AWK and R to parse 25TB

#63

Post by coliveira » Thu, Jun 27, 2019, 5:45 PM UTC

lordleft wrote: post #21

scottlocklin wrote: post #19

You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

AWK is such an amazing tool

Awk is the most useful tool that people largely ignore in the UNIX tool chest. If you think of any script that has simple logic and involves transformations on input data, it could be more easily written in awk and integrated with the shell. After learning awk your UNIX abilities will increase exponentially.

Re: Using AWK and R to parse 25TB

#64

Post by geogra4 » Thu, Jun 27, 2019, 5:48 PM UTC

I remember frantically needing to parse a few hundred megs of log files and join them up to the db rows that fired the individual errors.

Initially I was trying to use SQLite for it but I kept running out of memory and crashing the system. Turned out using grep, join, sort, and paste got the job done in seconds.

Re: Using AWK and R to parse 25TB

#65

Post by edraferi » Thu, Jun 27, 2019, 5:48 PM UTC

devonkim wrote: post #48

scottlocklin wrote: post #19

You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

Furthermore, as computers get faster and cheaper in every dimension what makes economic sense to use “Big Data” tooling and efforts gets substantially larger with it. The limits of single nodes 15 years ago were pretty serious but most problems businesses have even in the so-called enterprise can currently easily fit on a workstation costing maybe $5k and be crunched through in a couple hours or maybe minutes - a lot…

> The basics are missing by default

Absolutely. I really want to see advanced AI/ML tools developed to address THIS problem. Don’t make me solve the data before I use ML, give me ML to fix my data!

That’s hard though, because data chaos is unbounded and computers are still dumb. I think there’s still tons of room for improvement though.

Re: Using AWK and R to parse 25TB

#66

Post by dima55 » Thu, Jun 27, 2019, 5:55 PM UTC

srean wrote: post #45

Can someone add "join" to the language/standard please. Its awkward to have split but not join.

There's a "join" tool in the GNU coreutils. "man join"

Re: Using AWK and R to parse 25TB

#67

Post by mzs » Thu, Jun 27, 2019, 5:56 PM UTC

bsg75 wrote: post #26

Have used Mawk [1] in similar cases for a runtime savings, provided that the script works without any GNU Awk extensions. [1] https://invisible-island.net/mawk/

Thank you! For those that don't know, 15 years ago GNU awk was sometimes oddly REALLY slow. Mike's awk was not. Plus when things were overall slower back then, it mattered more.

But there were bugs in mawk and it seemed basically unmaintained. So you'd run into something and have to use gawk or perl instead.

That's no longer the case, the xterm guy adopted it, ten years ago, and now I know!

Re: Using AWK and R to parse 25TB

#68

Post by ramses0 » Thu, Jun 27, 2019, 6:04 PM UTC

derefr wrote: post #38

scottlocklin wrote: post #19

You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

I was writing a batchwise ETL tool (to break documents in some proprietary format down into rows to feed to Postgres's COPY command) and I achieved a remarkable level of IO parallelism by relying on Unix tooling to do my map-reducing for me. 1. I wrote a plain SQL mapper program, which spawns a worker-thread pool, where each worker opens its own "part" file for each SQL table, such that a document consumed by worker…

Look into "bash-reduce", but it'd be great to have something like "bark" (bash-spark) which consumed documents at a time... and you're right, it might not even be that difficult.

Re: Using AWK and R to parse 25TB

#69

Post by jeroenjanssens » Thu, Jun 27, 2019, 6:07 PM UTC

markus_zhang wrote: post #53

malshe wrote: post #20

Earlier quoted context omitted.

Thanks for the link. Very interesting. In OP's article there is a link to a book "Data Science at the Command Line" which sounds quite relevant: https://www.datascienceatthecommandline.com

Yeah I decide to read that book and install the Docker file. The thing is I'm kind of not sure how to setup the whole thing in Linux as I also need Python, some database and gcc. I think I should be able to find some tutorials for Ubuntu.

Although the docker image is based on Alpine Linux, examining the corresponding Dockerfile [1] may provide some guidance on how to install the tools and their requirements on Ubuntu. Let me know if you have any questions. Always happy to help.

[1] https://github.com/datascienceworkshops/dockerfiles/blob/mas...

Re: Using AWK and R to parse 25TB

#70

Post by samuell » Thu, Jun 27, 2019, 6:45 PM UTC

I counted to 15 awk calls in our latest pipeline processing drug compounds to build predictive machine learning models of them: https://github.com/pharmbio/ptp-project/blob/master/exp/2018...

One of the most eye-opening aspects of awk (goes for other pipeable commandline tools too), was how they support iterative development of pipelines regardless of data size.

We tried SQLite at some point for some of the stages because of pretty complicated selections sometimes, but it often won't give back a single result in minutes. Switching to AWK, I could immediately get some output, so I could quickly validate and iterate on the awk code until I got what I expected. The actual execution was likeways always very fast.