Using AWK and R to parse 25TB • Hacker News Board

Re: Using AWK and R to parse 25TB

#101

Post by flukus » Fri, Jun 28, 2019, 7:16 AM UTC

emmanueloga_ wrote: post #78

Unix pipelines, AWK, gnu parallel, R, all great stuff. If you have such an specific task, why not just write an "actual program" (as opposed to pipeline of scripts)? From the looks of it, it sounds like this problem could have been solved with, say, 50 lines of Java, C, Go, etc, etc. Maybe a bit more verbose but it would give you full control, you wouldn't need to lookup how to use command line parameters on S/O, and…

> If you have such an specific task, why not just write an "actual program" (as opposed to pipeline of scripts)?

With the pipeline you get free parallelism and it's much easier to iterate over the individual steps, run step manually, check the output, add it to the script. You can also trivially break bits into a make file for improved parellelism and incrementalism. Performance wise these tools have had a lot of work put into them, even when they're not the most efficient tool on paper they'll often beat out the most naive versions in "real" languages.

Re: Using AWK and R to parse 25TB

#102

Post by emmanueloga_ » Fri, Jun 28, 2019, 7:18 PM UTC

IndustrialJane wrote: post #98

emmanueloga_ wrote: post #88

Earlier quoted context omitted.

I get your point but the same error handling problems can appear in scripts and pipelines, no? In a program I'd try/catch defensively "just in case", if missing one line out of 25TB is not a bit deal. For parallel processing I'd reach for the nearest standard library at hand on the language of choice.

> For parallel processing I'd reach for the nearest standard library at hand on the language of choice. That is a good example of what I mean: The nearest standard library is likely to either buffer output in memory or not buffer at all (in which case you can have the start of one line ending with another line). This means you cannot deal with output bigger than physical RAM. And your test set will often be so small…

I agree with everything you said, as always, everything is a trade off. Good point about trickiness of memory management w/parallel processing! Would have to be extra careful to avoid hoarding RAM.

Re: Using AWK and R to parse 25TB

#103

Post by banku_brougham » Sat, Jun 29, 2019, 1:35 PM UTC

markus_zhang wrote: post #58

nstrayer wrote: post #28

Hi! Author of the post here. I can attempt to answer any questions if need be although it looks like others have done a great job doing that already!

Hi nstrayer thanks for the excellent article! I work as a data analyst and I never got to worry about big data as the DWH takes care of the aggregation for us, plus I only work in Windows. I see now that it would be very useful to learn *nix tools in general, as it seems that the skills to process (not to predict/analyze) terabytes+ data are very valuable and expensive to acquire and could be one's butter and bread.

Windows OS is making it easier to access nix terminal commands, but in my experience making the OS switch (mac machine but unix terminal, or cloud linux) has been game changing for me.

Re: Using AWK and R to parse 25TB

#104

Post by banku_brougham » Sat, Jun 29, 2019, 2:21 PM UTC

derefr wrote: post #38

scottlocklin wrote: post #19

You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...

I was writing a batchwise ETL tool (to break documents in some proprietary format down into rows to feed to Postgres's COPY command) and I achieved a remarkable level of IO parallelism by relying on Unix tooling to do my map-reducing for me. 1. I wrote a plain SQL mapper program, which spawns a worker-thread pool, where each worker opens its own "part" file for each SQL table, such that a document consumed by worker…

Could you share the code for these operations? I have some similar occasional use cases.

Re: Using AWK and R to parse 25TB

#105

Post by fpbarthel » Sat, Jun 29, 2019, 2:49 PM UTC

Just wondering if this problem could have been solved by a properly indexed table? The article says: “Eight minutes and 4+ terabytes of data queried later I had my results“. 4+ TB seems way too much for 60k patients and sounds like an inefficient table scan was performed.

Re: Using AWK and R to parse 25TB

#106

Post by fpbarthel » Sat, Jun 29, 2019, 3:14 PM UTC

fpbarthel wrote: post #105

Just wondering if this problem could have been solved by a properly indexed table? The article says: “Eight minutes and 4+ terabytes of data queried later I had my results“. 4+ TB seems way too much for 60k patients and sounds like an inefficient table scan was performed.

Also, wouldn’t partitioning only make sense if there is a sensible way to separate data that is more likely to be accessed vs data less likely to be accessed? Like is common with date data, since recent entries are often more relevant compared to old entries. For example, you could categorize SNPs by priority and eg. partition SNPs of high importance (frequently accessed) vs medium importance (sometimes accessed) vs low importance (rarely accessed).