Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk: `Begin { ` Part 1

jemma.dev

Post reply on HN
Print view View original

71–80 of 110 posts

Re: Awk: `Begin { ` Part 1

#71

Post by stevebmark »

mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

q is a great tool for CSV manipulation http://harelba.github.io/q/ . Awk is not

Re: Awk: `Begin { ` Part 1

#72

Post by nmz »

sidpatil wrote: post #33
necheffa wrote: post #32

Earlier quoted context omitted.

Everything you just described seems like reasons to not use CSV files in general, rather than not use AWK on CSV.

I'm disappointed we didn't use ASCII-separated values. https://news.ycombinator.com/item?id=7474600

And apparently an even better solution is just putting the size in the format itself. No need to parse when you can just seek.

https://blog.kowalczyk.info/article/fc9203f7c72a4532b1ae51d0...

Re: Awk: `Begin { ` Part 1

#73

Post by disown »

rethab wrote: post #15

I don't agree that you can learn AWK in its entirety in two hours, but I do (strongly) agree that you can teach yourself the basics in two hours and those two hours are extremely well spent considering that AWK is useful for so many things. Surely handling the intricacies of CSV can be a pain, but let's face it: Most tasks are simple and its results are easily verified for correctness. Obviously, I don't write my pro…

The biggest problem with awk is that unless you use it regularly, you'll easily forget how to use it. So it's important to keep a running cheatsheet like you did. Nothing more frustrating than forgetting the awk command and searching for it in your history only to find it aged out.

Re: Awk: `Begin { ` Part 1

#74

Post by drran »

stevebmark wrote: post #70
dkarl wrote: post #14

Earlier quoted context omitted.

I would love to have a command-line tool that reads CSV and has a ton of features to cover different quirks and errors, which can output cleaner formats that I can pipe into other command-line tools. csvkit [0] might be that tool; I discovered it after my last painful encounter with CSV files and haven't used it in anger yet. Among other things, it translates CSV to JSON, so you can compose it with jq. [0] https://cs…

"q" is the tool you're looking for http://harelba.github.io/q/ . Impossible to Google for, indispensable for CSV manipulation

> As of version 2.0.9, there's no need for any external dependency. Python itself (3.7), and any needed libraries are self-contained inside the installation, isolated from the rest of your system.

Oh, sh*t. I will look for something else.

Re: Awk: `Begin { ` Part 1

#75

Post by SoSoRoCoCo »

The O'Reilly book is "Sed & Awk" published in the 90's (https://www.oreilly.com/library/view/sed-awk/1565922255/). The languages work together as a team (like lex and yacc). And the book takes about 2 hours to read. Unix is built on the premise of "lots of simple tools that fit together with pipes".

Although this is like 1,000th article on how easy it is to learn awk. Why write yet another one?

Awk in 20 Minutes (2015) https://news.ycombinator.com/item?id=23048054

Learn just a little Awk (2010) https://news.ycombinator.com/item?id=17322412

Learn Awk by Example (2019) https://news.ycombinator.com/item?id=22455779

it goes on and on...

https://hn.algolia.com/?q=awk

Re: Awk: `Begin { ` Part 1

#76

Post by deathanatos »

dkarl wrote: post #16
firebacon wrote: post #10

Earlier quoted context omitted.

There is https://tools.ietf.org/html/rfc4180 Most CSV files do not follow this standard of course. But you could normalize all CSV files to RFC4180 (or any other consistent format) as the first step of your processing pipeline.

The issue with encountering CSV in the wild is that everybody who appreciates standards and interoperability ditched it a long time ago. If you are consuming CSV files in the wild, you can be sure that whoever is supplying them to you is using horrible tools to create them and will be unwilling or unable to address issues you find in them.

> The issue with encountering CSV in the wild is that everybody who appreciates standards and interoperability ditched it a long time ago.

I worked on a team that used CSV somewhat extensively. For the data we generated, it was RFC complaint. It's pretty trivial to get RFC-compliant CSVs, too; most languages have a library — ours was in the standard library, too.

We also had a ("terrible", as we joked) idea to create a subset of CSV that would contain typing information in a required header row. (We never did it, and it is a bad idea.)

> If you are consuming CSV files in the wild, you can be sure that whoever is supplying them to you is using horrible tools to create them and will be unwilling or unable to address issues you find in them.

…but this is absolutely true. We also consumed CSVs from external sources and contractors, and this was an absolute drain on our productivity. I've also worked with engineers of this caliber, and changing CSV wouldn't change the terrible output. I've seen folks approach eMail, HTTP with a cavalier "oh, it's a trivial text format, I don't need a library!" attitude, and inevitably get it wrong. Pointing out the flaws in their implementation and that a library would fulfill their use-case just fine is just met with more hacks (not fixes) to try to further munge the output into shape. It is decidedly not software engineering. I've seen this even with JSON.

But yeah, even with RFC standard CSV, you shouldn't be parsing it with awk. It is the wrong tool.

Re: Awk: `Begin { ` Part 1

#77

Post by mulmen »

mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

What format of data file is best for working with Awk? I assume some kind of tab delimited file? With a reliable ingestion layer you can convert all your files CSV-ish files to something ergonomic for Awk scripting.

JSON data files seem to be a similar deal. Sometimes they are actually properly formed JSON arrays and sometimes they are individual line-delimited objects.

Data files are a mess, is there a command line tool that is well suited to taking in many inconsistent formats and outputting something ergonomic?

Re: Awk: `Begin { ` Part 1

#78

Post by celticninja »

SoSoRoCoCo wrote: post #75

The O'Reilly book is "Sed & Awk" published in the 90's ( https://www.oreilly.com/library/view/sed-awk/1565922255/ ). The languages work together as a team (like lex and yacc). And the book takes about 2 hours to read. Unix is built on the premise of "lots of simple tools that fit together with pipes". Although this is like 1,000th article on how easy it is to learn awk. Why write yet another one? Awk in 20 Minutes (2…

The reason to write one is to record what you learn, to further embed it into your brain. Why post it? Whoever posted it found it interesting.

I had not seen the others so this is an intro to awk for me. But in general if something has made it to the front page of HN it was interesting enough for enough people to put it there.

Re: Awk: `Begin { ` Part 1

#79

Post by mulmen »

sergeykish wrote: post #40
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

$ ruby -rcsv -ne 'puts $_.parse_csv[1]' education.csv Enrolment in primary, secondary and tertiary education levels Total, all countries or areas AWK is a nice little language. Perl is all of it and shell, sed, grep. I am not proficient in Perl but I know some Ruby $ ruby -rEnglish -ne 'puts $_ if $NR Module English provides AWK names $ perl -MEnglish -ne 'print if $NR https://ruby-doc.org/stdlib-2.3.0/libdoc/English…

Ah Perl, the Swiss Army Chainsaw.

My first big kid job was taking over ownership of a Perl-based Oracle-backed data warehouse. Mostly it was SQL queries wrapped up with a Perl script executed by cron that output excel workbooks or csvs and emailed or dumped to a file server.

Most of the pivots and reporting tables were actually generated in Perl because it was just nicer to work with than Excel.

It was wonderful, I learned so much, mostly how to love Perl and CPAN.

We merged our telco billing system with our new parent company with some Perl, cron, a couple SQL queries and an FTP server.

I have said it for years and I will continue to repeat. If you could snap your fingers and delete all the Perl code in the world your lights would turn off.

Re: Awk: `Begin { ` Part 1

#80

Post by mulmen »

colejohnson66 wrote: post #65
firebacon wrote: post #10

Earlier quoted context omitted.

There is https://tools.ietf.org/html/rfc4180 Most CSV files do not follow this standard of course. But you could normalize all CSV files to RFC4180 (or any other consistent format) as the first step of your processing pipeline.

But if you’re writing code to normalize before sending to awk, why not just process in the normalization program instead of using awk’s bizarre syntax?

Depends on what you're using to normalize I suppose. Maybe that's Awk too! Maybe Awk is easier for exploration when the data is already clean but you have to write some parsing layer in Java or something and that's not conducive to one-liner exploration.
Post reply on HN