Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk: `Begin { ` Part 1

jemma.dev

Post reply on HN
Print view View original

11–20 of 110 posts

Re: Awk: `Begin { ` Part 1

#11

Post by komali2 »

mehrdadn wrote: post #6
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

Tangent, but another interesting problem with date parsing besides lack of standardization is that it's ridiculously context-sensitive. For example, if I schedule an event for Oct 1-Jan 1 right now, the most likely (least surprising) parse would be Oct 1, 2021 through January 1, 2022. Which is both surprising because parsing Oct 1 depends on the current date, and because the parsing of Jan 1 depends on the parse of O…

Heh, interesting, I actually assumed it would be oct 2020 to Jan 2021, and that you were describing an event already in progress.

Re: Awk: `Begin { ` Part 1

#12

Post by samoa42 »

asicsp wrote: post #9

Nice introduction (based on both tutorials). One suggestion would be to use -v FPAT='[^,]*|"[^"]+"' instead of BEGIN { FPAT = "[^,]*|\"[^\"]+\"" } > If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk. I can't work my head around this quote. That's a ridiculous claim. Even for a experienced programmer, learning a new programming languag…

> I've been using awk for past 2-3 years or so and I wrote a book on GNU awk one-liners

not to criticise your work, but this does not sit well with me.

Re: Awk: `Begin { ` Part 1

#13

Post by cafard »

genuinebyte wrote: post #4
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

The best description of dates and CSV files I've heard by far. Bravo. "Csv files are a little bit like like dates: superficially simple, with lots of corner cases. Largely for the same reason: lack of standardization."

I just attempted to run some government data through Python using the csv module. It worked like a charm, until it told me that a field had exceeded the maximum allowable length. A close look showed that a field began but did not end with a quotation mark. Using the csv.QUOTE_NONE flag resolved that, but did not remedy such quirks as short or long records.

CSV files, depending on who generates them, are a bit like dates if the status of a year as leap or not depended on whether the date of Easter were 0 mod 4.

Re: Awk: `Begin { ` Part 1

#14

Post by dkarl »

mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

I would love to have a command-line tool that reads CSV and has a ton of features to cover different quirks and errors, which can output cleaner formats that I can pipe into other command-line tools.

csvkit [0] might be that tool; I discovered it after my last painful encounter with CSV files and haven't used it in anger yet. Among other things, it translates CSV to JSON, so you can compose it with jq.

[0] https://csvkit.readthedocs.io/en/latest/index.html

Re: Awk: `Begin { ` Part 1

#15

Post by rethab »

I don't agree that you can learn AWK in its entirety in two hours, but I do (strongly) agree that you can teach yourself the basics in two hours and those two hours are extremely well spent considering that AWK is useful for so many things.

Surely handling the intricacies of CSV can be a pain, but let's face it: Most tasks are simple and its results are easily verified for correctness. Obviously, I don't write my production webserver with AWK -- I write simple script to parse data (locally), find semantic issues in CSV files, analyze log data, etc..

AWK is great! If you want a slightly bigger cheatsheet, here's mine: https://github.com/rethab/awk-scripts/blob/master/talk.txt

Re: Awk: `Begin { ` Part 1

#16

Post by dkarl »

firebacon wrote: post #10
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

There is https://tools.ietf.org/html/rfc4180 Most CSV files do not follow this standard of course. But you could normalize all CSV files to RFC4180 (or any other consistent format) as the first step of your processing pipeline.

The issue with encountering CSV in the wild is that everybody who appreciates standards and interoperability ditched it a long time ago. If you are consuming CSV files in the wild, you can be sure that whoever is supplying them to you is using horrible tools to create them and will be unwilling or unable to address issues you find in them.

Re: Awk: `Begin { ` Part 1

#17

Post by dotancohen »

mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI?

Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?

Re: Awk: `Begin { ` Part 1

#18

Post by cesaref »

My favourite bit of gawk (ok, an extended version of awk but it's probably what you are running) is the network support, which in the manual includes writing simple HTTP servers with it. I kid you not.

Re: Awk: `Begin { ` Part 1

#19

Post by asicsp »

dkarl wrote: post #14
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

I would love to have a command-line tool that reads CSV and has a ton of features to cover different quirks and errors, which can output cleaner formats that I can pipe into other command-line tools. csvkit [0] might be that tool; I discovered it after my last painful encounter with CSV files and haven't used it in anger yet. Among other things, it translates CSV to JSON, so you can compose it with jq. [0] https://cs…

See also https://github.com/BurntSushi/xsv

Re: Awk: `Begin { ` Part 1

#20

Post by basicexploit »

asicsp wrote: post #8
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

Yeah, here's what a robust solution for parsing csv with awk looks like: https://stackoverflow.com/questions/45420535/whats-the-most-...

I was looking for it so thanks a lot!
Post reply on HN