Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk: `Begin { ` Part 1

jemma.dev

Post reply on HN
Print view View original

21–30 of 110 posts

Re: Awk: `Begin { ` Part 1

#21

Post by Athas »

asicsp wrote: post #9

Nice introduction (based on both tutorials). One suggestion would be to use -v FPAT='[^,]*|"[^"]+"' instead of BEGIN { FPAT = "[^,]*|\"[^\"]+\"" } > If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk. I can't work my head around this quote. That's a ridiculous claim. Even for a experienced programmer, learning a new programming languag…

GNU Awk has lots of extensions and is a relatively big language, but I definitely think you can learn most of the original Awk in two weeks - and you can certainly learn enough to be useful in two hours. It's been a decade now, but my recollection is that when I spent a day or two idly reading the Awk book, and was able to immediately employ it in simple shell pipelines after reading not that many pages. I think Awk is useful once you've learned just a bit about it, and I don't think learning a lot about Awk is very useful - there are languages that are better suited for complex tasks.

Re: Awk: `Begin { ` Part 1

#22

Post by asicsp »

samoa42 wrote: post #12
asicsp wrote: post #9

Nice introduction (based on both tutorials). One suggestion would be to use -v FPAT='[^,]*|"[^"]+"' instead of BEGIN { FPAT = "[^,]*|\"[^\"]+\"" } > If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk. I can't work my head around this quote. That's a ridiculous claim. Even for a experienced programmer, learning a new programming languag…

> I've been using awk for past 2-3 years or so and I wrote a book on GNU awk one-liners not to criticise your work, but this does not sit well with me.

Could you elaborate? I didn't get your point.

Re: Awk: `Begin { ` Part 1

#23

Post by mdifrgechd »

mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

I have used awk a lot on csv files. I basically agree with your comments, for me the use case is preliminary exploration, not really much more. There is also the issue of one liners not really ending themselves to good coding practice generally, but it sure is great to dive into a file quickly.

What I came here to say is that when I have a csv file I'm looking at with awk, thos first thing I do is a `wc -l`, then get NF (number of fields) for the first row of the csv (here e.g. 80), followed by `awk -F, 'NF==80'` | wc -l`. If the numbers don't match, then I know it's not parsing properly.

The most common issue of course is commas in quotes strings. I have a small script that removes these, so I can still use awk easily. Anything more complex like newlines in quotes strings and maybe it's time to question if awk is still worth it.

Re: Awk: `Begin { ` Part 1

#24

Post by lmns »

dotancohen wrote: post #17
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?

Sqlite3 can import and export CSV via the CLI application. Both Ruby and Python provide CSV parsers in their standard library.

Re: Awk: `Begin { ` Part 1

#25

Post by asicsp »

Athas wrote: post #21
asicsp wrote: post #9

Nice introduction (based on both tutorials). One suggestion would be to use -v FPAT='[^,]*|"[^"]+"' instead of BEGIN { FPAT = "[^,]*|\"[^\"]+\"" } > If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk. I can't work my head around this quote. That's a ridiculous claim. Even for a experienced programmer, learning a new programming languag…

GNU Awk has lots of extensions and is a relatively big language, but I definitely think you can learn most of the original Awk in two weeks - and you can certainly learn enough to be useful in two hours. It's been a decade now, but my recollection is that when I spent a day or two idly reading the Awk book, and was able to immediately employ it in simple shell pipelines after reading not that many pages. I think Awk…

Fair enough, someone else made similar point that I had GNU awk in mind wrt all of awk

And yeah, it's possible to start using basic field processing and regexp features (if you already know it) in 2 hours and learn most of it in 2 weeks. But, that quote could've been something like you could get started in 2 hours instead of saying one could know all of awk.

Re: Awk: `Begin { ` Part 1

#26

Post by mehrdadn »

komali2 wrote: post #11
mehrdadn wrote: post #6

Earlier quoted context omitted.

Tangent, but another interesting problem with date parsing besides lack of standardization is that it's ridiculously context-sensitive. For example, if I schedule an event for Oct 1-Jan 1 right now, the most likely (least surprising) parse would be Oct 1, 2021 through January 1, 2022. Which is both surprising because parsing Oct 1 depends on the current date, and because the parsing of Jan 1 depends on the parse of O…

Heh, interesting, I actually assumed it would be oct 2020 to Jan 2021, and that you were describing an event already in progress.

Sure :-) even then, you have even more fun... imagine if the next item was Sep 1-Dec 1 and the list was known to be sorted! The parser would need to be stateful to be able to disambiguate!!

Even worse (well, maybe similar, but more surprising): imagine if on Feb 28, you parse "Feb 29-Mar 1"... both of those dates could land in an entirely different year than "Feb 28-Mar 1" would, depending on whether the current year is a leap year...

I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that.

And all of these problems are before we even consider time zones, leap seconds, daylight savings, syntactic ambiguities, etc... not just how they affect individual dates, but also ordered dates (/intervals) like above...

Re: Awk: `Begin { ` Part 1

#27

Post by 7thaccount »

lmns wrote: post #24
dotancohen wrote: post #17

Earlier quoted context omitted.

Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?

Sqlite3 can import and export CSV via the CLI application. Both Ruby and Python provide CSV parsers in their standard library.

Doesn't it convert everything to strings including numbers? Or can you specify types?

Re: Awk: `Begin { ` Part 1

#28

Post by rnestler »

dotancohen wrote: post #17
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?

There is https://github.com/BurntSushi/xsv but it doesn't seem to print with borders between cells.

Re: Awk: `Begin { ` Part 1

#30

Post by xashor »

mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

CSV to TSV via robust program, use AWK, and convert back if needed. Then you only have to remember the \\t\n\r escapes, not the whole quoting shenanigans.
Post reply on HN