Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk: `Begin { ` Part 1

jemma.dev

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Awk: `Begin { ` Part 1

#81

Post by mulmen »

jandrese wrote: post #68
mehrdadn wrote: post #6

Earlier quoted context omitted.

Tangent, but another interesting problem with date parsing besides lack of standardization is that it's ridiculously context-sensitive. For example, if I schedule an event for Oct 1-Jan 1 right now, the most likely (least surprising) parse would be Oct 1, 2021 through January 1, 2022. Which is both surprising because parsing Oct 1 depends on the current date, and because the parsing of Jan 1 depends on the parse of O…

Parsing anything that was originally developed by humans writing it on paper (or clay tablet, or whatever) is a nightmare. Natural means chaotic. If your CSV file contains any field entered by humans AWK isn't going to be powerful enough to parse it at scale. Someone somewhere is going to have the name 'Mbat"a, Sho,dlo' in some bizarre ass romanization (and this assume you're not accepting Unicode, which is a whole o…

I'm saving that as a test case name but making some small adjustments.

'Mbaät\"a, Sho,dló'

"a" followed by "ä" because some suggest encoding umlauts as double characters. When you decode that does it go first or second?

Answer: use Unicode.

Throw an escape character in there, before the quote character just to make it interesting.

This is a good time for everyone to review: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...

Re: Awk: `Begin { ` Part 1

#82

Post by JoshTriplett »

asicsp wrote: post #19
dkarl wrote: post #14

Earlier quoted context omitted.

I would love to have a command-line tool that reads CSV and has a ton of features to cover different quirks and errors, which can output cleaner formats that I can pipe into other command-line tools. csvkit [0] might be that tool; I discovered it after my last painful encounter with CSV files and haven't used it in anger yet. Among other things, it translates CSV to JSON, so you can compose it with jq. [0] https://cs…

See also https://github.com/BurntSushi/xsv

Seconding the recommendation for xsv; I've used it extensively and it works great.

Re: Awk: `Begin { ` Part 1

#83

Post by jhoechtl »

dkarl wrote: post #14
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

I would love to have a command-line tool that reads CSV and has a ton of features to cover different quirks and errors, which can output cleaner formats that I can pipe into other command-line tools. csvkit [0] might be that tool; I discovered it after my last painful encounter with CSV files and haven't used it in anger yet. Among other things, it translates CSV to JSON, so you can compose it with jq. [0] https://cs…

Use miller and never look back.

https://miller.readthedocs.io/en/latest/10min.html

It so much faster than csvkit

Re: Awk: `Begin { ` Part 1

#84

Post by stevebmark »

drran wrote: post #74
stevebmark wrote: post #70

Earlier quoted context omitted.

"q" is the tool you're looking for http://harelba.github.io/q/ . Impossible to Google for, indispensable for CSV manipulation

> As of version 2.0.9, there's no need for any external dependency. Python itself (3.7), and any needed libraries are self-contained inside the installation, isolated from the rest of your system. Oh, sh*t. I will look for something else.

Is your concern that you don't want to use a library that depends on python?

Re: Awk: `Begin { ` Part 1

#85

Post by heresie-dabord »

mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

> Csv is not standardized and the quoting rules are weird (and not standardized)

There is almost always some normalisation ("scrubbing") needed to prep a CSV. But CSV is viable and awk can rip through massive amounts of data. It is a brilliant and powerful tool.

> It's small enough to fit in your brain, unlike Perl

Perl is also brilliant and powerful. Setting aside the bigotry of people who dislike sigils and using braces for scope, many people who fail to learn Perl well have not tried to use Perl-OOP objects as primitives. Once you do this, Perl's versatility and speed are hard to beat.

Re: Awk: `Begin { ` Part 1

#86

Post by bingerman »

arminiusreturns wrote: post #45

My story about learning the power of awk was I was working on a dataset and had written a page or so Worth's of Perl... and kept bungling it. I emailed the scientist who wrote the paper whose technique I was emulating and he said, why not use awk, and gave me a basic example one liner, with a bit of modification it worked! Of course all that really means is I suck at Perl, but it was an eye opener for how powerful aw…

I learned awk before Perl and it helped a lot. I still use awk (and sed) regularly but for more complex tasks involving regexes and such Perl is my favourite.

Re: Awk: `Begin { ` Part 1

#88

Post by e12e »

forinti wrote: post #35
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

I parse a lot of CSV files and few are well-formed. People are careless when writing CSV files because they think it is simple: just put a comma between columns, right?

I just fixed a bug at work, parsing a rather opinionated csv file of products for a web shop. It had mostly good quoting, headers for columns - and uses semicolon for field separation (so, not technically csv, but..).

Funny thing was, a lot of the product names contained an ampersand (no problem there). But one product had an html entity encoded ampersand (&). I have no idea how that semicolon escaped, eh, escaping - but that one line suddenly had most of the columns off by one...

I can see how the entity got into the db (probably errant cutnpaste) - but I wonder at the csv writer that gleefully copied the extra separator to the csv export...

Re: Awk: `Begin { ` Part 1

#89

Post by protomyth »

firebacon wrote: post #10
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

There is https://tools.ietf.org/html/rfc4180 Most CSV files do not follow this standard of course. But you could normalize all CSV files to RFC4180 (or any other consistent format) as the first step of your processing pipeline.

I generally went for turning them into either tab-separated files or used the ASCII codes for record separator and its brethren depending on the job. I never wanted to touch CSV again after parsing it once.

Re: Awk: `Begin { ` Part 1

#90

Post by protomyth »

sidpatil wrote: post #33
necheffa wrote: post #32

Earlier quoted context omitted.

Everything you just described seems like reasons to not use CSV files in general, rather than not use AWK on CSV.

I'm disappointed we didn't use ASCII-separated values. https://news.ycombinator.com/item?id=7474600

Did it multiple times and it works great, but companies you send raw data to often get huffy. So, back it goes into CSV or some poorly defined fixed width text file.
Post reply on HN