Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk: `Begin { ` Part 1

jemma.dev

Post reply on HN
Print view View original

91–100 of 110 posts

Re: Awk: `Begin { ` Part 1

#91

Post by [deleted] »

e12e wrote: post #88
forinti wrote: post #35

Earlier quoted context omitted.

I parse a lot of CSV files and few are well-formed. People are careless when writing CSV files because they think it is simple: just put a comma between columns, right?

I just fixed a bug at work, parsing a rather opinionated csv file of products for a web shop. It had mostly good quoting, headers for columns - and uses semicolon for field separation (so, not technically csv, but..). Funny thing was, a lot of the product names contained an ampersand (no problem there). But one product had an html entity encoded ampersand (&). I have no idea how that semicolon escaped, eh, escapi…

[deleted]

Re: Awk: `Begin { ` Part 1

#92

Post by im3w1l »

sidpatil wrote: post #33
necheffa wrote: post #32

Earlier quoted context omitted.

Everything you just described seems like reasons to not use CSV files in general, rather than not use AWK on CSV.

I'm disappointed we didn't use ASCII-separated values. https://news.ycombinator.com/item?id=7474600

I don't see what issue they solve. You still need escaping, because eventually you will want to put an ascii-separated value into your spreadsheet.

Re: Awk: `Begin { ` Part 1

#93

Post by drran »

stevebmark wrote: post #84
drran wrote: post #74

Earlier quoted context omitted.

> As of version 2.0.9, there's no need for any external dependency. Python itself (3.7), and any needed libraries are self-contained inside the installation, isolated from the rest of your system. Oh, sh*t. I will look for something else.

Is your concern that you don't want to use a library that depends on python?

Nope. I don't want to use the tool which depends on an unsupported copy of Python.

Re: Awk: `Begin { ` Part 1

#94

Post by ensiferum »

Awk and shell scripting is the rabbit hole that is best avoided IMHO. Often it's just easier to crank out a "real" program using your lost familiar programming language or at very least python instead of starting to work with shell or awk scripting.

Re: Awk: `Begin { ` Part 1

#95

Post by mehrdadn »

enriquepablo wrote: post #44
mehrdadn wrote: post #26

Earlier quoted context omitted.

Sure :-) even then, you have even more fun... imagine if the next item was Sep 1-Dec 1 and the list was known to be sorted! The parser would need to be stateful to be able to disambiguate!! Even worse (well, maybe similar, but more surprising): imagine if on Feb 28, you parse "Feb 29-Mar 1"... both of those dates could land in an entirely different year than "Feb 28-Mar 1" would, depending on whether the current year…

I would say that you are mixing up "parsing" and "calendaring (or something of the sort)". As far as I understand parsing is syntactic analysis, i.e. going from a linear structure to a more complex structure (usually a tree); it should not add to the tree anything that was not in the linear structure. It shouldn't consider a semantic context (such as the current date) to produce an ast.

By parsing I mean it in the usual sense for a date... strptime, Date.Parse, etc... i.e. turning a string to a date (or multiple dates). You can call it something else if you'd like.

Re: Awk: `Begin { ` Part 1

#96

Post by mehrdadn »

monsieurbanana wrote: post #54
mehrdadn wrote: post #26

Earlier quoted context omitted.

Sure :-) even then, you have even more fun... imagine if the next item was Sep 1-Dec 1 and the list was known to be sorted! The parser would need to be stateful to be able to disambiguate!! Even worse (well, maybe similar, but more surprising): imagine if on Feb 28, you parse "Feb 29-Mar 1"... both of those dates could land in an entirely different year than "Feb 28-Mar 1" would, depending on whether the current year…

> I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that. You can google "nlp date extraction" maybe? I've used libraries in the past that do this. Note: they're far from being perfect. I ended up not using any, as each had…

Cool, thanks for the tip!

Re: Awk: `Begin { ` Part 1

#97

Post by lcuff »

heresie-dabord wrote: post #85
mauvehaus wrote: post #3

Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…

> Csv is not standardized and the quoting rules are weird (and not standardized) There is almost always some normalisation ("scrubbing") needed to prep a CSV. But CSV is viable and awk can rip through massive amounts of data. It is a brilliant and powerful tool. > It's small enough to fit in your brain, unlike Perl Perl is also brilliant and powerful. Setting aside the bigotry of people who dislike sigils and using b…

I suffered with Perl until I stopped assuming I could learn it by reading code, and read the Camel book. At about page 85 it says something approximating: "Everything in Perl happens in a context (scalar or Array) and until you realize that, you will be miserable." It still didn't come easy: I've realized we all have different minds. My mind doesn't find Perl natural, but loves Ruby. YMMV.

Re: Awk: `Begin { ` Part 1

#98

Post by lmns »

dotancohen wrote: post #55
lmns wrote: post #24

Earlier quoted context omitted.

Sqlite3 can import and export CSV via the CLI application. Both Ruby and Python provide CSV parsers in their standard library.

Thank you. I've used sqlite, but being able to query the data is not the same as being able to browse that data. For instance, consider a project that I'm currently working on. Google Docs sheet exported to CSV. The file documents dozens of servers used as load balancers, websites, and database servers. The files has gone from MS Excel to Google Docs, and been maintained by four admins over 12 years. There are no col…

I know, there is no easy fix for that.

Re: Awk: `Begin { ` Part 1

#99

Post by lmns »

7thaccount wrote: post #27
lmns wrote: post #24

Earlier quoted context omitted.

Sqlite3 can import and export CSV via the CLI application. Both Ruby and Python provide CSV parsers in their standard library.

Doesn't it convert everything to strings including numbers? Or can you specify types?

You have to provide a schema anyway, so you can treat specific columns as integers.

Re: Awk: `Begin { ` Part 1

#100

Post by dbro »

MrDOS wrote: post #66
dkarl wrote: post #14

Earlier quoted context omitted.

I would love to have a command-line tool that reads CSV and has a ton of features to cover different quirks and errors, which can output cleaner formats that I can pipe into other command-line tools. csvkit [0] might be that tool; I discovered it after my last painful encounter with CSV files and haven't used it in anger yet. Among other things, it translates CSV to JSON, so you can compose it with jq. [0] https://cs…

At my last employer, I built a filter program, creatively called CSVTools[0], to do something like this. One piece of the project parses CSVs and replaces the commas/newlines (in an escaping- and multiline-aware manner, of course) with ASCII record/unit separator characters[1] (0x1E and 0x1F); the other piece converts that format back into well-formed CSV files. I usually used this with GNU awk, and reconfigured RS[2…

Good idea! Looks similar to something I wrote called csvquote https://github.com/dbro/csvquote , which enables awk and other command line text tools to work with CSV data that contains embedded commas and newlines.
Post reply on HN