Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
Awk: `Begin { ` Part 1
51–60 of 110 posts
Re: Awk: `Begin { ` Part 1
#52Here's a basic network calculator I wrote in awk once because I had to do these calculations on a minimal SunOS system that didn't have any other way to do math from the shell.
https://gist.github.com/jaysoffian/7bb70e1065f085b46a00
You never know when a bit of awk will save your bacon.
Re: Awk: `Begin { ` Part 1
#53Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
If I'm deep enough into a csv file that I'm worried about everything I noted above, I usually just load the data into Postgres and trust that their devs got it right. Pretty presentation is free at that point :-)
Re: Awk: `Begin { ` Part 1
#54Earlier quoted context omitted.
Heh, interesting, I actually assumed it would be oct 2020 to Jan 2021, and that you were describing an event already in progress.
Sure :-) even then, you have even more fun... imagine if the next item was Sep 1-Dec 1 and the list was known to be sorted! The parser would need to be stateful to be able to disambiguate!! Even worse (well, maybe similar, but more surprising): imagine if on Feb 28, you parse "Feb 29-Mar 1"... both of those dates could land in an entirely different year than "Feb 28-Mar 1" would, depending on whether the current year…
> I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that.
You can google "nlp date extraction" maybe? I've used libraries in the past that do this.
Note: they're far from being perfect. I ended up not using any, as each had their weird corner cases.
Here's an old one: http://natty.joestelmach.com/try.jsp#
I tried with: "first week of december to end of january"
And it gave me: Tue Dec 01 15:31:30 UTC 2020 Fri Jan 31 15:31:30 UTC 2020
Edit: this one seems more polished: http://nlp.stanford.edu:8080/sutime/process
Re: Awk: `Begin { ` Part 1
#55Earlier quoted context omitted.
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
Sqlite3 can import and export CSV via the CLI application. Both Ruby and Python provide CSV parsers in their standard library.
Thank you. I've used sqlite, but being able to query the data is not the same as being able to browse that data.
For instance, consider a project that I'm currently working on. Google Docs sheet exported to CSV. The file documents dozens of servers used as load balancers, websites, and database servers. The files has gone from MS Excel to Google Docs, and been maintained by four admins over 12 years. There are no columns per se, rather "sections of rows", each of which isn't even consistent in itself. One section is double columned, with the top row listing public IP address, login name, etc and the second row listing LAN IP address, login password, etc.
sqlite is useless on these files and frankly I've seen these things more often than well-formatted spreadsheets at every company I've ever worked with. Nobody but software developers treat spreadsheet columns as database fields.
Re: Awk: `Begin { ` Part 1
#56Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…
CSV to TSV via robust program, use AWK, and convert back if needed. Then you only have to remember the \\t\n\r escapes, not the whole quoting shenanigans.
Even better just use TSV. Tabs are better separator for real world data, you can strip tabs unless you need to store verbatim string data.
Re: Awk: `Begin { ` Part 1
#57Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…
I would love to have a command-line tool that reads CSV and has a ton of features to cover different quirks and errors, which can output cleaner formats that I can pipe into other command-line tools. csvkit [0] might be that tool; I discovered it after my last painful encounter with CSV files and haven't used it in anger yet. Among other things, it translates CSV to JSON, so you can compose it with jq. [0] https://cs…
I love these TSV utilities: https://github.com/eBay/tsv-utils Granted they're for "T"sv files, not "C"sv, but there's a handy `csv2tsv` utility included.
There's also `xsv`, from the author of `ripgrep`: https://github.com/BurntSushi/xsv
Re: Awk: `Begin { ` Part 1
#58[deleted]
Re: Awk: `Begin { ` Part 1
#59Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…
I have used awk a lot on csv files. I basically agree with your comments, for me the use case is preliminary exploration, not really much more. There is also the issue of one liners not really ending themselves to good coding practice generally, but it sure is great to dive into a file quickly. What I came here to say is that when I have a csv file I'm looking at with awk, thos first thing I do is a `wc -l`, then get…
You can parse it with awk though.
https://github.com/Nomarian/Awk-Batteries/blob/master/Units/...
gawk -f $AWK/Units/csv.awk -e '{print NF}'
Re: Awk: `Begin { ` Part 1
#60> He said, “If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk.” It's hyperlinked to the Gawk manual, but it seems likely he actually meant A, W & K's The Awk Programming Language (1988), which you could conceivably read in 2 hours, as it's a joy to read. I used it and The C Programming Language as exemplars of great documentation when…
It's one of my favorite technical books, with many examples of the application of the language, including the design of small languages and experimentation with algorithms. It is freely available at archive.org: https://archive.org/details/pdfy-MgN0H1joIoDVoIC7
Much thanks