Before using AWK we need to sanitize the data files;
Awk: `Begin { ` Part 1
101–110 of 110 posts
Re: Awk: `Begin { ` Part 1
#102Earlier quoted context omitted.
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
It depends what I'm doing, but I typically use one of the following (starting with the most frequent): 1. IPython with Pandas if I expect to do any in-depth exploration/manipulation of the data. 2. A short Python script like `python -c 'import csv, sys; r = csv.reader(sys.stdin); ...' 3. https://github.com/BurntSushi/xsv if I want to quickly munge some data or extract a field. 4. Gnumeric if I want a GUI. 5. https://…
> 5. https://github.com/andmarti1424/sc-im if I want
> a TUI. This is closest to what you were asking for.
Thank you.
Re: Awk: `Begin { ` Part 1
#103Awk and shell scripting is the rabbit hole that is best avoided IMHO. Often it's just easier to crank out a "real" program using your lost familiar programming language or at very least python instead of starting to work with shell or awk scripting.
Awk is a full blown programming language though. comparable to python but allowing for much more compact code. Though I completely agree on avoiding shell. the only good shell I've seen is rc. but the relying on external utilities is a double edged sword. forking is too costly over time.
Re: Awk: `Begin { ` Part 1
#104Earlier quoted context omitted.
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
If I'm deep enough into a csv file that I'm worried about everything I noted above, I usually just load the data into Postgres and trust that their devs got it right. Pretty presentation is free at that point :-)
Postgress or even sqlite are great if the CSV is formatted like a database, with each column a distinct field. Alas, most spreadsheets created by non-software-devs are not so strict.
I need to browse the cells in a something like curses.
Re: Awk: `Begin { ` Part 1
#105Nice introduction (based on both tutorials). One suggestion would be to use -v FPAT='[^,]*|"[^"]+"' instead of BEGIN { FPAT = "[^,]*|\"[^\"]+\"" } > If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk. I can't work my head around this quote. That's a ridiculous claim. Even for a experienced programmer, learning a new programming languag…
This HN post inspired me to pick up and work through "The AWK Programming Language", so I have been doing that for the past two days. I'm currently at the beginning of chapter 5 (of 8).
That quote is definitely ridiculous. He may have been referring to just chapter 2, in which the "whole language" is outlined. But later chapters also take time, there even is an exercise about writing an assembler in AWK as well as implementing multiple algorithms.
Some stuff was already somewhat familiar to me from studying/practicing C earlier this year.
Overall a 10/10 book so far, but takes time and patience for sure!
Re: Awk: `Begin { ` Part 1
#106Standard awk warning: it's tempting to try to use awk on csv files. You'll even get good results on simple csv files that leave you encouraged to go further. Don't. Csv is not standardized and the quoting rules are weird (and not standardized). If you can live with a certain amount of loss of fidelity in your output, you can get away with using awk. If you want a coarse prototype, use awk. If you need robust, product…
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
Just a reply to myself, for anybody else looking for an application to browse spreadsheets on the CLI.
There are a few CLI applications for working with data stored in a CSV file, so long as the data is formed as a database: each row as a data entry, each column as a database field. The single CLI app that run on Linux for browsing a spreadsheet that is not formed as a database, SC-IM, does not support searching for arbitrary text!
Therefore I am writing osheet: A Linux CLI app for browsing a spreadsheet. It currently supports only XLS files, because that is what I need. It currently crashes a lot. It currently has no features other than searching. It won't even scroll yet to display all the rows and columns. I just banged it out last night when I should have been sleeping. But I'm working on it, you are invited to test it, report bugs, and pull requests are of course welcome. It's written in Python and Curses.
Re: Awk: `Begin { ` Part 1
#107Earlier quoted context omitted.
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
It depends what I'm doing, but I typically use one of the following (starting with the most frequent): 1. IPython with Pandas if I expect to do any in-depth exploration/manipulation of the data. 2. A short Python script like `python -c 'import csv, sys; r = csv.reader(sys.stdin); ...' 3. https://github.com/BurntSushi/xsv if I want to quickly munge some data or extract a field. 4. Gnumeric if I want a GUI. 5. https://…
Just a heads-up, if you're interested. Only SC-IM was fit for the non-database-format files I need to work with. But it does not have a search function! So I'm writing an app in Python for browsing spreadsheets:
https://github.com/dotancohen/osheet
Re: Awk: `Begin { ` Part 1
#108Earlier quoted context omitted.
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
Sqlite3 can import and export CSV via the CLI application. Both Ruby and Python provide CSV parsers in their standard library.
Just a heads-up, if you're interested. I'm writing an app in Python for browsing spreadsheets:
Re: Awk: `Begin { ` Part 1
#109Earlier quoted context omitted.
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
There is https://github.com/BurntSushi/xsv but it doesn't seem to print with borders between cells.
That's good for data formatted like a database. I need something to just browse the cells, so I'm writing an app in Python for browsing spreadsheets:
Re: Awk: `Begin { ` Part 1
#110Earlier quoted context omitted.
Segue to something that I've been fighting with for years: Do you have a suggestion for working with CSVs in the CLI? Something that will show a table, with a single pixel border between cells, that allows searching for a value, copying a cell?
https://www.visidata.org/
That's good for data formatted like a database. I need something to just browse the cells, so I'm writing an app in Python for browsing spreadsheets: