ASCII Delimited Text – Not CSV or TAB delimited text

Post by groovy2shoes » Wed, Mar 26, 2014, 9:11 PM UTC

sheetjs wrote: post #12

mjn wrote: post #8

Alas, I don't think this works with the standard Unix tools, which is the main way I process tab-delimited text. Changing the field delimiter to whatever you want is fine, since nearly everything takes that as a parameter. But newline as record separator is assumed by nearly everything (both in the standard set of tools, and in the very useful Google additions found in http://code.google.com/p/crush-tools/ ). Google'…

Awk lets you set both: $ echo -n "1,2,3|4,5|6|7,8,9,0" | awk 'BEGIN{FS=","; RS="|"} {print NF, $0}' 3 1,2,3 2 4,5 1 6 4 7,8,9,0 In fact, you can also specify the output delimiters as well: $ echo -n "1,2,3|4,5|6|7,8,9,0" | awk 'BEGIN{FS=","; RS="|";OFS="foo";ORS="bar"} {print NF, $0}' 3foo1,2,3bar2foo4,5bar1foo6bar4foo7,8,9,0bar

    $ echo -n 'a^_1^^b^_2^^c^_3^^' |awk 'BEGIN{FS="^_"; RS="^^"} {print $1": "$2}'
    a: 1
    b: 2
    c: 3

Post by tbrownaw » Wed, Mar 26, 2014, 9:14 PM UTC

lowboy wrote: post #122

tbrownaw wrote: post #103

Earlier quoted context omitted.

It is strictly less expressive, because it can't handle nesting. This makes it inferior.

What would make one pair of ASCII characters (comma/linefeed or tab/linefeed) handle nesting any better than another pair (unit separator/record separator)?

Post by pjbrunet » Wed, Mar 26, 2014, 9:24 PM UTC

jrochkind1 wrote: post #138

scelerat wrote: post #44

Earlier quoted context omitted.

ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key. Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Control-C and Control-D, End of Text and End of…

> Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. And this is in fact how they show up in vim (or at least my fairly uncustomized vim), using ^ to stand for ctrl as was once conventional: `^_` and `^^` The legacy MARC binary format still used for library data uses ascii 29, 30, and 31 -- although for reasons with probably some bizarre historical definition uses them D…

Post by bruceboughton » Wed, Mar 26, 2014, 9:25 PM UTC

markeganfuller wrote: post #124

Crito wrote: post #109

Earlier quoted context omitted.

Qwerty keyboards intended specifically for the US market (opposed to the UK for instance) frequently do not have AltGr keys. The thinkpad I'm using right now doesn't have one, and the Das Professional I have next to me doesn't have it either. Come to think of it, I'm not sure if I've ever owned a keyboard with an AltGr key.. To get around this, I have taken to using xmodmap to turn my right alt key into altgr.

I'm surprised to learn that the Das doesn't have it.

Post by tbrownaw » Wed, Mar 26, 2014, 9:34 PM UTC

rwmj wrote: post #136

tbrownaw wrote: post #90

Earlier quoted context omitted.

you should never try to parse CSV yourself Why? Writing a correct parser is not significantly harder than figuring out how to interface to an existing parser library, and allows cool things like heuristic parsing of malformed files. OTOH it's shocking how many people can't write a correct CSV generator , even after being explicitly told what they're doing wrong (which is always either "you need to put quotes around t…

If you knew enough about CSV to be able to write a correct parser, then you'd know enough not to write one lightly. Here are some surprising valid CSV files: https://forge.ocamlcore.org/plugins/scmgit/cgi-bin/gitweb.cg... The test program in the same directory shows the semantic content of each.