ASCII Delimited Text – Not CSV or TAB delimited text • Hacker News Board

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#111

Post by Pxtl » Wed, Mar 26, 2014, 8:01 PM UTC

derefr wrote: post #92

Pxtl wrote: post #26

Earlier quoted context omitted.

> Sure - you can't, say, nest ASCII tables into one another due to this limitation. In hindsight it's too bad we don't have similar characters that follow a more sexpr-ish layout - say, ListStart, ListEnd, and Delimiter. Then you could tree them endlessly. If you wanted to be really fancy you could add an "assignmentSeparator" character to officially bless key-value-pairs and encompass a nice JSON-ish format, but Lis…

You know what's nicer than delimiting beginnings and ends of things? Length prefixing. Protocol message formats and data encoding formats both already know what they're going to say before they say it, and so know its octet length. The only reason to use delimiters, ever, is for user-modifiable data (e.g. source code) where you might want to insert or delete characters and have the containing block remain valid. ---…

The problem: I have never encountered length-prefixed data. Ever. Every data interchange file I've ever dealt with has been either delimited or fixed-width fields (and the widths are not defined anywhere in the file).

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#112

Post by Eleutheria » Wed, Mar 26, 2014, 8:04 PM UTC

commandar wrote: post #108

Eleutheria wrote: post #75

EDI is actually a wonderful and simple ASCII format for complex documents in use for over 30 years. The underlaying mapping formats for specific industries are a pain to parse but everything is easily formatted using stars or pipes as field separators ST|101 NAM|john|doe ADR|123 sunset blv|sunrise city|CA DAT|20140326|birthday Ah, the joy of simplicity.

HL7 used in healthcare is similar. Just grabbing the first few segments from the example message in the wiki article: MSH|^~\&|MegaReg|XYZHospC|SuperOE|XYZImgCtr|20060529090131-0500||ADT^A01^ADT_A01|01052901|P|2.5 EVN||200605290901||||200605290900 PID|||56782445^^^UAReg^PI||KLEINSAMPLE^BARRY^Q^JR||19620910|M||2028-9^^HL70005^RA99113^^XYZ|260 GOODWIN CREST DRIVE^^BIRMINGHAM^AL^35209^^M~NICKELL’S PICKLES^10000 W 100TH…

And the good thing is that you can generate very complex structured documents, not only lists like in cvs.

Of course xml and json do the same, but more verbose.

ASCII formats like EDI were invented when every byte in transmission counted.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#113

Post by elwin » Wed, Mar 26, 2014, 8:06 PM UTC

sopooneo wrote: post #95

raldi wrote: post #79

Earlier quoted context omitted.

...and that's why, when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character. Similarly, Ctrl-L clears the screen in most Unix apps because that generates the "form feed" character, and on a line printer or paper-based teletype terminal, "form feed" means to advance to th…

what about Ctrl-C to break out of stuff? Does that correspond to an ascii control character?

Ctrl-C stops the program because the TTY driver watches for it and sends an interrupt signal when it appears. It's a slightly different mechanism.

There are more details in "man 3 termios", but be warned, the TTY layer is not a pleasant subject for reading.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#114

Post by Roboprog » Wed, Mar 26, 2014, 8:08 PM UTC

CSV is a solved problem - RFC 4180: http://tools.ietf.org/html/rfc4180#section-2

As used by Lotus 1-2-3 and undoubtedly others before there was an Excel.

Example record:

    42,"Hello, world","""Quotes,"" he said.","new
    line",x

Now go write a little state machine to parse it... (hint: track odd/even quotes, for starters)

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#115

Post by cpwright » Wed, Mar 26, 2014, 8:11 PM UTC

mikestew wrote: post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

Parsing and outputting CSV can definitely be a pain, and there are a lot of corner cases. I think that this is certainly one of those times where rolling your own should be avoided if at all possible. I am a big fan of Text::CSV or the Python csv module, but even using Text::CSV have had different behavior from different versions.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#116

Post by thwarted » Wed, Mar 26, 2014, 8:13 PM UTC

raldi wrote: post #79

thwarted wrote: post #73

Earlier quoted context omitted.

Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Vim has bindings for some control key combos, which is why you can't type them directly. I can type control-_ without issue, but to get an ASCII 30 (RS) to show up, I have…

...and that's why, when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character. Similarly, Ctrl-L clears the screen in most Unix apps because that generates the "form feed" character, and on a line printer or paper-based teletype terminal, "form feed" means to advance to th…

when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character.

This is slightly different than being able to actually generate those characters as input. If you put a control-d in a file, nothing will stop reading the file when it sees the control-d, it's just another byte. readline knows how to interpret a bunch of control characters. There's also the terminal driver that has interpretations, which you can see with `stty -a`. When the tty layer sees a control-d or a control-c, it closes stdin or generates SIGINT, respectively. But even this can be dependent on if you're on a pty or attached directly to a serial line. There's nothing special about the mapping of control-d to end-of-transmission character, that's just how the layer that sees it is interpreting it. You can, for example, change how control-c is interpreted by `stty intr ^B`, which will make control-b generate SIGINT. And there's a way to put the terminal driver in complete transparent/passthru mode.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#117

Post by jarrett » Wed, Mar 26, 2014, 8:13 PM UTC

mbauman wrote: post #28

Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…

> Precisely what makes them valuable is their difficulty to type or display.

Exactly. A naive[1] character-delimited format is only robust when its delimiter isn't on the keyboard. Which means it's practical to read and write only in specialized applications, not in a text editor. Which sort of defeats the point of a character-delimited format anyway. If your use cases are constrained to specialized applications, you may as well just use JSON or something similar, instead of a character-delimited format. (I'm imagining a world where we get to pick our ideal formats, not one where Excel happens to have an almost-working CSV implementation.)

[1] By "naive" I mean one where the format specification describes record and line characters only. Thus the format has no escaping system.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#118

Post by hnal943 » Wed, Mar 26, 2014, 8:15 PM UTC

markeganfuller wrote: post #100

Pxtl wrote: post #37

Earlier quoted context omitted.

> Precisely what makes them valuable is their difficulty to type. Again, if they'd caught on you'd imagine there would be some eventual convention in text-editors for what keybind would be used to enter them. Too bad the AltGr key (intended for entering rarely-used glyphs) doesn't appear on pure-English keyboards.

I'm looking at my English keyboard and it has Alt Gr...

Living in the US, I've never heard of an AltGr key until this discussion.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#119

Post by jloughry » Wed, Mar 26, 2014, 8:19 PM UTC

sopooneo wrote: post #95

raldi wrote: post #79

Earlier quoted context omitted.

...and that's why, when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character. Similarly, Ctrl-L clears the screen in most Unix apps because that generates the "form feed" character, and on a line printer or paper-based teletype terminal, "form feed" means to advance to th…

what about Ctrl-C to break out of stuff? Does that correspond to an ascii control character?

Ctrl-C was originally "End of Text".

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#120

Post by stingraycharles » Wed, Mar 26, 2014, 8:20 PM UTC

mbauman wrote: post #28

Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…

Is this a problem that's solved with a "movement" ? I'm willing to contribute parsers and perhaps a text editor plugin if other people are willing to chip in. But indeed, this is a problem that's solvable by making the tools support it. Since most of the tools are OSS, this is mostly solvable.