Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

41–50 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#41

Post by iclelland »

joosters wrote: post #23
mikestew wrote: post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

Interesting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as. Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, s…

It's more likely that it's because they are two separate physical actions (returning the head to the left, and advancing the paper one line). They could be used independently: You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again.

A carriage-return operation takes much longer than a single character, or even two or three. It doesn't make sense to issue two characters just to take up time. The printers always had to have some internal buffer memory (and handshaking over the communication lines to say when the buffer is full) in order not to lose any characters.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#42

Post by dfc »

JoshTriplett wrote: post #22

Leaving aside the pain of displaying and typing such characters... > Then you have a text file format that is trivial to write out and read in, with no restrictions on the text in fields or the need to try and escape characters. Phrases like that lead to lovely security bugs.

The thing that leads to "lovely security bugs" is the nonchalant mindset; it has nothing to do with the simple text format. The same attitude paired with ASN.1 data has caused just as many vulnerabilities.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#43

Post by mmasashi »

It does not solve the problem. Here is the points which I think.

1. Control characters are not supported in the almost of text editors. 2. Control characters are not human friendly. 3. The text may contain control characters in the field value.

In any formats, we cannot avoid the escape characters, so even I think CSV/TSV format is reasonable.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#44

Post by scelerat »

Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key.

Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance.

Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance.

Control-C and Control-D, End of Text and End of Transmission, still have utility in most shells, and it goes back directly to these ASCII control characters.

https://www.cs.tut.fi/~jkorpela/chars/c0.html

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#45

Post by Pxtl »

olalonde wrote: post #30
mikestew wrote: post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

Agreed. A few days ago, I learned that spaces are illegal between separators in CSV. For example, `"val1", "val2"` is illegal (it should be `"val1","val2"`). Kind of unintuitive given that in most languages, non-delimited spaces are insignificant.

I don't think CSV is a rigidly-defined format - I'm sure some implementations will happily accept spaces between the comma and the opening quote.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#46

Post by [deleted] »

Pxtl wrote: post #37
mbauman wrote: post #28

Earlier quoted context omitted.

It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…

> Precisely what makes them valuable is their difficulty to type. Again, if they'd caught on you'd imagine there would be some eventual convention in text-editors for what keybind would be used to enter them. Too bad the AltGr key (intended for entering rarely-used glyphs) doesn't appear on pure-English keyboards.

[deleted]

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#48

Post by Pxtl »

scelerat wrote: post #44
Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key. Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Control-C and Control-D, End of Text and End of…

Huh. I wonder how many of those are completely disused, or would be meaningless in a file?

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#49

Post by quesera »

htp wrote: post #31

Took some time to figure out how to type these on a Mac: 1. Go to System Preferences => Keyboard => Input Sources 2. Add Unicode Hex Input as an input source 3. Switch to Unicode Hex Input (assuming you still have the default keyboard shortcuts set up, press Command+Shift+Space) 4. Hold Option and type 001f to get the unit separator 5. Hold Option and type 001e to get the record separator 6. (Hold Option and type a c…

In Terminal, they are:

  FS: Control-\ 0x1c (field sep)
  GS: Control-] 0x1d (group sep)
  RS: Control-^ 0x1e (record sep)
  US: Control-_ 0x1f (unit sep)
(These control key equivalents have always been the canonical keystrokes to generate the codes)

But they have to be preceded by a Control-V (like in vi) to be treated as input characters. Control-V is the SYN code (synchronous idle), but has no special meaning in an interactive context, which is presumably why it was chosen.

The full set of control codes (0x00 - 0x1f) and their historical meanings are why Apple added the open/closed Apple keys, eventually the Command key. They wanted a set of keystrokes that were unambiguously distinct from the data stream.

Control-S, e.g., will pause text output in the Terminal (also xterm, etc). This was super useful in the days before scrollback. :) Control-Q to resume (actually flush all the buffered output).

Overloading Control sequences was an unforgivable sin committed by Microsoft.

...if I remember the history correctly, Apple decided that having both open/closed Apple keys was confusing, and having the Apple logo on the keyboard was tacky, so they renamed the key for the Mac, and Susan Kare selected a new glyph, which is a Scandinavian "point of interest" wayfinding symbol.

...as a further aside, Control-N and Control-O are the cause of the bizarre graphical glyphs you sometimes see if you do something silly like cat a binary file. Control-N initiates the character set switch, and Control-O restores it. This can be used to fix your Terminal when things go awry. Most people just close the window, but I hate losing history. :)

0x20 - 0x74, unshifted:

   !"#$%&'()*+,-./0123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
0x20 - 0x74, shifted:

   !"#$%&'()*→←↑↓/▮123456789:;?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_◆▒bcde°±▒☃┘┐┌└┼⎺⎻─⎼⎽├┤┴┬│≤≥π≠£·
...works in Firefox. YMMV.

Terminal-charset-quickfix: at shell, type "echo ^O". To get the literal ^O, use Control-V then Control-O.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#50

Post by derefr »

iclelland wrote: post #41
joosters wrote: post #23

Earlier quoted context omitted.

Interesting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as. Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, s…

It's more likely that it's because they are two separate physical actions (returning the head to the left, and advancing the paper one line). They could be used independently: You could print a line in bold , for instance, by issuing a CR without an LF and then printing the same line again. A carriage-return operation takes much longer than a single character, or even two or three. It doesn't make sense to issue two…

> You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again.

True, but mildly redundant: "overprinting" was explicitly the purpose of 0x08 backspace (which had nothing, originally, to do with 0x7F deletion.)

Post reply on HN