Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

31–40 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#31

Post by htp »

Took some time to figure out how to type these on a Mac:

1. Go to System Preferences => Keyboard => Input Sources

2. Add Unicode Hex Input as an input source

3. Switch to Unicode Hex Input (assuming you still have the default keyboard shortcuts set up, press Command+Shift+Space)

4. Hold Option and type 001f to get the unit separator

5. Hold Option and type 001e to get the record separator

6. (Hold Option and type a character's code as a 4-digit hex number to get that character)

Sadly, this doesn't seem to work everywhere throughout the OS- I can get control characters to show up in TextMate, but not in Terminal.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#32

Post by Balgair »

Haha! Oh wow, I just finished a project dealing with this exactly. The obvious problem is that most editors make dealing with the non-standard keyboard keys very difficult. As a consequence, most programs (Python, MatLab, etc) really don't like anything below 0x20. I was reading in binary through a serial port, and then storing data for records and processing. Any special character got obliterated in the transfer to MatLab, Python, etc. I ended up storing it as a very long HEX string and then parsing that sucker. I'd have loved to use special characters to have it auto sort into rows and columns, but that meant having it also escape things and wreck programs. Ces la vie.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#34

Post by chrisBob »

binarymax wrote: post #9

Protip - if it doesnt appear on keyboards, you can use ALT+DDD (DDD being 000 to 255) to enter a control character. For those on windows, drop into a command prompt and hold ALT while pressing 031 on the numpad. You will see it produce a ^_ character.

On what computer? Is this windows only? Now that I use a mac this might be the only thing I miss from windows computers.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#35

Post by eli »

joosters wrote: post #23
mikestew wrote: post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

Interesting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as. Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, s…

Depends on your platform. Even today a Unix/Linux line ending is almost always just LF.

See: http://en.wikipedia.org/wiki/Newline

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#36

Post by leoc »

Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

Yes, but nowadays this is just another bootstrap problem: editors don't support them because no documents use them, no documents use them because editors don't support them, and users scream and bawl because their cheese has moved. Using an editor with some good support for control-character-separated data would be pleasant, more pleasant than the usual experience of fiddling with CSV in a text editor. That said, another nasty aspect of the bootstrap process would be running control-character-separated documents through standard text-processing tools and pipelines and finding out how many of them strip non-tab control characters, turn them to spaces or mangle them pseudo-randomly.

I think the more fundamental problem with using the ASCII control characters is that there's no way (or rather, no obvious and standard way AFAICS) to use them to define (most) recursive data structures. That's not a problem when you stick to simple CSV-like arrays-of-arrays, but it's enough of a restriction to make one wonder if bringing ASCII back is worth the battles.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#37

Post by Pxtl »

mbauman wrote: post #28
Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…

> Precisely what makes them valuable is their difficulty to type.

Again, if they'd caught on you'd imagine there would be some eventual convention in text-editors for what keybind would be used to enter them. Too bad the AltGr key (intended for entering rarely-used glyphs) doesn't appear on pure-English keyboards.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#38

Post by thebelal »

co_dh wrote: post #11

How do you enter them ? in console, in editor? Since they are invisible, how do you find if you have entered a wrong character?

You can type them in console and vim with

    File separator - C-v C-\
    Group separator - C-v C-5
    Record separator - C-v C-6
    Unit separator - C-v C-7
They are all visible characters in both vim and emacs by default.

You can see them on the terminal with `cat -v`

It would be nice if more tools were built to take advantage of these characters, but there are some that do.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#39

Post by Pxtl »

leoc wrote: post #36
Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

Yes, but nowadays this is just another bootstrap problem: editors don't support them because no documents use them, no documents use them because editors don't support them, and users scream and bawl because their cheese has moved. Using an editor with some good support for control-character-separated data would be pleasant, more pleasant than the usual experience of fiddling with CSV in a text editor. That said, ano…

Absolutely. Too bad they went for a flat format instead of a sexpr-like format. You wouldn't even need 4 characters - just start,end, and separator.
Post reply on HN