ASCII Delimited Text – Not CSV or TAB delimited text • Hacker News Board

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#71

Post by npongratz » Wed, Mar 26, 2014, 6:36 PM UTC

scelerat wrote: post #44

Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key. Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Control-C and Control-D, End of Text and End of…

Vim will let you type the record separator, but you may have to precede it with Ctrl-V. In other words, type the Ctrl-V Ctrl-^ key sequence.

At least, that worked for me. Interesting that the unit separator does not have the same requirement to precede it with Ctrl-V.

More info in the vim docs:

http://vimdoc.sourceforge.net/htmldoc/insert.html#ins-specia...

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#72

Post by mbq » Wed, Mar 26, 2014, 6:37 PM UTC

dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

Still, most *SV parsers can use arbitrary char or even regexp for a separator.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#73

Post by thwarted » Wed, Mar 26, 2014, 6:38 PM UTC

scelerat wrote: post #44

Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key. Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Control-C and Control-D, End of Text and End of…

Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance.

Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance.

Vim has bindings for some control key combos, which is why you can't type them directly. I can type control-_ without issue, but to get an ASCII 30 (RS) to show up, I have to type control-v first (just like when at the shell prompt).

Useful hint: if you look at the output of man ascii (at least on linux with man-pages-3.22) find the control character you want to type in the left column and look in the same row on in the right column to find the letter/key to use with the control key.

For example, to type NAK, it's . Vertical tab is :

   $ echo  | od -c
   0000000  \v  \n

This is useful for control characters that don't have backslash escape expansions.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#74

Post by wiredfool » Wed, Mar 26, 2014, 6:40 PM UTC

Pxtl wrote: post #37

mbauman wrote: post #28

Earlier quoted context omitted.

It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…

> Precisely what makes them valuable is their difficulty to type. Again, if they'd caught on you'd imagine there would be some eventual convention in text-editors for what keybind would be used to enter them. Too bad the AltGr key (intended for entering rarely-used glyphs) doesn't appear on pure-English keyboards.

I'd bet that they were on the eleventyzillion key IBM terminal keyboards.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#75

Post by Eleutheria » Wed, Mar 26, 2014, 6:41 PM UTC

EDI is actually a wonderful and simple ASCII format for complex documents in use for over 30 years.

The underlaying mapping formats for specific industries are a pain to parse but everything is easily formatted using stars or pipes as field separators

    ST|101
    NAM|john|doe
    ADR|123 sunset blv|sunrise city|CA
    DAT|20140326|birthday

Ah, the joy of simplicity.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#76

Post by wiredfool » Wed, Mar 26, 2014, 6:42 PM UTC

mbreese wrote: post #51

Pxtl wrote: post #45

Earlier quoted context omitted.

I don't think CSV is a rigidly-defined format - I'm sure some implementations will happily accept spaces between the comma and the opening quote.

This is one reason why I prefer tab delimited files... The format is pretty simple with few edge cases. There really only one caveat to worry about - fields with tab characters. And that's extremely rare in my field. CSV on the other hand has a few different variations.

The thing that gets to me is that the CSV RFC is over 4000, and is less than 10 years old.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#77

Post by dxbydt » Wed, Mar 26, 2014, 6:43 PM UTC

Istof wrote: post #66

dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

TLDR: it's superior, but don't do it...

That's unfortunately a very accurate summary:) Real estate data, traffic data, weather data, population demographics, stock prices, tweets - I've parsed all that and more. Every one of them was a giant Tsv (except finance ones, which were csv's because Excel). Say you purchase the database containing every single home sold/bought in California for past decade. That's 11 20gb Tsv's with 250 tab separated columns plus 1 data dictionary which tells you what each of the 250 columns mean.That's what Reology sells you - gigantic txt files with tabs, that are easy to handle with awk, cut, sed and more.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#78

Post by sigil » Wed, Mar 26, 2014, 6:43 PM UTC

Meh. What if some data has ASCII 28-31 in it? If you're not using a "real" escaping mechanism, and instead relying on the assumption that certain characters don't appear in your data, then I don't see anything wrong with using \t and \n (ie TSV). Either way, you know your data, and you're using whatever fits it best.

If you need something that's never, ever going to break for lack of escaping, might I suggest doing percent-encoding (aka url encoding) on tabs ("%09"), newlines ("%0a") and percent characters ("%25")? Percent encoding and decoding can be made very fast, is recognizable to most developers, and can be used to escape and unescape anything, including unicode characters. Unlike C-escaping, which doesn't generalize and accommodate these things nearly so well.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#79

Post by raldi » Wed, Mar 26, 2014, 6:44 PM UTC

thwarted wrote: post #73

scelerat wrote: post #44

Earlier quoted context omitted.

ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key. Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Control-C and Control-D, End of Text and End of…

Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Vim has bindings for some control key combos, which is why you can't type them directly. I can type control-_ without issue, but to get an ASCII 30 (RS) to show up, I have…

...and that's why, when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character.

Similarly, Ctrl-L clears the screen in most Unix apps because that generates the "form feed" character, and on a line printer or paper-based teletype terminal, "form feed" means to advance to the next page -- a nice empty clear piece of paper.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#80

Post by slavik81 » Wed, Mar 26, 2014, 6:47 PM UTC

This seems a little better than it is. Those control characters are appealing because they're rarely used. Making them important by using them in a common data exchange format will dramatically increase the rate at which you find them in the data you're trying to store.

Ultimately, this is a language problem. If we invent new meta-language to describe data, we're going to use it when creating content. That means the meta-language will be used in regular language. Which means you're going to have to transform it when moving it into or out of that delimited file.

There is no fixed-length encoding you can use to handle meta-information without imposing restrictions on the content. You're always going to end up with escape sequences.