ASCII Delimited Text – Not CSV or TAB delimited text
ronaldduncan.wordpress.com
ASCII Delimited Text – Not CSV or TAB delimited text
1–10 of 286 posts
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#2Now I feel silly for having glossed over the control characters since I was a kid. Those characters are decidedly useful on a machine level, though the benefit of CSV/TSV is that it's human friendly.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#3It doesn't solve the problem, although it does make it far less likely to run into it.
For a trivial example, try building an ASCII table using this format, with columns for numeric code, description, and actual character. You'll once again run into the whole escaping problem when you try to write out the row for character 31.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#4My guess is that TSV/CSV won out simply because anyone can easily type those characters from any standard keyboard on any platform.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#5Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-characters.html
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#6It doesn't solve the problem, although it does make it far less likely to run into it. For a trivial example, try building an ASCII table using this format, with columns for numeric code, description, and actual character. You'll once again run into the whole escaping problem when you try to write out the row for character 31.
Sure, but this is a very special application. whitespace and '"' are much more common in normal text, so using dedicated characters for telling entries apart should be superior in almost all cases.
(The problem reminds me of what it's like using "/" as a delimiter when using `sed` to edit file paths.)
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#7Makes sense, but it's practically a (very simple) binary storage format at that point. You can't count on being able to edit a document with control characters in a text editor. And I wouldn't trust popular spreadsheet software with it either.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#8Alas, I don't think this works with the standard Unix tools, which is the main way I process tab-delimited text. Changing the field delimiter to whatever you want is fine, since nearly everything takes that as a parameter. But newline as record separator is assumed by nearly everything (both in the standard set of tools, and in the very useful Google additions found in http://code.google.com/p/crush-tools/). Google's defaults are ASCII (or UTF-8) 0xfe for the field separator, and '\n' for the record separator. I guess that's a bit safer than tabs, but the kind of data I put in TSV really shouldn't have embedded tabs in a field... and I check to make sure it doesn't, because they're likely to cause unexpected problems down the line. Generally I want all my fields to be either numeric data, or UTF-8 strings without formatting characters.
Not to mention that one of the advantages of using a text record format at all is that you can view it using standard text viewers.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#9Protip - if it doesnt appear on keyboards, you can use ALT+DDD (DDD being 000 to 255) to enter a control character. For those on windows, drop into a command prompt and hold ALT while pressing 031 on the numpad. You will see it produce a ^_ character.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#10I've done this.
Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when somebody hand-edits the file.