mind. blown.
ASCII Delimited Text – Not CSV or TAB delimited text
91–100 of 286 posts
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#92Earlier quoted context omitted.
For CSV forbidding commas in data is not practical. For ASCII delimiters, forbidding ASCII delimiters in data is practical. Sure - you can't, say, nest ASCII tables into one another due to this limitation. But for simple structure, it doesn't hurt to have ASCII separators in the toolbox. The only big problem I see is that they're rendered as invisible characters, which will make debugging harder. If we wouldn't have…
> Sure - you can't, say, nest ASCII tables into one another due to this limitation. In hindsight it's too bad we don't have similar characters that follow a more sexpr-ish layout - say, ListStart, ListEnd, and Delimiter. Then you could tree them endlessly. If you wanted to be really fancy you could add an "assignmentSeparator" character to officially bless key-value-pairs and encompass a nice JSON-ish format, but Lis…
You know what's nicer than delimiting beginnings and ends of things? Length prefixing. Protocol message formats and data encoding formats both already know what they're going to say before they say it, and so know its octet length.
The only reason to use delimiters, ever, is for user-modifiable data (e.g. source code) where you might want to insert or delete characters and have the containing block remain valid.
---
And now, a fun tangent, to prove that how deeply-rooted this confusion is in CS: user-modifiable data was originally the sole use-case for \0-terminated "C strings" in C.
C has two separate types which get conflated nowadays: char arrays, and \0-terminated strings. Most "strings"--as we'd expect to find them in other languages--were, in C, actually char arrays: you knew their length, either because they were string literals and you could sizeof them, or because you had #defined both FOO and FOO_LEN, or because you had just allocated len bytes on the heap for foo, so you could just pass len along with foo. Because you knew their length, you didn't need to use the string.h functions to manipulate them. It was idiomatic (and perfectly-safe) C, when dealing with char arrays, to just iterate through them with a for loop.
The concept of \0-termination, and thus what we think of as "C strings", only applied to string buffers: fixed-size, stack-allocated, uninitialized char arrays. The string.h functions are all meant to be employed to manipulate string buffers, and the \0 is intended to mark where the buffer stops being useful data, and starts being uninitialized garbage.
The strings in string buffers had short lifetimes, and didn't usually outlive the stack frame the buffer was declared in. Generally, you'd declare a string buffer, populate it using some combination of string literals, strcat(3), sprintf(3), and system calls, and then pass the string--still sitting inside the buffer--to a system call like fstat(2) to get what you're really after. That would be the end of the both string buffer's, and the string's, lifetime.
If you ever did want to preserve the contents of a string buffer into something you could pass around, though, this would be idiomatic:
int give_me_a_path_string(char **out)
{
char buf[MAX_PATH];
/* ... */
int len = strlen(buf);
*out = memcpy(malloc(len), buf, len);
return len;
}
So:
• C has "C strings", but they were only intended as buffers.
• C also has "char arrays", which are really what you should think of as C's equivalent to a "string" datatype. char arrays, not "C strings", are the fundamental data structure for representing and persisting strings in C.
• char arrays are less like "C strings" than they are like Pascal strings: they come in two parts, a block of memory N chars wide, and an int containing N. You don't examine the block to determine the length; the length is explicit.
• Pascal (and thus most modern languages with strings) put both the length and the character-block on the heap as a unit. C puts the character-block on the heap, but puts the length on the stack. This is more efficient under C's Unix-rooted assumptions: you need the length on the stack if you want to work with it to immediately shove the string through a pipe.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#93Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…
Or just google for "ASCII table" and open the first hit:
But your link does have more in-depth explanations (including historical info) for some of the control characters.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#94Earlier quoted context omitted.
To overprint a whole line using 0x08, you'd need one 0x08 for each character in the line. So an N-character line overprinted that way would take N 3 characters in memory. Using CR, you'd need N 2 + 1 characters.
Even ignoring additional time needed to send those extra characters, it also was a lot faster than those control-H's, and (I guess) caused way less wear on your printer.
That presumes you're going like A^HAB^HB; you could just emit foo, and then strlen(foo) ^Hs.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#95Earlier quoted context omitted.
Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Vim has bindings for some control key combos, which is why you can't type them directly. I can type control-_ without issue, but to get an ASCII 30 (RS) to show up, I have…
...and that's why, when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character. Similarly, Ctrl-L clears the screen in most Unix apps because that generates the "form feed" character, and on a line printer or paper-based teletype terminal, "form feed" means to advance to th…
what about Ctrl-C to break out of stuff? Does that correspond to an ascii control character?
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#96Earlier quoted context omitted.
Interesting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as. Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, s…
It's more likely that it's because they are two separate physical actions (returning the head to the left, and advancing the paper one line). They could be used independently: You could print a line in bold , for instance, by issuing a CR without an LF and then printing the same line again. A carriage-return operation takes much longer than a single character, or even two or three. It doesn't make sense to issue two…
"You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again."
Last I checked, this still works even on laser printers (at least on a LaserJet), when sending data to it as plain text. It's not actually printing over itself, but it knows to make the repeated characters bold.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#97I will be sure to use this if I ever encounter data that is guaranteed to be pure ASCII again.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#98It doesn't solve the problem, although it does make it far less likely to run into it. For a trivial example, try building an ASCII table using this format, with columns for numeric code, description, and actual character. You'll once again run into the whole escaping problem when you try to write out the row for character 31.
For CSV forbidding commas in data is not practical. For ASCII delimiters, forbidding ASCII delimiters in data is practical. Sure - you can't, say, nest ASCII tables into one another due to this limitation. But for simple structure, it doesn't hurt to have ASCII separators in the toolbox. The only big problem I see is that they're rendered as invisible characters, which will make debugging harder. If we wouldn't have…
For ASCII delimiters, forbidding ASCII delimiters in data is practical.
Data formats that can't handle recursion are never practical. They only work until they don't, at which point they're entrenched and impossible to replace.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#99Meh. What if some data has ASCII 28-31 in it? If you're not using a "real" escaping mechanism, and instead relying on the assumption that certain characters don't appear in your data, then I don't see anything wrong with using \t and \n (ie TSV). Either way, you know your data, and you're using whatever fits it best. If you need something that's never, ever going to break for lack of escaping, might I suggest doing p…
I think the answer is that those shouldn't occur within your data. If you're dealing with binary data, why are you using a text-based file format? If your data is textual, it shouldn't have control character delimiters within it, as they are reserved for that context.
So, strip them out of your data if you have to. If you think they need to be preserved or escaped, IMO you're doing something wrong.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#100Earlier quoted context omitted.
It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…
> Precisely what makes them valuable is their difficulty to type. Again, if they'd caught on you'd imagine there would be some eventual convention in text-editors for what keybind would be used to enter them. Too bad the AltGr key (intended for entering rarely-used glyphs) doesn't appear on pure-English keyboards.
I'm looking at my English keyboard and it has Alt Gr...