I've used these in ASCII files and they are quite useful. But as most folks point out, actually using control characters for "control" conflicts with a lot of legacy usage of "some other control." Which is kind of too bad. Maybe when the world adopts Unicode we'll solve this, oh wait...
ASCII Delimited Text – Not CSV or TAB delimited text
81–90 of 286 posts
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#82Took some time to figure out how to type these on a Mac: 1. Go to System Preferences => Keyboard => Input Sources 2. Add Unicode Hex Input as an input source 3. Switch to Unicode Hex Input (assuming you still have the default keyboard shortcuts set up, press Command+Shift+Space) 4. Hold Option and type 001f to get the unit separator 5. Hold Option and type 001e to get the record separator 6. (Hold Option and type a c…
In Terminal, they are: FS: Control-\ 0x1c (field sep) GS: Control-] 0x1d (group sep) RS: Control-^ 0x1e (record sep) US: Control-_ 0x1f (unit sep) (These control key equivalents have always been the canonical keystrokes to generate the codes) But they have to be preceded by a Control-V (like in vi) to be treated as input characters. Control-V is the SYN code (synchronous idle), but has no special meaning in an intera…
Most terminals will let you use ctrl-6 for ctrl-^ and ctrl-7 (and sometimes ctrl-/) for ctrl-_.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#83Haha! Oh wow, I just finished a project dealing with this exactly. The obvious problem is that most editors make dealing with the non-standard keyboard keys very difficult. As a consequence, most programs (Python, MatLab, etc) really don't like anything below 0x20. I was reading in binary through a serial port, and then storing data for records and processing. Any special character got obliterated in the transfer to…
C'est la vie ;)
Gratzie
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#84Earlier quoted context omitted.
> You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again. True, but mildly redundant: "overprinting" was explicitly the purpose of 0x08 backspace (which had nothing, originally, to do with 0x7F deletion.)
To overprint a whole line using 0x08, you'd need one 0x08 for each character in the line. So an N-character line overprinted that way would take N 3 characters in memory. Using CR, you'd need N 2 + 1 characters.
Even ignoring additional time needed to send those extra characters, it also was a lot faster than those control-H's, and (I guess) caused way less wear on your printer.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#85If you use unprintable characters in your file, it's no longer human-editable as text. It may as well be XML (i.e. technically text-based but not practically human-readable).
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#86It does not solve the problem. Here is the points which I think. 1. Control characters are not supported in the almost of text editors. 2. Control characters are not human friendly. 3. The text may contain control characters in the field value. In any formats, we cannot avoid the escape characters, so even I think CSV/TSV format is reasonable.
You are correct in that it does not solve a problem. Furthermore, the article tries to create a problem with CSV that does not exist. > CSV breaks depending on the implementation on Quotes, Commas and lines CSV does not break; the implementation is broken if it doesn't parse CSV properly. With a proper implementation, CSV solves every problem that will arise from this method.
The problem with CSV is that it looks so simple that nobody ever uses a real library to do it—they just roll their own. So you end up with a million implementations that are all buggy in various different ways. If you receive a CSV formatted file you can never be sure if it's actually good, valid CSV, or some invalid crap from that some programmer that reinvented the wheel because it was "so easy".
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#87How do you enter them ? in console, in editor? Since they are invisible, how do you find if you have entered a wrong character?
> Since they are invisible, how do you find if you have entered a wrong character?
od -cRe: ASCII Delimited Text – Not CSV or TAB delimited text
#88Related: This tool:
https://github.com/dbro/csvquote
will convert all the record/field separators (such as tabs/newlines for TSV) into non-printing characters and then in the end reverse it. Example:
csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#89Would it work if the file was encoded with UTF8 or UTF16?
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#90This is factually wrong about CSV, which can store any character including commas and even \0 (zero byte), provided it's implemented correctly (a rather large proviso admittedly, but you should never try to parse CSV yourself). Here is a CSV parser which does get all the corner cases right: https://forge.ocamlcore.org/scm/browser.php?group_id=113
you should never try to parse CSV yourself
Why? Writing a correct parser is not significantly harder than figuring out how to interface to an existing parser library, and allows cool things like heuristic parsing of malformed files.
OTOH it's shocking how many people can't write a correct CSV generator, even after being explicitly told what they're doing wrong (which is always either "you need to put quotes around the data" or "you need to double any quotes that are part of the data") and given examples.