Earlier quoted context omitted.
C'est la vie ;)
Gratzie
Grazie |-)
ASCII Delimited Text – Not CSV or TAB delimited text
131–140 of 286 posts
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#132Earlier quoted context omitted.
...and that's why, when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character. Similarly, Ctrl-L clears the screen in most Unix apps because that generates the "form feed" character, and on a line printer or paper-based teletype terminal, "form feed" means to advance to th…
when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character. This is slightly different than being able to actually generate those characters as input. If you put a control-d in a file, nothing will stop reading the file when it sees the control-d, it's just another byte. r…
Sure, the default is overridable, and the character has no magic effect when found in a binary file, but there's a reason Ctrl-D was chosen as the standard keystroke sequence to end a stdin transaction, and it has everything to do with the fact that "D" shows up next to "end of transmission" in `man ascii`.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#133I've used these in ASCII files and they are quite useful. But as most folks point out, actually using control characters for "control" conflicts with a lot of legacy usage of "some other control." Which is kind of too bad. Maybe when the world adopts Unicode we'll solve this, oh wait...
I bet someone stubborn would find a way to screw it up anyway (I guess stubborn people are the biggest problem with csv...).
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#134This is factually wrong about CSV, which can store any character including commas and even \0 (zero byte), provided it's implemented correctly (a rather large proviso admittedly, but you should never try to parse CSV yourself). Here is a CSV parser which does get all the corner cases right: https://forge.ocamlcore.org/scm/browser.php?group_id=113
Thanks, I came here to make sure someone said this. Seriously, escaping special characters in CSV isn't all that difficult if you do it right from the start.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#135I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…
It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…
> but I imagine they were initially on keyboards
I wouldn't be surprised if the last time they were on a keyboard, it was a teletype keyboard or a keyboard that punched cards!
Although, actually, were they just typed using the control key from the start?
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#136This is factually wrong about CSV, which can store any character including commas and even \0 (zero byte), provided it's implemented correctly (a rather large proviso admittedly, but you should never try to parse CSV yourself). Here is a CSV parser which does get all the corner cases right: https://forge.ocamlcore.org/scm/browser.php?group_id=113
you should never try to parse CSV yourself Why? Writing a correct parser is not significantly harder than figuring out how to interface to an existing parser library, and allows cool things like heuristic parsing of malformed files. OTOH it's shocking how many people can't write a correct CSV generator , even after being explicitly told what they're doing wrong (which is always either "you need to put quotes around t…
If you knew enough about CSV to be able to write a correct parser, then you'd know enough not to write one lightly.
Here are some surprising valid CSV files:
https://forge.ocamlcore.org/plugins/scmgit/cgi-bin/gitweb.cg...
The test program in the same directory shows the semantic content of each.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#137Would it work if the file was encoded with UTF8 or UTF16?
I had the same question and found this:
http://en.wikipedia.org/wiki/Unicode_control_characters
Seems like the same control characters are present.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#138I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…
ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key. Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Control-C and Control-D, End of Text and End of…
> Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance.
And this is in fact how they show up in vim (or at least my fairly uncustomized vim), using ^ to stand for ctrl as was once conventional: `^_` and `^^`
The legacy MARC binary format still used for library data uses ascii 29, 30, and 31 -- although for reasons with probably some bizarre historical definition uses them DIFFERENTLY than defined in ascii.
0x1D == 29 == ascii group seperator == MARC Record separator
0x1E == 30 == ascii record seperator == MARC Field Terminator
0x1F == 31 == ascii unit separator == MARC subfield seperator
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#139Leaving aside the pain of displaying and typing such characters... > Then you have a text file format that is trivial to write out and read in, with no restrictions on the text in fields or the need to try and escape characters. Phrases like that lead to lovely security bugs.
The thing that leads to "lovely security bugs" is the nonchalant mindset; it has nothing to do with the simple text format. The same attitude paired with ASN.1 data has caused just as many vulnerabilities.
It's not just the nonchalant mindset; it's the thought that because you pick something you don't expect to form part of your input domain, you don't have to escape. Either you have to actually restrict your input domain, or you need escaping.
Re: ASCII Delimited Text – Not CSV or TAB delimited text
#140Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.
I was surprised to see you list tsv as more common than csv. I encounter csv's on a pretty regular basis, but I don't think I've had to parse a tsv in the past 3 or 4 years. As a junior web developer, I don't have much experience though. 9 times out of 10, the csv is coming from or going to Excel, or a system that was designed to support Excel. If you don't mind my asking, what types of data do you regularly work wit…
TSV is nicer for output (on stderr/out or a logfile), so tends to crop op if you want to parse the output/logfile of something. I haven’t seen Excel in use at my workplace yet.