Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

181–190 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#181

Post by mzs »

drivers99 wrote: post #96
iclelland wrote: post #41

Earlier quoted context omitted.

It's more likely that it's because they are two separate physical actions (returning the head to the left, and advancing the paper one line). They could be used independently: You could print a line in bold , for instance, by issuing a CR without an LF and then printing the same line again. A carriage-return operation takes much longer than a single character, or even two or three. It doesn't make sense to issue two…

"You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again." Last I checked, this still works even on laser printers (at least on a LaserJet), when sending data to it as plain text. It's not actually printing over itself, but it knows to make the repeated characters bold.

less (among other unix tools) does this too (but you have to do one character, bs and the character again). There are more, like _, bs, character underlines (like cat there is ul that handles this specifically). If your terminal supports os (overstrike) in it's terminal description it handles that natively.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#182

Post by Terretta »

pjbrunet wrote: post #145
jrochkind1 wrote: post #138

Earlier quoted context omitted.

> Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. And this is in fact how they show up in vim (or at least my fairly uncustomized vim), using ^ to stand for ctrl as was once conventional: `^_` and `^^` The legacy MARC binary format still used for library data uses ascii 29, 30, and 31 -- although for reasons with probably some bizarre historical definition uses them D…

Back in the day with a numeric keypad you could type Alt-(number) to get any ASCII character, but not sure if that still works. In Firefox apparently Alt-1 takes you to the first tab, Alt-2 second tab...

Alt-(3 digits)

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#183

Post by danielweber »

jrochkind1 wrote: post #142
danielweber wrote: post #129

Earlier quoted context omitted.

I was disappointed that Ctrl-S and Ctrl-Q don't correspond (at least on the first ASCII chart I googled) to anything, because Ctrl-S stops text on my Unix terminals the exact same way it did on my brother's Apple ][e in 1984.

> I was disappointed that Ctrl-S and Ctrl-Q don't correspond (at least on the first ASCII chart I googled) to anything I'm pretty sure every possible combination of 7-bits is assigned SOME name in ascii, and Ctrl-any-case-insensitive-letter is a defined 7bit value. This chart says control-s is `DC3 (Device Control, X-OFF)` and control-q is `DC1 (Device Control, X-ON)`. Yup, that's what they do alright. http://www.uni…

Oh, thank you. The reference I found[1] only showed them as "device control." I thought I imagined them having that.

Also, the classic control-G is still in C. "\a" is "alert (beep)."

[1] http://www.asciitable.com/

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#184

Post by BorgHunter »

pjbrunet wrote: post #145
jrochkind1 wrote: post #138

Earlier quoted context omitted.

> Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. And this is in fact how they show up in vim (or at least my fairly uncustomized vim), using ^ to stand for ctrl as was once conventional: `^_` and `^^` The legacy MARC binary format still used for library data uses ascii 29, 30, and 31 -- although for reasons with probably some bizarre historical definition uses them D…

Back in the day with a numeric keypad you could type Alt-(number) to get any ASCII character, but not sure if that still works. In Firefox apparently Alt-1 takes you to the first tab, Alt-2 second tab...

In most graphical *nix environments that I'm familiar with, if you hold Control and Shift¹ and type U, you can type any Unicode character by its (hex) code point. For instance, if you want an em dash, which is U+2014, hold Control and Shift, and type "U2014".

¹You can release Control and Shift after typing "U", in which case the character will appear after you type a space. Or, you can hold Control and Shift while typing the code point, in which case the character will appear after you release either modifier key.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#185

Post by danielweber »

fineline wrote: post #178
danielweber wrote: post #129

Earlier quoted context omitted.

I was disappointed that Ctrl-S and Ctrl-Q don't correspond (at least on the first ASCII chart I googled) to anything, because Ctrl-S stops text on my Unix terminals the exact same way it did on my brother's Apple ][e in 1984.

I think you mean your brother's Apple //e - they switched from the original ][ to // for later models. That's what I started programming on too - except in my case it was my Dad's.

Ah, but the boot screen said "APPLE ]["

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#186

Post by mzs »

jff wrote: post #58
derefr wrote: post #50

Earlier quoted context omitted.

> You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again. True, but mildly redundant: "overprinting" was explicitly the purpose of 0x08 backspace (which had nothing, originally, to do with 0x7F deletion.)

To overprint a whole line using 0x08, you'd need one 0x08 for each character in the line. So an N-character line overprinted that way would take N 3 characters in memory. Using CR, you'd need N 2 + 1 characters.

I had a daisey wheel printer in the late '80s that had a few characters of buffer. I had to know that it took quite some time for every CR. It would do the niave bold of the full line with CR, but if it got X^HX it would hit the X and then slide the head over a bit to the right to smear and get the bold effect, which looked much better. It was not uncommon and that's another reason a lot code did it the BS way for OS capable HC devices.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#187

Post by tracker1 »

I actually really appreciate this article, though I've known about it for decades now. In fact, I used to return javascript results in a post target frame back in the mid-late 90's and would return them in said delimited format... field/record/file separated, so that I could return a bunch of data. Worked pretty well with the ADO Recordset GetString method.

Of course, I was one of those odd ducks doing a lot of Classic ASP work with JScript at the time.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#188

Post by scintill76 »

sigil wrote: post #106
scintill76 wrote: post #99

Earlier quoted context omitted.

I think the answer is that those shouldn't occur within your data. If you're dealing with binary data, why are you using a text-based file format? If your data is textual, it shouldn't have control character delimiters within it, as they are reserved for that context. So, strip them out of your data if you have to. If you think they need to be preserved or escaped, IMO you're doing something wrong.

It's nice to be able to use the unix toolset (grep, cut, sort, join, etc) on all kinds of data, not just strictly "textual" data. They're often my tool of last and only resort when dealing with very large datasets. Sure, you could wait for that dump of all of wikipedia to import into a nice indexed and queryable database, but why not start grepping it immediately? Maybe you want to sort by a key that's textual, but t…

If you've actually got, say, a JPG file embedded in the middle of a CSV or something, it's just not designed for that IMO. But if you use the reserved control characters, you can at least output any text data without escaping (at least if "textual" is defined as "a string of characters that are not ASCII delimiter control chars", which ought to be easy to assume, unless something is corrupted or deliberately trying to mess things up.) I think you can actually more easily and reliably grep, because you don't need state to know whether the comma byte is a comma character or a delimiter. You simply use a comma when you mean a literal comma, and the control char when you want the delimiter. Anyway, as others have pointed out, there are other obstacles to widespread adoption of these control chars.

I agree that cut, sort, etc. are good to be familiar with. Someone else[1] linked a "csvquote" utility that pre-chews (and un-chews at the end of the text-processing pipeline) CSV data to make it work better with standard UNIX utilities. Looks neat, so I'll be keeping it in mind next time I'm processing CSV with UNIX utils.

[1]: https://news.ycombinator.com/item?id=7475793

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#189

Post by rcthompson »

JadeNB wrote: post #167
rcthompson wrote: post #85

If you use unprintable characters in your file, it's no longer human-editable as text. It may as well be XML (i.e. technically text-based but not practically human-readable).

> It may as well be XML (i.e. technically text-based but not practically human-readable). Is this really a standard complaint about XML? I thought the main complaint was that it wasn't human- writeable . I wouldn't want to read novels in XML, but I've never had a problem opening up an XML file in a text editor to get at bits of it.

Well, you'll often get XML files with a single line and no whitespace between elements, and that makes things a lot more interesting. Basically, you can't rely it being practical to quickly poke around in the text of an XML file. I always feel sad when I have to read the text of an XML file to get information.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#190

Post by tmalsburg2 »

thebelal wrote: post #38
co_dh wrote: post #11

How do you enter them ? in console, in editor? Since they are invisible, how do you find if you have entered a wrong character?

You can type them in console and vim with File separator - C-v C-\ Group separator - C-v C-5 Record separator - C-v C-6 Unit separator - C-v C-7 They are all visible characters in both vim and emacs by default. You can see them on the terminal with `cat -v` It would be nice if more tools were built to take advantage of these characters, but there are some that do.

In Emacs, one way to enter these characters is to use `M-x ucs-insert` and then enter the hex code of the separator:

    M-x ucs-insert 1c
    M-x ucs-insert 1d
    M-x ucs-insert 1e
    M-x ucs-insert 1f
for file, group, record, and unit separators. Is there an easier way?
Post reply on HN