Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

211–220 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#211

Post by thwarted »

raldi wrote: post #132
thwarted wrote: post #116

Earlier quoted context omitted.

when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character. This is slightly different than being able to actually generate those characters as input. If you put a control-d in a file, nothing will stop reading the file when it sees the control-d, it's just another byte. r…

Sure, the default is overridable, and the character has no magic effect when found in a binary file, but there's a reason Ctrl-D was chosen as the standard keystroke sequence to end a stdin transaction, and it has everything to do with the fact that "D" shows up next to "end of transmission" in `man ascii`.

Thank you for clarifying something that no one was disputing or event mentioned.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#212

Post by chinpokomon »

eli wrote: post #35
joosters wrote: post #23

Earlier quoted context omitted.

Interesting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as. Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, s…

Depends on your platform. Even today a Unix/Linux line ending is almost always just LF. See: http://en.wikipedia.org/wiki/Newline

Unix = LF Mac = CR DOS = CRLF

The DOS line endings are inherited from previous systems, and while they precisely convey carriage and paper movement of a printer it can be a pain to deal with today.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#213

Post by gwu78 »

Why not use the non-printing char as the comma instead of the record separator.

1. Replace all the commas in the text with the unique non-printing char before converting to CSV.

2. Convert this char back to a comma when processing the CSV for output to be read by humans.

Because commas in text are usually followed by a space, the CSV may still even be readable when using the non-printing char.

I must admit I've never understood why others view CSV as so troublesome vis-a-vis other popular formats.

in: sed 's/,/%2c/g' out: sed 's/%2c/,/g'

I guess I need someone to give me a really hairy dataset for me to understand the depth of the problem with CSV.

Meanwhile, I love CSV for its simplicity.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#214

Post by hrabago »

gress wrote: post #196
3JPLW wrote: post #155

Earlier quoted context omitted.

You're missing my point. If this were a "solved problem," then it would be no better than the status quo with TSV. Tabs are a similar kind of control character, except they can be typed and displayed easily. And TSV files are a mess because tabs end up in the fields themselves. It's a catch-22. (I'm exaggerating; this would still be somewhat better since there will never be a big key on your keyboard dedicated to the…

Tabs are not a record separator. They are a formatting code. That's why a mess is created when they are used to separate records.

You just gave further proof that tabs are a catch-22.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#215

Post by userbinator »

Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

In the standard CP437 PC charset, those characters are up/down triangular arrows. The standard MS-DOS editor also easily supports entering them with Alt+30/31, so maybe 20 years ago a proposal to do this might've been more accepted than it is today.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#216

Post by fshaun »

tmalsburg2 wrote: post #190
thebelal wrote: post #38

Earlier quoted context omitted.

You can type them in console and vim with File separator - C-v C-\ Group separator - C-v C-5 Record separator - C-v C-6 Unit separator - C-v C-7 They are all visible characters in both vim and emacs by default. You can see them on the terminal with `cat -v` It would be nice if more tools were built to take advantage of these characters, but there are some that do.

In Emacs, one way to enter these characters is to use `M-x ucs-insert` and then enter the hex code of the separator: M-x ucs-insert 1c M-x ucs-insert 1d M-x ucs-insert 1e M-x ucs-insert 1f for file, group, record, and unit separators. Is there an easier way?

C-q (quoted-insert) will insert the next input character ignoring any other bindings. So: C-q C-\ C-q C-] C-q C-^ C-q C-_

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#217

Post by chinpokomon »

ithkuil wrote: post #192
mantrax3 wrote: post #13

Earlier quoted context omitted.

For CSV forbidding commas in data is not practical. For ASCII delimiters, forbidding ASCII delimiters in data is practical. Sure - you can't, say, nest ASCII tables into one another due to this limitation. But for simple structure, it doesn't hurt to have ASCII separators in the toolbox. The only big problem I see is that they're rendered as invisible characters, which will make debugging harder. If we wouldn't have…

If only they had included an official escape character in ascii ... oh wait

That ESC is to allow more control characters to be defined than what was initially put into the standard. Consider for instance how ANSI colors are written to the terminal. They are escape sequence control (ESC? I'm not sure if the original designer meant to be meta) characters, identified by the escape sequence (ESC)[.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#218

Post by qu4z-2 »

thwarted wrote: post #211
raldi wrote: post #132

Earlier quoted context omitted.

Sure, the default is overridable, and the character has no magic effect when found in a binary file, but there's a reason Ctrl-D was chosen as the standard keystroke sequence to end a stdin transaction, and it has everything to do with the fact that "D" shows up next to "end of transmission" in `man ascii`.

Thank you for clarifying something that no one was disputing or event mentioned.

Yeah, I hate it when people point out interesting things I didn't know before too.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#219

Post by egypturnash »

oneeyedpigeon wrote: post #197
Crito wrote: post #171

Earlier quoted context omitted.

Of those, |, @, #, ^, {, } and ~ are all keys that you need shift for on US Qwerty keyboards, so I imagine that the typing experience for those is relatively comparable.

IF those characters are actually displayed on the corresponding keys. If it's like the # on the Mac keyboard - not displayed anywhere - it's a right pain to learn in the first place without any visual cues.

I'm looking at a couple Macs sitting around the house and they all have a pretty obvious # right above the 3. Perhaps you mean some other key?

(I do really miss the days when Mac keyboards had the weird symbol they use for "alt" in menus on the key.)

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#220

Post by awwaiid »

Crito wrote: post #171
flux_w42 wrote: post #165

Earlier quoted context omitted.

Here in Belgium we have to AltGr-the hell out of our keyboard during development. These characters can only be entered using the AltGr key on a Belgian Azerty keyboard (be-latin1): '|' '@' '#' '^' '{' '}' '[' ']' '\' and '~' Its not realy a problem, as long as you're used to it :-)

Of those, |, @, #, ^, {, } and ~ are all keys that you need shift for on US Qwerty keyboards, so I imagine that the typing experience for those is relatively comparable.

I fixed my keyboard -- @,#,^,{,}.... no shift by default. Have to press shift to get 2,3,6,[,],9,0,...

Must be annoying to have to press shift to get those symbol chars while you're coding, eh?

Post reply on HN