Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

51–60 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#51

Post by mbreese »

Pxtl wrote: post #45
olalonde wrote: post #30

Earlier quoted context omitted.

Agreed. A few days ago, I learned that spaces are illegal between separators in CSV. For example, `"val1", "val2"` is illegal (it should be `"val1","val2"`). Kind of unintuitive given that in most languages, non-delimited spaces are insignificant.

I don't think CSV is a rigidly-defined format - I'm sure some implementations will happily accept spaces between the comma and the opening quote.

This is one reason why I prefer tab delimited files... The format is pretty simple with few edge cases. There really only one caveat to worry about - fields with tab characters. And that's extremely rare in my field.

CSV on the other hand has a few different variations.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#52

Post by wglb »

chiph wrote: post #21
trebor wrote: post #2

Now I feel silly for having glossed over the control characters since I was a kid. Those characters are decidedly useful on a machine level, though the benefit of CSV/TSV is that it's human friendly.

Trivia: Carriage return and Line feed are separate characters because they used to be separate operations for devices like Teletypes. Want double-spaced text? CRLFLF. Working with a slow device? CRCRCRLF to give the carriage time to return. Baudot4Life, yo.

Or DEL characters more likely.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#53

Post by stronglikedan »

mmasashi wrote: post #43

It does not solve the problem. Here is the points which I think. 1. Control characters are not supported in the almost of text editors. 2. Control characters are not human friendly. 3. The text may contain control characters in the field value. In any formats, we cannot avoid the escape characters, so even I think CSV/TSV format is reasonable.

You are correct in that it does not solve a problem. Furthermore, the article tries to create a problem with CSV that does not exist.

> CSV breaks depending on the implementation on Quotes, Commas and lines

CSV does not break; the implementation is broken if it doesn't parse CSV properly. With a proper implementation, CSV solves every problem that will arise from this method.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#54

Post by russell »

joosters wrote: post #23
mikestew wrote: post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

Interesting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as. Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, s…

In the days of mechanical line printers CR by itself allowed overprinting for special effects like underlining and bold.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#55

Post by wglb »

chrisBob wrote: post #34
binarymax wrote: post #9

Protip - if it doesnt appear on keyboards, you can use ALT+DDD (DDD being 000 to 255) to enter a control character. For those on windows, drop into a command prompt and hold ALT while pressing 031 on the numpad. You will see it produce a ^_ character.

On what computer? Is this windows only? Now that I use a mac this might be the only thing I miss from windows computers.

Yes, I think that is windows only.

So on Mac OS X Mavericks: http://support.apple.com/kb/PH13867

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#56

Post by csixty4 »

joosters wrote: post #23
mikestew wrote: post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

Interesting web page! Despite many years of using ASCII and knowing some of the more common control codes, I had never even thought about what the other mysterious 0-31 codes were defined as. Something that the page doesn't mention is that CR+LF were originally two separate control codes because the action of returning the print head to the left hand side would take too long with a standard line printer. Therefore, s…

Yes. Separate Carriage Return & Line Feed date back to Murray's 1901 variant of Baudot encoding, and ASCII was created to standardize the various teletype encodings out there so it inherited this way of doing things.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#57

Post by dxbydt »

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#58

Post by jff »

derefr wrote: post #50
iclelland wrote: post #41

Earlier quoted context omitted.

It's more likely that it's because they are two separate physical actions (returning the head to the left, and advancing the paper one line). They could be used independently: You could print a line in bold , for instance, by issuing a CR without an LF and then printing the same line again. A carriage-return operation takes much longer than a single character, or even two or three. It doesn't make sense to issue two…

> You could print a line in bold, for instance, by issuing a CR without an LF and then printing the same line again. True, but mildly redundant: "overprinting" was explicitly the purpose of 0x08 backspace (which had nothing, originally, to do with 0x7F deletion.)

To overprint a whole line using 0x08, you'd need one 0x08 for each character in the line. So an N-character line overprinted that way would take N3 characters in memory.

Using CR, you'd need N2 + 1 characters.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#59

Post by slowmover »

mikestew wrote: post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

> One might question why all control codes in the ASCII character set have low values, but the DEL control code has value 127. This is, because this specific character was defined for deleting data on paper tapes. Most paper tapes in that time used 7 holes to code the data. The value 127 represents a binary pattern were all seven bits are high, so when using the DEL character on an existing paper tape, all holes are punched and existing data is erased.

I love this, it shows just how old the roots of ASCII are.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#60

Post by sigil »

mjn wrote: post #8

Alas, I don't think this works with the standard Unix tools, which is the main way I process tab-delimited text. Changing the field delimiter to whatever you want is fine, since nearly everything takes that as a parameter. But newline as record separator is assumed by nearly everything (both in the standard set of tools, and in the very useful Google additions found in http://code.google.com/p/crush-tools/ ). Google'…

> Alas, I don't think this works with the standard Unix tools.

It kind of works with standard unix tools.

    cut -d$'\37' -f ...
    sort -t$'\37' -k ...
    join -t$'\37' ...
Those will parse ASCII-31-separated fields. But records are still newline separated, no way to change that AFAIK, short of running everything through

    tr '\036' '\n'
first. Which defeats the purpose of choosing "weird" delimiters in the first place.

(Also note that the $'\..' syntax is bash-specific and doesn't exist in POSIX sh.)

Post reply on HN