Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

231–240 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#231
post #171

Earlier quoted context omitted.

Of those, |, @, #, ^, {, } and ~ are all keys that you need shift for on US Qwerty keyboards, so I imagine that the typing experience for those is relatively comparable.

I fixed my keyboard -- @,#,^,{,}.... no shift by default. Have to press shift to get 2,3,6,[,],9,0,... Must be annoying to have to press shift to get those symbol chars while you're coding, eh?

I think it's french keyboards that are like that by default. Symbols are the default and you have to press the Shift-key to get numbers.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#232

Earlier quoted context omitted.

IF those characters are actually displayed on the corresponding keys. If it's like the # on the Mac keyboard - not displayed anywhere - it's a right pain to learn in the first place without any visual cues.

I'm looking at a couple Macs sitting around the house and they all have a pretty obvious # right above the 3. Perhaps you mean some other key? (I do really miss the days when Mac keyboards had the weird symbol they use for "alt" in menus on the key.)

Which keyboard do you have? The standard aluminium full size one doesn't label it, at least in the uk

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#233
post #35

Earlier quoted context omitted.

Depends on your platform. Even today a Unix/Linux line ending is almost always just LF. See: http://en.wikipedia.org/wiki/Newline

Unix = LF Mac = CR DOS = CRLF The DOS line endings are inherited from previous systems, and while they precisely convey carriage and paper movement of a printer it can be a pain to deal with today.

Somewhat ironic that this detail of how teletypewriters work was not carried over into Unix whereas every intricacy of how video terminals worked still is with us today.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#234
post #136

Earlier quoted context omitted.

If you knew enough about CSV to be able to write a correct parser, then you'd know enough not to write one lightly. Here are some surprising valid CSV files: https://forge.ocamlcore.org/plugins/scmgit/cgi-bin/gitweb.cg... The test program in the same directory shows the semantic content of each.

testcsv6.csv at that link is malformed. DQUOT is used to (1) escape itself, and (2) enclose strings. It is not a generalized escape character the way backslash is in C-family languages. Interpreting it as a generalized escape causes two problems. One, if you generate files that way, they will be unreadable by parsers written according to the RFC. Two, if you read files that way, you will silently garble files generat…

I'm afraid you're wrong about this. Excel generates and parses "0 as a zero byte. The RFC doesn't discuss how CSV files work in the real world. This is exactly what I was talking about in my comment above.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#235
post #5

Anyone that's ever had to parse arbitrary data knows of the approximately 14 jiggityzillion corner cases involved when sucking in or outputting CSV/TAB delimited formats. Yet much like virtual memory and virtual machines, we find that a solution has existed since the 60s. For those wondering about the history and use of all those strange characters in your ASCII table: http://www.lammertbies.nl/comm/info/ascii-charac…

Agreed. A few days ago, I learned that spaces are illegal between separators in CSV. For example, `"val1", "val2"` is illegal (it should be `"val1","val2"`). Kind of unintuitive given that in most languages, non-delimited spaces are insignificant.

Spaces are valid in fields, even at the start or end. Furthermore you cannot start quoting in the middle of a field, which is why this doesn't work. For unquoted fields you can put as many spaces as you like after the separator, but they become part of the field, then.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#236
post #45

Earlier quoted context omitted.

Agreed. A few days ago, I learned that spaces are illegal between separators in CSV. For example, `"val1", "val2"` is illegal (it should be `"val1","val2"`). Kind of unintuitive given that in most languages, non-delimited spaces are insignificant.

I don't think CSV is a rigidly-defined format - I'm sure some implementations will happily accept spaces between the comma and the opening quote.

There is RFC 4180, which is probably the best formal description of the format we have.

My own CSV parsers (I have written a few by now) usually parse that as if the space before (or after) the quote wasn't there. It's nonetheless something to avoid when writing CSV files (Postel's law, etc.).

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#237

Earlier quoted context omitted.

I'm from Croatia, and we have the AltGr key. However, I discovered that Alt + Control = AltGr when I needed to use it at work[1], so it's simply a shortcut, I think. [1]: We have (I recently switched to an UK keyboard, because it suits me better) keyboards at work, because the Croatian (all slavic languages, to be honest) is horrendously counterproductive for programming. Google the layout, and you'll realise why. An…

Perhaps, but I think it still goes against the original intent. Ctrl-~ or Ctrl-^ should give you a record separator (RS) and Ctrl-Del or Ctrl-_ should give you a unit separator (US). For the same reason Ctrl-m or Ctrl-M should give you carriage return (CR). This is because ASCII values from 00-1F are control characters and effectively grounded the most significant bits 7 and 6. Shift similarly would toggle or ground…

This hasn't been true since IBM keyboards became popular. For example, on older keyboards shift+number would simply toggle a bit, so shift+2 would be a double quote, etc., but this hasn't been common for decades now. Unfortunately.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#238
post #35

Earlier quoted context omitted.

Depends on your platform. Even today a Unix/Linux line ending is almost always just LF. See: http://en.wikipedia.org/wiki/Newline

Unix = LF Mac = CR DOS = CRLF The DOS line endings are inherited from previous systems, and while they precisely convey carriage and paper movement of a printer it can be a pain to deal with today.

Unix = LF Mac = CR DOS = CRLF

Please add to that list also

HTTP = CRLF

http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html

However, I still have to see I single webserver that does not accept LF alone instead of CRLF.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#239

Earlier quoted context omitted.

Living in the US, I've never heard of an AltGr key until this discussion.

I'm from Croatia, and we have the AltGr key. However, I discovered that Alt + Control = AltGr when I needed to use it at work[1], so it's simply a shortcut, I think. [1]: We have (I recently switched to an UK keyboard, because it suits me better) keyboards at work, because the Croatian (all slavic languages, to be honest) is horrendously counterproductive for programming. Google the layout, and you'll realise why. An…

I'm a Spanish living in UK and I love using UK keyboards for the same reason: the layout is way better for programming!

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#240
post #88

Related: This tool: https://github.com/dbro/csvquote will convert all the record/field separators (such as tabs/newlines for TSV) into non-printing characters and then in the end reverse it. Example: csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u It's underrated IMO.

Thanks for bringing up csvquote. I wrote it last year, and am happy to hear that other people find it useful.

It is indeed a simple state machine (see https://github.com/dbro/csvquote/blob/master/csvquote.c), and it translates CSV/TSV files into files which follow the spirit of what's described in the original article in this thread.

But instead of using control characters as separators, it uses them INSIDE the quoted fields. This makes it easy to work with the standard UNIX text manipulation tools, which expect tabs and newlines to be the field and record separators.

The motivation for writing the tool was to work with CSV files (usually from Excel) that were hundreds of megabytes. These files came from outside my organization, and often from nontechnical people - so it would have been difficult to get them into a more convenient format. That's the killer feature of the CSV/TSV format: it's readable by the large number of nontechnical information workers, in almost every application they use. I can't think of a file format that is more widely recognized (even if it's not always consistently defined in practice).

Post reply on HN