Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII and Unicode quotation marks

cl.cam.ac.uk

Post reply on HN
Print view View original

91–100 of 195 posts

Re: ASCII and Unicode quotation marks

#91

Post by cgtyoder »

lisper wrote: post #41

The fact that ASCII does not have balanced quotes is one of the great catastrophes of computing. It makes everything more complicated than it needs to be, from embedding code in strings to parsing CSV files, to regexps. For example, if I want to embed a quoted string in another quoted string, I have to escape the inner quotes like so: "This is string containing an embedded \"quoted\" string" Then I have to think abou…

> The fact that ASCII does not have balanced quotes is one of the great catastrophes of computing.

Okay.

Re: ASCII and Unicode quotation marks

#92

Post by ComputerGuru »

diamondo25 wrote: post #71
PickNChoose wrote: post #67

Earlier quoted context omitted.

You don't want any 'rare' cases at all. That's the point. Stop using "punctuation" when you are attempting to "delimit" text. Use a character that is not punctuation, specifically designed for "field delimiter" purposes. Trying to do two things at once is ridiculous.

I've been trying to push TSV (tab separated values) as a standard response/implementation when they ask for CSV. "Yes but its comma separated!", sure is, but text can contain commas... I have seen issues with Google Spreadsheets not recognizing the tabs however... Excel doesnt know what to do with a TSV either. But both have a complete wizard for parsing CSV...

> but text can contain commas

Erm.. text can contain tabs, too. This problem was solved so, so long ago when all the various ANSI/ASCII/whatever encodings were compiled by specifically reserving not one but two characters precisely to serve as field and record delimiters.

0x30 and 0x31 solved not only the problem of having commas or tabs in your text preventing you from treating them as field delimiters, but also allowing you to include new lines and carriage returns in your fields, too!

0x30 is the unit separator (aka field delimiter) and 0x31 is the record separator (aka, well, the record separator).

I _believe_ there was a record key on some standardized keyboard layout back in the day, too.

Edit: sorry, they are decimal, not hex. Thanks @jrochkind1

Re: ASCII and Unicode quotation marks

#93

Post by evincarofautumn »

ttepasse wrote: post #90

The usage of of an accent as syntax in markup and programming languages annoys me to no end. And it will still be used, to this day, the latest example are template string in Javascript. • It is semantically idiotic because it's an accent, not a character. • It is visually annoying because you almost can't see the thing. • It is bad for usability, because on non-US keyboards the accents are implemented as dead keys.…

Same, I’ve never cared for it. For these reasons I’ve decided to take a stand and avoid using the grave accent for anything in a programming language I’m working on. Same goes for the dollar sign, because it’s somewhat Americentric, and as a currency character it doesn’t have any great semantic or mnemonic value except for, well, currency units. I guess you could argue for $trings (BASIC) or $calars (Perl) if you have $igils, but I don’t.

Sacrificing these bits of ASCII is fine by me, because the language is small enough, and I also allow Unicode. For example, curved quotes are allowed and can be nested or contain ASCII quotes without escaping:

    // Character literals
    ‘'’
    =
    '\''

    // Text literals
    “Some "text" with “curved quotes”.”
    =
    "Some \"text\" with “curved quotes”."
For the sake of usability, of course, everything in the core language & standard library has an ASCII spelling, like in Perl6. I’d like for other languages to adopt this view as well. If new languages allow proper Unicode notation in some sensible places, then programming editors’ input methods will catch up, e.g., automatically replacing “->” with “→” or “\theta” with “θ” (like Emacs’ TeX input mode).

Also, does anyone know of a reference for keyboard layouts from around the world that includes estimates of the number of people using them? I’ve tried to keep things relatively easy to type on all the major layouts I know of, but I don’t want to alienate anyone if I can help it.

Re: ASCII and Unicode quotation marks

#94

Post by taftster »

ComputerGuru wrote: post #92
diamondo25 wrote: post #71

Earlier quoted context omitted.

I've been trying to push TSV (tab separated values) as a standard response/implementation when they ask for CSV. "Yes but its comma separated!", sure is, but text can contain commas... I have seen issues with Google Spreadsheets not recognizing the tabs however... Excel doesnt know what to do with a TSV either. But both have a complete wizard for parsing CSV...

> but text can contain commas Erm.. text can contain tabs, too. This problem was solved so, so long ago when all the various ANSI/ASCII/whatever encodings were compiled by specifically reserving not one but two characters precisely to serve as field and record delimiters. 0x30 and 0x31 solved not only the problem of having commas or tabs in your text preventing you from treating them as field delimiters, but also all…

Right. And if you add 0x28 (file separator) and 0x29 (group separator) to the mix, then you have a whole set of nice options to concatenate multiple data files into a single stream, etc.

Re: ASCII and Unicode quotation marks

#95

Post by slavik81 »

coldtea wrote: post #79
diamondo25 wrote: post #71

Earlier quoted context omitted.

I've been trying to push TSV (tab separated values) as a standard response/implementation when they ask for CSV. "Yes but its comma separated!", sure is, but text can contain commas... I have seen issues with Google Spreadsheets not recognizing the tabs however... Excel doesnt know what to do with a TSV either. But both have a complete wizard for parsing CSV...

There is actually an ASCII character that doesn't appear in strings, and is meant to be used as such a separator. Actually two of them, record (30) and unit separator (31).

Sadly, eventually someone will want to enter one document as a field in another document and then you end up needing escaping anyways. Using a rare symbol for the delimiter would still be nice for typing documents by hand, but it would have to be available on modern keyboards to be convenient.

Re: ASCII and Unicode quotation marks

#96

Post by ComputerGuru »

leephillips wrote: post #37
PeterisP wrote: post #31

Earlier quoted context omitted.

Well, since my keyboard doesn't have an mdash key, if there's no support for something like — or --- then I can't use mdashes.

Nobody's keyboard had an em-dash key (well, maybe some compositor keyboards...). It's well worth the time to figure out how to enter at least the most useful Unicode characters. See, here's an em-dash: —.

I use the code behind this web app: http://latex2unicode.herokuapp.com/

It converts latex to unicode where possible. It's pretty impressive how much of Latex can be replaced with Unicode today.

This program translates LaTeX markup to human-readable Unicode when possible.

Here's the default text from that webapp:

Basic math notations: ∵ A͡B + B͡C ≠ A͡C ∴ ∬∜x̅ ξᶿ⁺¹ - ⅜ ≤ Σ ζᵢ ∴ ∃x∀y x ∈ Â

Easily type in hundreds of other symbols and special characters: , ℵ, Œ, ⇊, etc.

Font styles support: 𝔹𝕝𝕒𝕔𝕜 𝔹𝕠𝕒𝕣𝕕 𝔹𝕠𝕝𝕕, 𝔉𝔯𝔞𝔨𝔱𝔲𝔯, 𝐁𝐨𝐥𝐝 𝐅𝐚𝐜𝐞, 𝓒𝓪𝓵𝓵𝓲𝓰𝓻𝓪𝓹𝓱𝓲𝓬, 𝐼𝑡𝑎𝑙𝑖𝑐, 𝙼𝚘𝚗𝚘𝚜𝚙𝚊𝚌𝚎.

Now type in this box and try it yourself. ⌣̈

Re: ASCII and Unicode quotation marks

#97

Post by 13of40 »

CSB: Years ago I was working on a team that developed a scripting language and we had this recurring problem where someone would write up a code sample in a Word document and it would break if you cut and pasted it because all of the single and double quotes would be Unicode. My boss was this tough guy who tried to snap the whole team to a standard of strictly disabling that behavior in all of our Office applications, but I piped up and said maybe we should just make the language treat all of those characters like apostrophes and quotes.... I think around version 5 they finally made an API for doing proper anti-injection escaping because you pretty much needed a PhD to get it right due to all of the variations introduced by the extended characters.

Re: ASCII and Unicode quotation marks

#98

Post by stevelosh »

mbrock wrote: post #17
treve wrote: post #10

It just occurred to me how much easier certain text-operations (like syntax highlighting, regular expressions and other parsers) if we consistently used the right unicode symbols for quotes and apostrophes

The only languages I know off the top of my head that use balanced delimiters for strings are M4 and Perl 6. Hey, imagine being able to nest strings without escaping! What a concept!

Common Lisp doesn't have it built in, but the cl-interpol library adds this (and you can add your own custom delimiters too).

http://weitz.de/cl-interpol/#syntax

Re: ASCII and Unicode quotation marks

#99

Post by delinka »

13of40 wrote: post #97

CSB: Years ago I was working on a team that developed a scripting language and we had this recurring problem where someone would write up a code sample in a Word document and it would break if you cut and pasted it because all of the single and double quotes would be Unicode. My boss was this tough guy who tried to snap the whole team to a standard of strictly disabling that behavior in all of our Office applications…

Or ... use a text editor?

Re: ASCII and Unicode quotation marks

#100

Post by alanh »

While I can’t expect many to follow suit, I myself often type educated quotes and nice apostrophes. The macOS keyboard combinations (nearly-intuitive combinations of Option-(Shift)-[ and -] for “”‘’) have long been committed to muscle memory. And since nearly all (web) file formats seem to be UTF-8, the days of manually typing “ and friends are long, long gone.

Benefits of typing and using typographer’s quotes directly in your JS/JSON/HTML/source:

1. No backslashes or other escape sequences needed!

2. WYSIWYG

3. Retina screens and gorgeous modern fonts mean that your sloppy quotes will look extra bad if you just use ASCII quotes

Post reply on HN