Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII and Unicode quotation marks

cl.cam.ac.uk

Post reply on HN
Print view View original

171–180 of 195 posts

Re: ASCII and Unicode quotation marks

#171

Post by yorwba »

pluma wrote: post #166
PeterisP wrote: post #31

Earlier quoted context omitted.

Well, since my keyboard doesn't have an mdash key, if there's no support for something like — or --- then I can't use mdashes.

If you're on Linux (maybe macOS supports this too?) you can open the keyboard settings and turn one of several keys into the Compose/Multi key (I picked CapsLock because I rarely hit it on accident and don't use it). Then you can type all kinds of weird combinations: https://www.x.org/releases/X11R7.7/doc/libX11/i18n/compose/e... For example, em dash is Compose+minus+minus+minus, en dash is Compose+minus+minus+period…

The best way to enter arbitrary Unicode characters on Linux I have found is fcitx's https://fcitx-im.org/Unicode

Just press Ctrl-Alt-Shift-U, type (part of) the name of the character and select from the list of results.

— (em dash), ︱ (presentation form for vertical em dash), ⤐ (rightwards two-headed triple-dash arrow), ﷽ (arabic ligature bismillah ar-rahman ar-raheem) are all easy to enter.

Re: ASCII and Unicode quotation marks

#172

Post by dragonwriter »

stephengillie wrote: post #70
dragonwriter wrote: post #62

Earlier quoted context omitted.

Also Ruby: %q{This is a string with an %q{embedded quote}.}

Powershell: "This is a string with an 'embedded quote'." It's helpful to remember that quotes will interpret the variables inside, while apostrophes will not. Very useful for scripting the creation of scripts. Example: "It is $time" > It is 15:22 'It is $time' > It is $time "'$time' is $time" > '$time' is 15:22

That's not a string where the same thing used for quoting is used inside the string without escaping, nor is it an example of the distinct begin-vs-end quote pairs approach under discussion.

But, yes, having single and double quoted strings is another way to avoid escaping (which Ruby and a number of other languages discussed as supporting the approach being discussed also support.)

Re: ASCII and Unicode quotation marks

#173

Post by aoloe »

LeoNatan25 wrote: post #165
daemin wrote: post #162

Earlier quoted context omitted.

It would be fine if it just changed visually depending on the context, without actually changing the underlying character data.

Then it would have to be a feature of the font, and I am not familiar with any font that does that automatically. Technically, it might be possible, with advanced scripting available in OpenType, but considering how even the most basic of ligatures are unsupported under most Linux and Windows fonts, I would not have my hopes up to have this any time soon.

As far as i know it's (very) hard to do correctly.

There are (small or big) differences among languages and it is not always obvious to detect ,if the quote should be converted at all, if it's a left one, or a right one.

As a reference, this is a Python Script trying to do the conversion in Scribus:

https://wiki.scribus.net/canvas/Convert_Typewriter_Quotes_to...

There is also an article by the author of the script, explaining his work and pointing to at least one common case it cannot handle:

https://opensource.com/article/17/3/python-scribus-smart-quo...

Putting all this logic in a font might or not be something your really want...

But I 100%: this should be done at the font level... and hard replacing characters is not a good solution.

Re: ASCII and Unicode quotation marks

#174

Post by TeMPOraL »

LeoNatan25 wrote: post #164
jimmies wrote: post #161

Earlier quoted context omitted.

I never said I hate the 69 quotation thing. I said I hate the automatic conversion of the straight quote to the 69 quote behind my back. Imagine they automatically replace the "fi" with the styled "fi" behind your back and breaks text searches and normal text files. You can be snobby all you want, as long as you initialed it and know you don't break programs' assumptions. No one says you can't have nice shit with LaT…

Regarding fi, it just shows how broken most search engines are. As a reader, what is the difference between between fi and fi? Or between a and а? If the search engine is distinguishing those because of Unicode characters, it has failed completely.

Unicode turns trivial string comparison problems into pretty much AI-complete ones. Same visual characters, different meaning. Different characters, but same visual meaning for the person who initiated the query. Up/down-case not well defined, and even if it is, it's not always reversible (i.e. tolower(toupper(tolower(string))) is not equal to tolower(string)). There's lots of that.

The longer I live, the bigger fan of simplification and standardization I become. Like with dates and times. Timezones and DST are a fucking nightmare, because politics. And then doubly so, because people enjoy themselves with their favourite regional writing formats. At this point I'm all for enforcing ISO 8601 on every communication involving dates and times.

Some people object that this is turning humans into machines, etc. So be it. Nature isn't perfect, and clear communications doesn't come to us naturally. Yet it is absolutely vital in a technological society.

Re: ASCII and Unicode quotation marks

#175

Post by TeMPOraL »

daemin wrote: post #162
LeoNatan25 wrote: post #158

Earlier quoted context omitted.

Must not be a fan of typography then. Sadly, the tech world is full of the likes of you, so even if people care, in many cases—such as dashes—they are not even aware of the alternatives. Gladly Apple has decided not to follow your ““““opinion”””” by default.

It would be fine if it just changed visually depending on the context, without actually changing the underlying character data.

And here it would be creating a very insidious problem - representation (external format) disconnected from underlying data (internal format).

What I mean is, for example, this: when I see " in a comment, I know the browser is actually storing the " character somewhere in memory. I know that when I send this comment form, HN will receive " character. If I copy and paste the comment into my Emacs, and save it, I know my hard drive now stores the " character.

When external format gets completely disconnected from internal format, understanding anything about what happens with the data gets much more difficult.

Re: ASCII and Unicode quotation marks

#176

Post by mark-r »

jiggunjer wrote: post #170
mark-r wrote: post #39

Earlier quoted context omitted.

If I need a special character, I find a web page or document that has it and use copy/paste.

I have an A4 pinned next to me with all the windows Alt codes. Old-school methods are fastest and I subconsciously learn them by heart over the years.

Nice idea. I don't need them often enough to bother. Do you have a source for the chart?

Re: ASCII and Unicode quotation marks

#177

Post by athenot »

jimmies wrote: post #132

I hate the "" -> “” thing with a passion. I don't know how much productivity that the world has lost with that “” shit. It doesn't look that much better, and it always fucks with me at random times. That shit is on the list of annoying problems that shouldn't exist in the first place, along with the \nl\cr thing, and the txt saved as rtf thing, and the UTF-8 encoding-character-at-the-beginning-of-the-file or whatever…

On a US-English keyboard on the mac, you can always type the proper glyphs directly on the keyboard. That's always been harder on windows where one would have to type some numerical character code each time they wanted some non-ascii character. That's where the auto-replace originated.

On the mac:

    Option [          “      (English open quote)
    Option Shift [    ”      (English close quote)
A lesser-known feature is that some other quotations are also possible from that same US-English keyboard:

    Option \          «      (French open quote)
    Option Shift \    »      (French close quote)
    Option Shift W    „      (German open quote)

Re: ASCII and Unicode quotation marks

#178

Post by nitrogen »

jrochkind1 wrote: post #112
ComputerGuru wrote: post #92

Earlier quoted context omitted.

> but text can contain commas Erm.. text can contain tabs, too. This problem was solved so, so long ago when all the various ANSI/ASCII/whatever encodings were compiled by specifically reserving not one but two characters precisely to serve as field and record delimiters. 0x30 and 0x31 solved not only the problem of having commas or tabs in your text preventing you from treating them as field delimiters, but also all…

I find no matter what you do you will _sometimes_ need escaping. There will, eventually, come a time when you want to embed an ASCII 30 (0x usually means hex, it's actually decimal code 30, but hex 0x1E) RS Record Separator in some record delimited by 30 RS. So you'll need some method of escaping anyway. Or it'll be annoying. I have spent some time working with MARC 21 binary encoding (used for library cataloging rec…

I think it's common to use ESC (0x1b) and then set the high bit on the next byte, so ESC itself would be sent as 0x1b, 0x8b.

Re: ASCII and Unicode quotation marks

#179

Post by Zarel »

daemin wrote: post #162
LeoNatan25 wrote: post #158

Earlier quoted context omitted.

Must not be a fan of typography then. Sadly, the tech world is full of the likes of you, so even if people care, in many cases—such as dashes—they are not even aware of the alternatives. Gladly Apple has decided not to follow your ““““opinion”””” by default.

It would be fine if it just changed visually depending on the context, without actually changing the underlying character data.

The problem is that you can't really detect which single quotation mark to use unambiguously.

    'n'
can either be an abbreviation of "and", or a single-quoted letter "n".

as an abbreviation of "and", it should be rendered as [right single quote]n[right single quote]

’n’

and as a single-quoted letter "n", it should be rendered as [left single quote]n[right single quote]

‘n’

Re: ASCII and Unicode quotation marks

#180

Post by LeoNatan25 »

TeMPOraL wrote: post #175
daemin wrote: post #162

Earlier quoted context omitted.

It would be fine if it just changed visually depending on the context, without actually changing the underlying character data.

And here it would be creating a very insidious problem - representation (external format) disconnected from underlying data (internal format). What I mean is, for example, this: when I see " in a comment, I know the browser is actually storing the " character somewhere in memory. I know that when I send this comment form, HN will receive " character. If I copy and paste the comment into my Emacs, and save it, I know…

What about ligatures and other font character substitution features? What will you Emacs do then and why does it matter? As we advance further, we should strive to disconnect from the technology—which should just work, despite how complex it is.
Post reply on HN