Live data from Hacker News

It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

hsivonen.fi

271–280 of 287 posts

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#271

Can anyone recommend a good intro to understanding string encoding article?

https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

After reading this, I did a search for mentions of this article and found this StackOverflow gem. The top answer basically picks up where the JoelOnSoftware article leaves off and filled in the rest of the blanks for me.

https://stackoverflow.com/questions/2241348/what-are-unicode...

Still have more reading to do and a lot to learn but this was super informative, so thank you internet stranger.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#272

I think that string length is one of those things that people (including me) don't realise they never actually want. In a production system, I have never actually wanted string length. I have wanted: - Number of bytes this will be stored as in the DB - Number of monospaced font character blocks this string will take up on the screen - Number of bytes that are actually being stored in memory "String length" is just a…

Very true. Rust’s handling of strings was an eye opener for me.

Seemed awkward but I eventually realized I rarely cared about number of characters. Even when dealing with substrings, I really only cared a means to describe “stuff” before/after not literal indices.

Counting Unicode characters is actually a disservice.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#273

Earlier quoted context omitted.

This is American imperialism at its worst. I'm serious. Lots of people around the world learn programming from sources in their native language, especially early in their career, or when software development is not their actual job. Enforcing ASCII is the same as enforcing English. How would you feel if all cooking recipes were written in French? If all music theory was in Italian? If all industrial specifications we…

Well I'm not American and I can tell you that we do not see English source code as imperialism. In fact it's awesome that we have one common very simple character set and language that works everywhere and can do everything. I have only encountered source code using my native language (German) in comments or variable names in highly unprofessional or awful software and it is looked down upon. You will always get an u…

I once saw an electrical schematic from a non-English speaking designer.

None of the signals were intuitive because they weren’t the typical English abbreviations!

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#274

Earlier quoted context omitted.

> do you have a script you'd like me to run against python3? Just toss me a pastebin link, and ideally the version of python3 to run Here's a diff: diff --git a/ibmfilter b/ibmfilter index 245d32c..2633335 100755 --- a/ibmfilter +++ b/ibmfilter @@ -1,6 +1,5 @@ -#!/usr/bin/python2 -tt -# vim:set fileencoding=utf-8 - +#!/usr/bin/python3 + from subprocess import * import sys import os, select @@ -10,8 +9,8 @@ special =…

So, the patch failed with both my original file and the pastebin one - perhaps due to indentation of Hacker News, so I manually applied since it did seem pretty straightforward - honestly given how short the file was, it would have taken up the same amount of space here as the diff I think, but hopefully I applied it correctly. Manual copy/paste in python always worries me w/ the significant white-space thing (one of…

> perhaps due to indentation of Hacker News, so I manually applied since it did seem pretty straightforward

Yes, that would be exactly why. You can use e.g. `sed` to remove leading whitespace from each line (I used it to add the leading whitespace for posting).

> ... that said, it does do one odd thing (following is output on launching):

Yes, that's the warning I mentioned. The original code requests to use a buffer size of 1, which is no longer supported (it now means to use line buffering).

> It is much slower to render.

Avoiding line buffering (by requiring a buffer size of 2 or more) might fix that. Actually, it might be a good idea to use a significantly larger buffer, so that e.g. an entire ANSI colour code can be read all at once.

The other issues are, I'm pretty sure, because of other things that changed in how `subprocess` works. Fixing things at this level would indeed require quite a bit more hacking around with the low-level terminal APIs.

> I mean, I only formed my opinion that maybe there was a problem with python3 byte/string handling to just how many attempts there were... Were they trying to do things in a more idiomatic python3 fashion? Did the python3 APIs change? Does the error hint at something more concerning?

Most likely, other attempts either a) didn't understand what the original code was doing in precise enough detail, or b) didn't know how to send binary data to standard output properly (Python 3 defaults to opening standard output as a text stream).

All of that said: I think that nowadays you should just be able to get a build of NetHack that just outputs UTF-8 characters directly; failing that, you can use the `locale` command to tell your terminal to expect cp437 data.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#275
post #257
post #254

Earlier quoted context omitted.

> The reason for this lack of handling of Unicode linguistically is simply to make the lexer smaller and faster. No, it is actually for security reasons. Once you allow non-ASCII identifiers, identifiers will become non identifiable. Only zig recognized that. Nim allows insecure identifiers. https://github.com/rurban/libu8ident/blob/master/doc/c11.md#...

Reading is fundamental. I was referring to the Nim lexer. Obviously the reason that it "allows insecure identifiers" is not "actually for security reasons". It is, as I stated, for reasons of performance ... I know this from reading the code and the author's statements.

Yes, you are right. Andi didn't care at all, same as PHP.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#276

Earlier quoted context omitted.

So, the patch failed with both my original file and the pastebin one - perhaps due to indentation of Hacker News, so I manually applied since it did seem pretty straightforward - honestly given how short the file was, it would have taken up the same amount of space here as the diff I think, but hopefully I applied it correctly. Manual copy/paste in python always worries me w/ the significant white-space thing (one of…

> perhaps due to indentation of Hacker News, so I manually applied since it did seem pretty straightforward Yes, that would be exactly why. You can use e.g. `sed` to remove leading whitespace from each line (I used it to add the leading whitespace for posting). > ... that said, it does do one odd thing (following is output on launching): Yes, that's the warning I mentioned. The original code requests to use a buffer…

Well, I also use it for other old terminal apps (BBS games), and the remapping of characters was fun too, using unicode to make certain features more distinguishable. (Only downside is it messes up the crystal ball, if you remapped that char, since it wants the standard values, but you can just use memorised values or turn it off temporarily)

The unfortunate thing is the "lag" is a bit annoying with some apps, so I'll probably still use the python2 one for now.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#277

Earlier quoted context omitted.

> perhaps due to indentation of Hacker News, so I manually applied since it did seem pretty straightforward Yes, that would be exactly why. You can use e.g. `sed` to remove leading whitespace from each line (I used it to add the leading whitespace for posting). > ... that said, it does do one odd thing (following is output on launching): Yes, that's the warning I mentioned. The original code requests to use a buffer…

Well, I also use it for other old terminal apps (BBS games), and the remapping of characters was fun too, using unicode to make certain features more distinguishable. (Only downside is it messes up the crystal ball, if you remapped that char, since it wants the standard values, but you can just use memorised values or turn it off temporarily) The unfortunate thing is the "lag" is a bit annoying with some apps, so I'l…

oh. and, using 2 did silence the warning, but performance was still bad compared to python2, and it still requires ctrl-c to exit

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#278
post #102

Earlier quoted context omitted.

It's neither American nor imperialism -- those are both category mistakes. Andreas Rumpf, the designer of Nim, is Austrian. All the keywords of Nim are in English, the library function names are in English, the documentation is in English, Rumpf's book Mastering Nim is in English, the other major book for the language, Nim In Action (written by Dominik Picheta, nationality unknown but not American) is in English ...…

I mean, the keywords of a programming language have to be in some language (unless you go the cursed route of Excel). I'm arguing against the position that non-ASCII identifiers should be disallowed.

> I'm arguing against the position that non-ASCII identifiers should be disallowed.

Maybe I'm tired, but I've read this multiple times and can't quite figure out your desired position.

I *think* you are in favor of non -ASCII identifiers?

Like I said, I must be tired.

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#279
post #69

Earlier quoted context omitted.

That has nothing to do with UTF-8; that's a Unicode issue, and one that's entirely unescapable if you are the Unicode Consortium and your goal is to be compatible with all legacy charsets.

Yep, that's the point I was making - that choosing fixed 4-byte code-points doesn't significantly reduce the complexity of capturing everything that Unicode does.

Thanks for explaining!

Re: It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

#280
post #197
post #96

Earlier quoted context omitted.

You might like https://fsharpforfunandprofit.com/series/property-based-test...

In this particular case it was simply a matter of not enough corner cases defined. I was however using property-based testing, doing things like reversing then un-reversing the UTF-8 strings, re-ordering code points, merging strings, etc for verification. The datasets were in a variety of languages (including emojis) and so I mistakenly thought I had covered all the bases. But thank you for the link, it's turning out…

Nice! Glad to be of service.

Python, of all languages, probably has the best property based testing library out there with "hypothesis". I sometimes even use it to drive tests for my Haskell and OCaml and Rust code. The author of Hypothesis wrote a few nice articles about why his approach is better (and I agree), however I can't find them at the moment..

Post reply on HN