Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compressing Icelandic name declension patterns into a 3.27 kB trie

alexharri.com

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#61

Post by KTibow »

silvestrov wrote: post #43

One more optimization idea: instead of the trie mapping to the suffix string directly, then instead make an array of unique suffixes and let the trie map to the index into the array, e.g. const suffixes = [",,,", "a,u,u,u", ",,i,s", ",,,s", "i,a,a,a", ...]; and then use the index of this list in the var serializedInput = "{e:{n:{ein:0_r: ...

I (Claude Code) tried this and it actually increased the gzipped size by 100b (3456 -> 3556), only reducing the non-compressed size by 20%, likely because gzip is really good at interning repeated patterns already.

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#62

Post by mpascale00 »

charcircuit wrote: post #32
sneak wrote: post #25

Earlier quoted context omitted.

Software sold today does not require maintenance. Software to work in the future requires maintenance. I am not buying future software. I am buying today software. Increasingly I am not buying software at all.

Even ignoring security, bug fixes, new features, etc it is also not fair that you can get value from the app every month, but the developer doesn't get to capture a reward for any of this value. Having people pay monthly for value they get monthly seems reasonable.

I disagree. You can read a book or listen to a record, watch a dvd, unlimited times, having fairly paid upfront a price for the item. A computer is general purpose and lets you check your email every day, hell even lets you create new value in the form of new software, without the manufacturer receiving a royalty.

The idea of capturing reward post-receipt is feudalistic.

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#63

Post by nkrisc »

tempodox wrote: post #52

Is Icelandic name declension deterministic enough that this method reliably works? That would be a lucky break. Language is typically quite messy.

It probably helps that Iceland has a relatively small population and the language is actively managed by the government.

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#64

Post by alexharri »

ryanjshaw wrote: post #46

An interesting article but I was surprised there was no discussion about what humans do to address this problem?

As a native Icelandic speaker, I have an intuition for how to decline names -- I don't really think about it consciously. I'd assume that for most people it's just pattern matching.

Native speakers very frequently decline names in ways that are not technically perfect but sound correct enough. For example, my name (Alex) should not be declined, but people frequently use the declension pattern (Alex, Alex, Alexi, Alexar).

There's some parallel to be drawn with how the compressed trie applies patterns that it's learned to names. That's at least how I thought about it when designing the library.

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#65

Post by charcircuit »

BenjiWiebe wrote: post #50
charcircuit wrote: post #32

Earlier quoted context omitted.

Even ignoring security, bug fixes, new features, etc it is also not fair that you can get value from the app every month, but the developer doesn't get to capture a reward for any of this value. Having people pay monthly for value they get monthly seems reasonable.

Does that mean you'd be in favor of subscriptions for owning a vehicle, rather than paying outright? Or a house? The manufacturer/builder gets paid once, and you get value monthly.

Leasing cars and renting houses is already a common practice. So yes I believe these make sense to exist.

The existence of purchasing cars and houses with no ongoing cost to the builder is due to competition.

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#66

Post by jer0me »

esafak wrote: post #41
dmurray wrote: post #10

For the 800 names that were missing declension data in the database, it seems like the most straightforward thing to do would be to assign their declensions by hand. It shouldn't take a native speaker more than a couple of hours (if some name they haven't seen before is ambiguous, then whatever they guess at least won't sound obviously wrong to other native speakers). Alternatively, very cheap to ask an LLM to do it.…

I wonder if existing LLMs already know these patterns?

The Icelandic government has been proactive about helping OpenAI train its models on the language to stave off extinction: https://openai.com/index/government-of-iceland/

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#67

Post by LoneGeek »

treetalker wrote: post #3

I remember that when I was first learning Spanish in high school, I found a piece of (Windows) software that pelted you with a series of pairs of an infinitive and a tense, and you had to conjugate the infinitive accordingly. (Spanish conjugation typically changes the end of the word; irregular verbs tend to involve stem changes). It was fantastic practice and really ingrained the rules; I became a whiz at it. When I…

If you can read Russian, there is a Python app for morphological analysis called pymorphy3. Documentation: https://pymorphy2.readthedocs.io/en/stable/.

It is based on an OpenCorpora dictionary: https://opencorpora.org/dict.php

This dictionary is based on a Zaliznyak dictionary, which is always referenced in Wiktionary's articles.

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#68

Post by charcircuit »

mpascale00 wrote: post #62
charcircuit wrote: post #32

Earlier quoted context omitted.

Even ignoring security, bug fixes, new features, etc it is also not fair that you can get value from the app every month, but the developer doesn't get to capture a reward for any of this value. Having people pay monthly for value they get monthly seems reasonable.

I disagree. You can read a book or listen to a record, watch a dvd, unlimited times, having fairly paid upfront a price for the item. A computer is general purpose and lets you check your email every day, hell even lets you create new value in the form of new software, without the manufacturer receiving a royalty. The idea of capturing reward post-receipt is feudalistic.

The existence of products in competitive markets is not a counter example to what my point was. I recommend looking at the terms bottom up pricing and top down pricing. The former is about creating a price based off of how much it costs to do business and then adding a profit margin. The latter is creating price in line with how much value it offers customers. The existence of products using bottom up pricing doesn't mean top down pricing does not exist.

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#69

Post by alexharri »

dmurray wrote: post #10

For the 800 names that were missing declension data in the database, it seems like the most straightforward thing to do would be to assign their declensions by hand. It shouldn't take a native speaker more than a couple of hours (if some name they haven't seen before is ambiguous, then whatever they guess at least won't sound obviously wrong to other native speakers). Alternatively, very cheap to ask an LLM to do it.…

It would be good to cover more names for sure -- that's an ongoing process at DIM. Names are frequently added to the approved list of Icelandic names, so there's always going to be some lag.

I would not be confident enough myself to add the data myself since I'd probably be wrong a lot of the time. When reviewing the results for the top 100 unknown names I frequently got results that I thought _might_ be wrong, but I wasn't sure. For those, I looked up similar names in DIM to verify, and often thought "huh, I would not have declined those names like this". For that reason, I rely on the DIM data as the source of truth since it's maintained by experts on the language.

Re: Compressing Icelandic name declension patterns into a 3.27 kB trie

#70

Post by alexharri »

radpanda wrote: post #23

> There are, in fact, 88 approved Icelandic names with this exact pattern of declension, and they all end with “dur”, “tur” or “ður”. … > But that quickly breaks down. There are other names ending with “ður” or “dur” that follow a different pattern of declension My “everything should be completely orderly” comp-sci brain is always triggered by these almost trivial problems that end up being much more interesting. Is…

Hmm, good idea. There are names that have the exact same pronunciation yet have different patterns of declension, for example:

- Ástvaldur -> ur,,i,ar - Baldur -> ur,ur,ri,urs

The "aldur" ending is pronounced in the exact same manner, but applying the declension pattern of "Ástvaldur" to "Baldur" would yield:

- Baldur - Bald - Baldi - Baldar

The three last forms feel very wrong (I asked my partner to verify and she cringed).

Spoken Icelandic is surprisingly close to its written form. I wouldn't expect very different results for the trie if a "phonetic" version of names and their endings were used instead of their written forms

Post reply on HN