Live data from Hacker News

Trie in JavaScript: The data structure behind autocomplete

stackfull.dev

21–30 of 59 posts

Re: Trie in JavaScript: The data structure behind autocomplete

#21
post #18

Earlier quoted context omitted.

There's a searching concept known as edit (levenshtein) distance that is close but not exactly what you're talking about. To rephrase your question: is there a datastructure that can pre-compute the levenshtein distance for a set or results? Yes, and it's crazy complicated. See Levenshtein Automata[1]. In reality, I would let the professionals handle this by either using Open/ElasticSearch or Apache Lucene. [1] http:…

>I would let the professionals handle this... If this site doesn't contain "the professionals" then the phrase has zero meaning.

Fair enough. "Crazy complicated" and "the professionals" are both intended as subjective terms. To ME this stuff might as well be magic.

Re: Trie in JavaScript: The data structure behind autocomplete

#22

The basic trie only matches exact consecutive characters. Does anyone know the data structure/trie variant for matching partial sequential characters? e.g. ABD matches A Brown Dog ^ ^ ^

There's a searching concept known as edit (levenshtein) distance that is close but not exactly what you're talking about. To rephrase your question: is there a datastructure that can pre-compute the levenshtein distance for a set or results? Yes, and it's crazy complicated. See Levenshtein Automata[1]. In reality, I would let the professionals handle this by either using Open/ElasticSearch or Apache Lucene. [1] http:…

An in-between option that is relatively simple and a rather fast (though not as fast as the automata) optimization to levenstein distance and based on Tries is: http://stevehanov.ca/blog/?id=114.

Re: Trie in JavaScript: The data structure behind autocomplete

#23
Tries are also used to efficiently store names of functions in macOS / iOS binaries (Mach-O format) and work great for that purpose because a lot of functions have the same prefixes [0].

Indexes in mongodb use a similar concept (they call that prefix compression [1]), and that allows them to store indexes more efficiently.

We use them at Pyroscope and we wrote a blog post about our storage design [2] There are even some animations :)

[0] https://www.m4b.io/reverse/engineering/mach/binaries/2015/03...

[1] https://docs.mongodb.com/manual/reference/glossary/#std-term...

[2] https://github.com/pyroscope-io/pyroscope/blob/main/docs/sto...

Re: Trie in JavaScript: The data structure behind autocomplete

#24

Fun fact: the trie data structure was invented by Ed Fredkin, a pioneer in Quantum computing as well as cellular automata

This is the best kind of correct but it's an interesting phrasing since Fredkin is a kind of Quantum supremacy skeptic (hotkey-find Classicatopia in: https://windowsontheory.org/2017/10/30/the-different-forms-o...).

Fredkin's contribution were to reversible computation and since Quantum computers are a kind of reversible computer, many ideas and gate notions carried over.

Re: Trie in JavaScript: The data structure behind autocomplete

#25
post #18

Earlier quoted context omitted.

>I would let the professionals handle this... If this site doesn't contain "the professionals" then the phrase has zero meaning.

Fair enough. "Crazy complicated" and "the professionals" are both intended as subjective terms. To ME this stuff might as well be magic.

I'd flip that and decide that there is no algorithm that exists and is documented that you can't implement yourself, for amusement and educational value or, given time and the will, more than that. Just making that decision has value for you even if you don't actually do it terribly often.

Same as you can design a cpu in HDL and flash it to an FPGA. (The "Nand to Tetris" course seems popular) Same as you can write an OS. Same as you can write a compiler and/or interpreter.

Believing these things (and they're actually true!) gets us all away from learned helplessness. There's enough of that when it comes to actual silicone...

Nobody understands it all. _You_ understand as much of it as you choose, in the direction that takes you, as deep as you want to go. It's just work, a lot of it, but no more than that.

Re: Trie in JavaScript: The data structure behind autocomplete

#26

The basic trie only matches exact consecutive characters. Does anyone know the data structure/trie variant for matching partial sequential characters? e.g. ABD matches A Brown Dog ^ ^ ^

If I understand correct, you're wondering how you might match "a bro do" with "a brown dog" or "a brown door", etc. Similar to what Google does. I've done this before. It's not a single data structure, but a process.

First, you normalize your strings to index (convert to lower case, throw out stop words such as "a", "the", "is", etc.) then you generate prefixes (or n-grams, I suppose). Say you want to do musicians and you have "Bob Dylan" as an entry. That would become "bob dylan" which would generate the prefixes "bo", "bob", "bob d", etc. You include spaces but once you hit a space, you start generating new prefixes. So you would also start doing "dy", "dyl", "dyla", and "dylan". The max length of the prefixes depends on how much memory you're willing to use.

Lets say Bob Dylan is mapped to some musician table in your DB and he has id 1000. I've done this with Redis, but you can use a trie as well. You'd store every prefix in the trie and the data on the node would be an array of musician ids. Now the magic: if you search "bo dyl" you would look up the prefix "bo", grab all the ids and then look up the prefix "dyl" and grab the ids (you would also try the full "bo dyl" first, but let's just assume nothing is there). Once you have those two arrays, you would take the intersection of them. Which, in our case, would leave an array containing id = 1000 along with any others that matched both.

That's, in general, how a simple autocomplete works. But you're better off using Redis or Elasticsearch if your data set is large. And if it's not, well... a simple hashmap is probably competitive and easier than tries.

Re: Trie in JavaScript: The data structure behind autocomplete

#29
post #27

for (let character... Please use const. Everything should be const until there is a proven need otherwise.

I reckon there's a kind of broken rationale where people think, "ok well this value will mutate over time so I can't possibly use a const", even though a new block scope is initialized on each iteration. The `for(let i in blah)` idiom is basically cargo-culted now without thought.

Re: Trie in JavaScript: The data structure behind autocomplete

#30

The basic trie only matches exact consecutive characters. Does anyone know the data structure/trie variant for matching partial sequential characters? e.g. ABD matches A Brown Dog ^ ^ ^

A quick hack for this is to create variations and include them in your main trie (with links to the canonical value/object/etc.). E.g. you might create a variation w/o vowels.

There are heuristics you can apply here rather than implementing full fuzziness or levenshtein comparators etc. Ultimately it depends on your use-case and what common variations users might try to use.

Post reply on HN