The Bluesky Dictionary • Hacker News Board

Re: The Bluesky Dictionary

#11

Post by bangaladore » Wed, Aug 06, 2025, 10:09 PM UTC

wantlotsofcurry wrote: post #10

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

Maybe I'm being naive, but with only ~275k words to check against, this doesn't seem like a particularly hard problem. Ingest post, split by words, check each word via some db, hashmap, etc... and update metadata.

Re: The Bluesky Dictionary

#12

Post by gpm » Wed, Aug 06, 2025, 10:09 PM UTC

wantlotsofcurry wrote: post #10

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

Probably just a big hashtable mapping word -> the number of times it's been seen, and another hashset of all the words it hasn't seen. When a post comes in you hash all the words in it and look them up in the hashtable, increment it, and if the old value was 0 remove it from the hash set.

250k words at a generous 100 bytes per word is only 25MB of memory...

Re: The Bluesky Dictionary

#13

Post by spullara » Wed, Aug 06, 2025, 10:10 PM UTC

I did this against a pretty large tweet archive and got hits on about 125k of the words in the unix dictionary.

Re: The Bluesky Dictionary

#14

Post by f311a » Wed, Aug 06, 2025, 10:15 PM UTC

wantlotsofcurry wrote: post #10

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

You can probably fit all words under 10-15MB of memory, but memory optimisations are not even needed for 250k words...

Trie data structures are memory-efficient for storing such dictionaries (2-4x better than hashmaps). Although not as fast as hashmaps for retrieving items. You can hash the top 1k of the most common words and check the rest using a trie.

The most CPU-intensive task here is text tokenizing, but there are a ton of optimized options developed by orgs that work on LLMs.

Re: The Bluesky Dictionary

#15

Post by pona-a » Wed, Aug 06, 2025, 10:18 PM UTC

For a moment I thought it would be an AT-Proto based Urban Dictionary clone.

Re: The Bluesky Dictionary

#16

Post by stwrzn » Wed, Aug 06, 2025, 10:31 PM UTC

wantlotsofcurry wrote: post #10

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

I very much hope that the backend uses one of the bluesky jetstream endpoints. When you only subscribe to new posts, it provides a stream of around 20mbit/s last time I checked, while the firehose was ~200mbit/s.

Re: The Bluesky Dictionary

#17

Post by atlgator » Wed, Aug 06, 2025, 10:32 PM UTC

75345d4c wrote: post #7

I just saw it indexed "eluvium," but the post was referring to a band with that same name

I checked out the author's other projects and this is common issue. For example, he has a "lean checker" for bluesky that claims it is right-leaning simply because of all the people saying "That's right," "He was right," etc. None of the supposed right-leaning posts were actually conservative in nature. They just used to word right to mean correct.

Re: The Bluesky Dictionary

#18

Post by timeon » Wed, Aug 06, 2025, 10:33 PM UTC

GalaxyNova wrote: post #3

fascinating! I think it's really cool that this is possible, and at the same time kine of sad that the norm is slowly moving towards more locked-down APIs.

> slowly moving towards

Depends what we accept as norm.

Re: The Bluesky Dictionary

#19

Post by avibagla1 » Wed, Aug 06, 2025, 10:39 PM UTC

wantlotsofcurry wrote: post #10

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

Hey! this is my site - it's not all that complex, i'm just using a sqlite db with two tables - one for stats, the other for all the words that's just word | count | first use | last use | post.

I... did not expect this to be so popular

Re: The Bluesky Dictionary

#20

Post by avibagla1 » Wed, Aug 06, 2025, 10:39 PM UTC

stwrzn wrote: post #16

wantlotsofcurry wrote: post #10

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

I very much hope that the backend uses one of the bluesky jetstream endpoints. When you only subscribe to new posts, it provides a stream of around 20mbit/s last time I checked, while the firehose was ~200mbit/s.

yes it does!