Live data from Hacker News

Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

deephn.org

31–40 of 118 posts

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#31
post #8

This is nice! How are the stories tagged?

We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.

[deleted]

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#32
Interesting, it doesn't seem to show the dead pages. I have been indexing dead HN pages for several years now, with the idea that eventually I'd run some statistics on it to better understand both the auto-dead filters and the people who are submitting dead content.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#33
post #17

I assume you are using Symspell for fuzzy search and prioritize candidate frequency over edit distance. My input "algolai" returns results for "angola" even though "algolia" was just 1 transposition away. Anyway just dropping in to say that stuff on your github helped me figure out a lot while building a search engine from scratch. Good luck with SeekStorm!

We are combining auto correction (SymSpell) and auto completion (PruningRadixTrie). That is sometimes tricky. We are using a static spelling correction dictionary which does not contains the term algolia, but angola. SymSpell is very fast, but requires a lot of memory. Therefore we are using a static dictionary, and not a dynamic dictionary derived from each individual index of each customer. Auto completion on the other hand we are dynamically generating for each individual index/customer.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#35

I was hoping it would do full-text search of the text inside the linked pages, like hndex.org but with ranking.

Yes, it does exactly that. If you check "Web" on the left sidebar, and uncheck "Stories" it will find the posts where "friendship" occurs in the linked web pages. But we still show the title of the original hacker news story in the search result and highlight the search term if it ALSO occurs in the title.

If you scroll long enough within the search results you will finally reach results where the search term is not in the title, but only within the linked web page.

It is more obvious if you search for queries which are less popular than "friendship"

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#37

Seems odd to publish a benchmark against Lucene where your results are "Lucene crashes at 4 concurrent users". Since there are thousands of people using Lucene and products built on Lucene every day, I think the issue may be in your benchmarking code.

There is no secret about the benchmark, its all open source: https://github.com/wolfgarbe/LuceneBench

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#39
post #7

It’s fun to see Bitcoin posts from 2008-2010. I wonder how many bought and are now retired because of them. Seeing recommendations to use MtGox is pretty funny too https://news.ycombinator.com/item?id=1998144

Also fun to see the contrarian Venus explorers are at least a decade old ;)

SpaceX aims to put man on Mars in 10-20 years

https://news.ycombinator.com/item?id=2479053

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#40
The effort definetely gets my upvote.

What did you use to crawl the pages and how long did it take? Curious about your experience doing it and crawler integration with Seekstorm.

Is is possible to easilly expand the index with embeddings (vectors) and perform semantic search in parallel?

Your pricing indicates that hosting similar index as this demo would cost $500/month. Wondering what kind of infrastructure is supporting the demo? Thanks!

ps. Small quirk: https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash...

First three tags seem not to be relevant to the post itself.

Post reply on HN