This is nice! How are the stories tagged?
We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.
Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
31–40 of 118 posts
Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#32Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#33I assume you are using Symspell for fuzzy search and prioritize candidate frequency over edit distance. My input "algolai" returns results for "angola" even though "algolia" was just 1 transposition away. Anyway just dropping in to say that stuff on your github helped me figure out a lot while building a search engine from scratch. Good luck with SeekStorm!
Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#34Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#35I was hoping it would do full-text search of the text inside the linked pages, like hndex.org but with ranking.
If you scroll long enough within the search results you will finally reach results where the search term is not in the title, but only within the linked web page.
It is more obvious if you search for queries which are less popular than "friendship"
Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#36Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#37Seems odd to publish a benchmark against Lucene where your results are "Lucene crashes at 4 concurrent users". Since there are thousands of people using Lucene and products built on Lucene every day, I think the issue may be in your benchmarking code.
Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#38Love it! I would kill for a reverse order button to see oldest posts first
Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#39It’s fun to see Bitcoin posts from 2008-2010. I wonder how many bought and are now retired because of them. Seeing recommendations to use MtGox is pretty funny too https://news.ycombinator.com/item?id=1998144
SpaceX aims to put man on Mars in 10-20 years
Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages
#40What did you use to crawl the pages and how long did it take? Curious about your experience doing it and crawler integration with Seekstorm.
Is is possible to easilly expand the index with embeddings (vectors) and perform semantic search in parallel?
Your pricing indicates that hosting similar index as this demo would cost $500/month. Wondering what kind of infrastructure is supporting the demo? Thanks!
ps. Small quirk: https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash...
First three tags seem not to be relevant to the post itself.