Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages • Hacker News Board

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#21

Post by wolfgarbe » Tue, Apr 13, 2021, 2:13 PM UTC

yamrzou wrote: post #8

This is nice! How are the stories tagged?

We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#22

Post by 0xmohit » Tue, Apr 13, 2021, 2:13 PM UTC

Curious to know how does it differ from HN Search [1]? HN Search API documentation [2].

[1] https://hn.algolia.com

[2] https://hn.algolia.com/api

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#23

Post by crazypython » Tue, Apr 13, 2021, 2:14 PM UTC

I was hoping it would do full-text search of the text inside the linked pages, like hndex.org but with ranking.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#24

Post by wolfgarbe » Tue, Apr 13, 2021, 2:18 PM UTC

yewenjie wrote: post #11

Nice to see full-text of the articles linked and really fast. Do you provide an API like Algolia does for HN?

DeeepHN is powered by our SeekStorm search-as-a-service, which has a full-featured API: https://seekstorm.com/docs

SeekStorm is intended that the user can index and search their own private data.

But if there is demand we could also allow to search in a central public index for web data or other public data.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#25

Post by wolfgarbe » Tue, Apr 13, 2021, 2:20 PM UTC

nitramm wrote: post #14

It is really great tool too see what used to be very popular posts in the past. I have one observation. Posts linking to twitter - https://deephn.org/?sort=score&filter=%7B%22score%22%3A%7B%2... - have most of the preview occupied with navigation and other not interested parts. So maybe it is possible to use some hacky solution to show the content part for the most popular websites, or if multiple pages are downloade…

You are right. The preview is far from perfect. We just wanted to ship early. We will continue to fix and improve.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#26

Post by jankotek » Tue, Apr 13, 2021, 2:23 PM UTC

0xmohit wrote: post #22

Curious to know how does it differ from HN Search [1]? HN Search API documentation [2]. [1] https://hn.algolia.com [2] https://hn.algolia.com/api

It searches content of referenced webpages. Also has filtering by year intervals (for example 2012-2016), domains, tags etc.. And I find usability better.

I think it is great product.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#27

Post by pelagicAustral » Tue, Apr 13, 2021, 2:24 PM UTC

syntaxing wrote: post #3

Neat but the back button hijacking is not cool.

I'm a bit curious about this 'back button hijacking', since the bug has now been fixed I'm afraid I'm not able to test. What was the unintended behaviour?

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#28

Post by reivalc » Tue, Apr 13, 2021, 2:25 PM UTC

Neat! I wonder if you quantified the performance of your search somehow? I guess labelling data for that would be laborious.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#29

Post by johtso » Tue, Apr 13, 2021, 2:26 PM UTC

crazypython wrote: post #23

I was hoping it would do full-text search of the text inside the linked pages, like hndex.org but with ranking.

I think it is supposed to be doing that?

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#30

Post by crazypython » Tue, Apr 13, 2021, 2:29 PM UTC

johtso wrote: post #29

crazypython wrote: post #23

I was hoping it would do full-text search of the text inside the linked pages, like hndex.org but with ranking.

I think it is supposed to be doing that?

If I search "friendship" all the article titles have the word friendship in them. In my experience many HN titles don't contain the topic word.