Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

deephn.org

Post reply on HN
Print view View original

51–60 of 118 posts

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#51

Post by wolfgarbe »

freediver wrote: post #40

The effort definetely gets my upvote. What did you use to crawl the pages and how long did it take? Curious about your experience doing it and crawler integration with Seekstorm. Is is possible to easilly expand the index with embeddings (vectors) and perform semantic search in parallel? Your pricing indicates that hosting similar index as this demo would cost $500/month. Wondering what kind of infrastructure is supp…

Crawling speed is between 100...1000 pages per second. We crawled about 4 million linked unique web pages

The pricing for an Index like DeepHN would be $99/month: While we are indexing 30 million Hacker news posts, for DeepHN we are combining a single HN story with all its comments and its linked webpage into a single SeekSorm document. So that we index just 4 million SeekStorm documents.

Yes, it would be possible to expand index with embeddings (vectors) and perform semantic search. This would we an auxiliary step between crawling and indexing.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#52

Post by zoomablemind »

Amazingly fast! I'm used to seeing a Firefox's load progress bar for most of requests on my mobile, yet here the results "just show up".

More observations:

- When trying to go back after clicking the original HN link, there's no response.

- Clicking on what I understand are tags (hashtags) in a given post has no effect.

- Sorting by date is working, though it'd be nice to see the date.

Also a question: how do you generate the tags (hashtags) from the original contents?

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#53

Post by freediver »

wolfgarbe wrote: post #51
freediver wrote: post #40

The effort definetely gets my upvote. What did you use to crawl the pages and how long did it take? Curious about your experience doing it and crawler integration with Seekstorm. Is is possible to easilly expand the index with embeddings (vectors) and perform semantic search in parallel? Your pricing indicates that hosting similar index as this demo would cost $500/month. Wondering what kind of infrastructure is supp…

Crawling speed is between 100...1000 pages per second. We crawled about 4 million linked unique web pages The pricing for an Index like DeepHN would be $99/month: While we are indexing 30 million Hacker news posts, for DeepHN we are combining a single HN story with all its comments and its linked webpage into a single SeekSorm document. So that we index just 4 million SeekStorm documents. Yes, it would be possible to…

What is being used as a crawler and is it integrated with Seekstorm?

The same article referenced here https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash... contains the phrase 'well-defined'

Any idea why doesn't the article surface when searching for this ?

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#54

Post by wolfgarbe »

youngNed wrote: post #48

So, can you explain what is going on with 'covid' articles from <2019 - are these where the original page that was linked to has now become a spam link?

Not necessarily a spam link. Linked pages from older post have sometimes updated their content.

Sometimes this is legitimate, sometimes spam.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#55

Post by daibo »

wolfgarbe wrote: post #33
daibo wrote: post #17

I assume you are using Symspell for fuzzy search and prioritize candidate frequency over edit distance. My input "algolai" returns results for "angola" even though "algolia" was just 1 transposition away. Anyway just dropping in to say that stuff on your github helped me figure out a lot while building a search engine from scratch. Good luck with SeekStorm!

We are combining auto correction (SymSpell) and auto completion (PruningRadixTrie). That is sometimes tricky. We are using a static spelling correction dictionary which does not contains the term algolia, but angola. SymSpell is very fast, but requires a lot of memory. Therefore we are using a static dictionary, and not a dynamic dictionary derived from each individual index of each customer. Auto completion on the o…

I am combining symmetric deletes with a trie for fuzzy autocompletion and it is working pretty well.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#56

Post by dang »

myth_drannon wrote: post #9
exdsq wrote: post #7

It’s fun to see Bitcoin posts from 2008-2010. I wonder how many bought and are now retired because of them. Seeing recommendations to use MtGox is pretty funny too https://news.ycombinator.com/item?id=1998144

Earlier post 2009: https://news.ycombinator.com/item?id=599852 "Well this is an exceptionally cute idea, but there is absolutely no way that anyone is going to have any faith in this currency." Written by co-founder/CEO of Pachyderm and first employee of RethinkDB none the less.

People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. But if you go back and look at that original 3-comment Bitcoin thread, what you find is that those 3 comments span an entire spectrum: elated, skeptical, and bewildered: https://news.ycombinator.com/item?id=599852.

The first cryptocurrency thread on HN—about 9 months earlier—was even more interesting: https://news.ycombinator.com/item?id=253963. It contains another skeptical comment: "The bigger problem is that everyone has a incentive to run their computers day and night cranking out solutions, which burns up lots of natural resources and processor time for a zero-sum result." (https://news.ycombinator.com/item?id=253999), which seems astonishingly prescient now, even though the objection is still hotly (no pun intended) debated.

Cherry-picking the wrongest-seeming skeptical comment from that collection of data points is a case study in survivorship bias. (I don't mean to pick on you personally! This is common of course.) The infamous Dropbox comment from 2008 is similar in that it has gotten repeated out of context in a way that is unfair to the original commenter (https://news.ycombinator.com/item?id=23229275). When we repeat these things, it says more about us than it does about the thing we're repeating.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#57

Post by morrbo »

wolfgarbe wrote: post #24
yewenjie wrote: post #11

Nice to see full-text of the articles linked and really fast. Do you provide an API like Algolia does for HN?

DeeepHN is powered by our SeekStorm search-as-a-service, which has a full-featured API: https://seekstorm.com/docs SeekStorm is intended that the user can index and search their own private data. But if there is demand we could also allow to search in a central public index for web data or other public data.

How does seekstorm perform on large documents? Say documents 1MB-10MB in size, are you able to search entire docs or just paragraphs within the doc?

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#58

Post by wolfgarbe »

zoomablemind wrote: post #52

Amazingly fast! I'm used to seeing a Firefox's load progress bar for most of requests on my mobile, yet here the results "just show up". More observations: - When trying to go back after clicking the original HN link, there's no response. - Clicking on what I understand are tags (hashtags) in a given post has no effect. - Sorting by date is working, though it'd be nice to see the date. Also a question: how do you gen…

>> When trying to go back after clicking the original HN link, there's no response.

The original HN link is opened in a new browser page/tab. The search results should remain unchanged in the previous tab. So you don't use the back button, but the previous browser tab.

>> Clicking on what I understand are tags (hashtags) in a given post has no effect.

Search for "google", go to the first result. there is a hashtag "oracle", click and the search results are filtered by that hashtags. Please post an example where its doesn't work, so that we can fix it.

The date is shown in the preview panel on the right hand side. But wa are thinking adding the time also to the result list.

We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#59

Post by lotsofpulp »

dang wrote: post #56
myth_drannon wrote: post #9

Earlier quoted context omitted.

Earlier post 2009: https://news.ycombinator.com/item?id=599852 "Well this is an exceptionally cute idea, but there is absolutely no way that anyone is going to have any faith in this currency." Written by co-founder/CEO of Pachyderm and first employee of RethinkDB none the less.

People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. But if you go back and look at that original 3-comment Bitcoin thread, what you find is that those 3 comments span an entire spectrum: elated, skeptical, and bewildered: https://news.ycombinator.com/item?id=599852 . The first cryptocurrency thread on HN—about 9 months earlier—was even more int…

> People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior.

I like to see wrong “obvious” predictions to humble myself, even if not from me.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#60

Post by doh »

abcc8 wrote: post #45

I'm curious how this site is compliant with copyright law. I've thought of working on similar projects, but the possibility of being sued has dissuaded me.

Search engines are generally considered fair use [0], at least in US. So up until HN doesn't send cease and desist, it's in the clear. Once it does, then it's all about what the claims are. Google litigated the crap out of this [1] over the years, so there is a positive precedent minefield all over the place.

[0] https://www.everycrsreport.com/reports/RL33810.html

[1] https://en.wikipedia.org/wiki/Fair_use#Text_and_data_mining

Post reply on HN