Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

deephn.org

Post reply on HN
Print view View original

81–90 of 118 posts

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#81

Post by localhost »

wolfgarbe wrote: post #72
localhost wrote: post #65

Have you tried against a corpus of text like email? How well does that work? I've been frustrated with the search speed (and results) from email search in general and was wondering if I could do better with an offline search?

That would definitely work. We are working on local client that could index your local PDF and Word documents. So importing emails would be a nice addition.

I'd be interested in seeing how well it performs vs. integrated search. Outlook search (which is what I use) suffers from performance and relevance issues, yet when I tried this experiment with another full-text search provider (Azure Search) it performed worse on relevancy.

I use email as kind of an activity stream - generally I can remember roughly the people involved and the time period. Would be lovely to have search that performs well in this regard.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#82

Post by julienchastang »

dang wrote: post #56
myth_drannon wrote: post #9

Earlier quoted context omitted.

Earlier post 2009: https://news.ycombinator.com/item?id=599852 "Well this is an exceptionally cute idea, but there is absolutely no way that anyone is going to have any faith in this currency." Written by co-founder/CEO of Pachyderm and first employee of RethinkDB none the less.

People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. But if you go back and look at that original 3-comment Bitcoin thread, what you find is that those 3 comments span an entire spectrum: elated, skeptical, and bewildered: https://news.ycombinator.com/item?id=599852 . The first cryptocurrency thread on HN—about 9 months earlier—was even more int…

What this also shows to me is just how difficult it is to predict the future. Here is another thought experiment: There is probably a wonderful technology / opportunity, perhaps lying in plain sight, that we will be kicking ourselves in ten years for not having gotten involved in. But what is it? "Every day brings new opportunities to become very rich, most of which we are bound to miss" [1]

[1] https://www.wsj.com/articles/missed-teslas-12-551-rise-dont-...

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#84

Post by karterk »

goodmachine wrote: post #76
karterk wrote: post #49

Really cool demo. I'm curious to know what kind of hardware this is this hosted on. Full disclosure: I work on a similar fast, typo tolerant, fuzzy search engine search engine called Typesense ( https://github.com/typesense/typesense ).

Yeah, was also interested. It would be nice to see Typesense take up the same challenge for comparison ;)

Happy to benchmark if the dataset is shared!

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#85

Post by wolfgarbe »

morrbo wrote: post #57
wolfgarbe wrote: post #24

Earlier quoted context omitted.

DeeepHN is powered by our SeekStorm search-as-a-service, which has a full-featured API: https://seekstorm.com/docs SeekStorm is intended that the user can index and search their own private data. But if there is demand we could also allow to search in a central public index for web data or other public data.

How does seekstorm perform on large documents? Say documents 1MB-10MB in size, are you able to search entire docs or just paragraphs within the doc?

Currently the maximum size of a single document is limited to 1 MByte. That is an artificial limitation resulting from the fact, that SeekStorm does not only indexes the content of the document, but also stores the original document. Somewhere there has to be a limit.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#87

Post by arafalov »

wolfgarbe wrote: post #80
arafalov wrote: post #71

Earlier quoted context omitted.

It seems you tried MMapDirectory and then commented it out. https://github.com/wolfgarbe/LuceneBench/blob/master/LuceneB... So, you may be hitting SimpleFSDirectory instead, which does have issues with too many searches. Could you share the reasons, MMapDirectory did not work for you?

"NIOFSDirectory and MMapDirectory implementations face file-channel issues in Windows and memory release problems respectively. To overcome such environment peculiarities Lucene provides the FSDirectory.open() method. When invoked, it tries to choose the best implementation depending on the environment." https://www.baeldung.com/lucene-file-search

Right. I found you are running on Windows after posting the comment. Makes sense, though there were some fixes in the latest Lucene I believe.

The Lucene's own benchmarks are at: https://home.apache.org/~mikemccand/lucenebench/ , though I admit to not know enough about benchmarking to form a strong opinion.

Either way, good luck with the project/service. More competition is always great. The open-source components look interesting too.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#88

Post by freediver »

wolfgarbe wrote: post #69
freediver wrote: post #53

Earlier quoted context omitted.

What is being used as a crawler and is it integrated with Seekstorm? The same article referenced here https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash... contains the phrase 'well-defined' Any idea why doesn't the article surface when searching for this ?

The crawler is a part of SeekStorm. "Well-defined": Just a guess: We are doing key text extraction, i.e. we try not to index boilerplate stuff an and menu items. As "well-defined" is within a short list item, it might be accidentally skipped. So, that is not yet perfect, and considering the diversity in web page structure it probably never will. But we will try to improve.

Yes, if the preview is the representation of what you have indexed, then half of that article is missing. You may have identified the weakest link in your stack - crawler/extractor (which is notoriously hard to do, would be good if you provided more detail eg. do you use headless browser or simple GET request, do you crawl PDFs etc). Little use of the advanced stack on top of it, if the data does not end in the index in the first place. Hope you provide an update on this in the future. I'd probably sign up for a plan.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#89

Post by wolfgarbe »

superasn wrote: post #78

Very cool and useful. I was trying to find all comments on HN which contains the word youtube.com (basically trying to find some recently linked to videos from comments) but it returns 0 results when i set the search filter to comments(1). Any idea what I'm doing wrong? https://deephn.org/?q=youtube.com&sort=time&in=Comments

We will look into this.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#90

Post by wolfgarbe »

karterk wrote: post #49

Really cool demo. I'm curious to know what kind of hardware this is this hosted on. Full disclosure: I work on a similar fast, typo tolerant, fuzzy search engine search engine called Typesense ( https://github.com/typesense/typesense ).

SeekStorm uses dedicated root servers with NVMe SSDs, hosted by IONOS https://www.ionos.com/servers/intel-servers
Post reply on HN