Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

deephn.org

Post reply on HN
Print view View original

71–80 of 118 posts

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#71

Post by arafalov »

wolfgarbe wrote: post #37
matt_wilson_206 wrote: post #13

Seems odd to publish a benchmark against Lucene where your results are "Lucene crashes at 4 concurrent users". Since there are thousands of people using Lucene and products built on Lucene every day, I think the issue may be in your benchmarking code.

There is no secret about the benchmark, its all open source: https://github.com/wolfgarbe/LuceneBench

It seems you tried MMapDirectory and then commented it out. https://github.com/wolfgarbe/LuceneBench/blob/master/LuceneB...

So, you may be hitting SimpleFSDirectory instead, which does have issues with too many searches.

Could you share the reasons, MMapDirectory did not work for you?

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#72

Post by wolfgarbe »

localhost wrote: post #65

Have you tried against a corpus of text like email? How well does that work? I've been frustrated with the search speed (and results) from email search in general and was wondering if I could do better with an offline search?

That would definitely work. We are working on local client that could index your local PDF and Word documents. So importing emails would be a nice addition.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#73

Post by dheera »

NicoJuicy wrote: post #63
ehmmmmmmmm wrote: post #61

Earlier quoted context omitted.

It's probably not compliant. As with all good things, if you're in the Land of the Free (TM), eventually people will hunt you down if you try to do good things. Alternatively, base your site out of a country which gives you the freedom to do things like this. Build it in China and you'll be applauded instead of sued. Copyright suits for petty things like this would laughed off when it's something actually useful.

China's monopolies are far worse than in other countries. Founders there have no choice but to sell or get copied by the platform that you're building on. It's baba or tenscent.

Yes, that's also true, but on the flip side, technological advancement of society as a whole is not stifled by idiotic IP law.

I agree, some anti-monopoly regulation is in order in China, and I'm not a fan of the Alibaba/Tencent monopolies on the ecosystem. However, I think there are other ways to go about fostering and encouraging small businesses and protecting creators and their intentions than American-style IP law, which allows you to sit on an invention or work and hinder society from having access to the fruits of science, which I'm vehemently against. IP isn't even real property, IMHO.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#74

Post by wolfgarbe »

pelagicAustral wrote: post #27
syntaxing wrote: post #3

Neat but the back button hijacking is not cool.

I'm a bit curious about this 'back button hijacking', since the bug has now been fixed I'm afraid I'm not able to test. What was the unintended behaviour?

If you were entering deephn.org in your browser address bar, you couldn't use the browser back button to return to the previous page. That was a bug caused by the navigation code within our dynamic HTML page. It is fixed now.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#76

Post by goodmachine »

karterk wrote: post #49

Really cool demo. I'm curious to know what kind of hardware this is this hosted on. Full disclosure: I work on a similar fast, typo tolerant, fuzzy search engine search engine called Typesense ( https://github.com/typesense/typesense ).

Yeah, was also interested. It would be nice to see Typesense take up the same challenge for comparison ;)

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#77

Post by dang »

lotsofpulp wrote: post #59
dang wrote: post #56

Earlier quoted context omitted.

People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. But if you go back and look at that original 3-comment Bitcoin thread, what you find is that those 3 comments span an entire spectrum: elated, skeptical, and bewildered: https://news.ycombinator.com/item?id=599852 . The first cryptocurrency thread on HN—about 9 months earlier—was even more int…

> People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. I like to see wrong “obvious” predictions to humble myself, even if not from me.

Ok, you get a pass. I'm generalizing from myself of course.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#78

Post by superasn »

Very cool and useful. I was trying to find all comments on HN which contains the word youtube.com (basically trying to find some recently linked to videos from comments) but it returns 0 results when i set the search filter to comments(1). Any idea what I'm doing wrong?

https://deephn.org/?q=youtube.com&sort=time&in=Comments

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#79

Post by zoomablemind »

wolfgarbe wrote: post #58
zoomablemind wrote: post #52

Amazingly fast! I'm used to seeing a Firefox's load progress bar for most of requests on my mobile, yet here the results "just show up". More observations: - When trying to go back after clicking the original HN link, there's no response. - Clicking on what I understand are tags (hashtags) in a given post has no effect. - Sorting by date is working, though it'd be nice to see the date. Also a question: how do you gen…

>> When trying to go back after clicking the original HN link, there's no response. The original HN link is opened in a new browser page/tab. The search results should remain unchanged in the previous tab. So you don't use the back button, but the previous browser tab. >> Clicking on what I understand are tags (hashtags) in a given post has no effect. Search for "google", go to the first result. there is a hashtag "o…

> Please post an example where its doesn't work, so that we can fix it.

Note: I'm trying this in a mobile browser (Firefox).

- Search for a word 'Decentralized'

- pick the second result: 'A decentralized web would give power back to the people online'

- On the result's (preview?) page, click the tag 'Decentralization'

I get no response, nothing updates, no new tab opens. Not sure what is the intended action (I'd assume it should be a new set of results corresp. to the hashtag). Same issue with 'google' example that you described.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#80

Post by wolfgarbe »

arafalov wrote: post #71
wolfgarbe wrote: post #37

Earlier quoted context omitted.

There is no secret about the benchmark, its all open source: https://github.com/wolfgarbe/LuceneBench

It seems you tried MMapDirectory and then commented it out. https://github.com/wolfgarbe/LuceneBench/blob/master/LuceneB... So, you may be hitting SimpleFSDirectory instead, which does have issues with too many searches. Could you share the reasons, MMapDirectory did not work for you?

"NIOFSDirectory and MMapDirectory implementations face file-channel issues in Windows and memory release problems respectively. To overcome such environment peculiarities Lucene provides the FSDirectory.open() method. When invoked, it tries to choose the best implementation depending on the environment." https://www.baeldung.com/lucene-file-search
Post reply on HN