Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

deephn.org

Post reply on HN
Print view View original

101–110 of 118 posts

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#101

Post by myth_drannon »

dang wrote: post #56
myth_drannon wrote: post #9

Earlier quoted context omitted.

Earlier post 2009: https://news.ycombinator.com/item?id=599852 "Well this is an exceptionally cute idea, but there is absolutely no way that anyone is going to have any faith in this currency." Written by co-founder/CEO of Pachyderm and first employee of RethinkDB none the less.

People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. But if you go back and look at that original 3-comment Bitcoin thread, what you find is that those 3 comments span an entire spectrum: elated, skeptical, and bewildered: https://news.ycombinator.com/item?id=599852 . The first cryptocurrency thread on HN—about 9 months earlier—was even more int…

It's just humbling. Others and me, everyday we miss out hundreds of opportunities, it's a good reminder of life complexity. Things like that should be printed and posted on an office wall.

Also from a creator's perspective it's important to see these cases of once in a generation invention (good or bad) and the comments on it.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#102

Post by e12e »

TekMol wrote: post #100
abcc8 wrote: post #45

I'm curious how this site is compliant with copyright law. I've thought of working on similar projects, but the possibility of being sued has dissuaded me.

They copy the whole text of web pages that are linked in the HN posts. So totally a mass breach of copyright. If you are afraid of being sued, you certainly should not do something like this. On the upside, there is probably not much provable damage a publisher could ask to be repaired. So they probably are only liable to pay for the legal expenses of whoever sues them.

I copy the whole text of web pages I visit. It's not automatically breach of copyright.

That said, meaningful fulltext search will need data. But both Bing and Google crawl pages.

It's not even immediately clear that displaying a copy of a page is materially different from caching (which http do in many, many layers).

Now, if présent the text as your own - that might be a problem.

Ed: see also: archive.org.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#103

Post by dastx »

SamBam wrote: post #50
dastx wrote: post #4

Earlier quoted context omitted.

This kind of thing is enough for me to never want to use a site. If it's a bug, high recommend you fix it, if it's not, highly recommend you reconsider your position.

It is so common in SPA that the long-press of the back button to see the list of recent sites should be muscle-memory by now for everyone. Yes, it's an annoyance. Yes, sites should fix it. But not ever using a site because of it seems silly when it's literally a half-second longer click. (Also, for this site I don't actually see the bug. So either they fixed it very rapidly, or GP was just referring to individual sea…

> It is so common in SPA that the long-press of the back button to see the list of recent sites should be muscle-memory by now for everyone.

This is not a good excuse for laziness.

> But not ever using a site because of it seems silly when it's literally a half-second longer click.

I highly disagree. With this site at least it is actually possible to leave. Many websites I've come across with this issue, it is entirely impossible to leave without physically holding down the back button in the browser to get a list of history items, and then clicking a site from earlier.

> (Also, for this site I don't actually see the bug. So either they fixed it very rapidly, or GP was just referring to individual searchers being in the history, which is common for any search engine.)

Looks like they've fixed it.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#104

Post by slashdev »

Bender wrote: post #67

We're sorry but seekstorm-deephn doesn't work properly without JavaScript enabled. Please enable it to continue. Could the folks with JS disabled get a small semi-static page with the top interesting HN stats, and a message that say something like "Here are some simple stats. For the full interactive experience, please enable javascript"?

For the less than 1% of such people?

I doubt that it's worth it. If somebody can't be bothered to enable JavaScript to use the site, they're not a desirable customer anyway.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#105

Post by wolfgarbe »

TekMol wrote: post #100
abcc8 wrote: post #45

I'm curious how this site is compliant with copyright law. I've thought of working on similar projects, but the possibility of being sued has dissuaded me.

They copy the whole text of web pages that are linked in the HN posts. So totally a mass breach of copyright. If you are afraid of being sued, you certainly should not do something like this. On the upside, there is probably not much provable damage a publisher could ask to be repaired. So they probably are only liable to pay for the legal expenses of whoever sues them.

DeepHN does not copy the whole page, only the first 200 words or less. DeepHN does not copy the whole website but only a fraction of a single page of that web site. We always provide the original link. That should make it fair use. Of course we would prefer if there was a clear and concise legislation, valid in all countries, which could be followed.

We mainly surface historical content, which receives additional traffic from DeepHN, instead of taking value away from the website.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#106

Post by wolfgarbe »

freediver wrote: post #88
wolfgarbe wrote: post #69

Earlier quoted context omitted.

The crawler is a part of SeekStorm. "Well-defined": Just a guess: We are doing key text extraction, i.e. we try not to index boilerplate stuff an and menu items. As "well-defined" is within a short list item, it might be accidentally skipped. So, that is not yet perfect, and considering the diversity in web page structure it probably never will. But we will try to improve.

Yes, if the preview is the representation of what you have indexed, then half of that article is missing. You may have identified the weakest link in your stack - crawler/extractor (which is notoriously hard to do, would be good if you provided more detail eg. do you use headless browser or simple GET request, do you crawl PDFs etc). Little use of the advanced stack on top of it, if the data does not end in the index…

No, the preview is NOT the representation of what we have indexed. The preview ist limited to about 200 words to be compliant with fair use legislation. Indexing is limited to 1 MB per document.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#107

Post by stjohnswarts »

dheera wrote: post #73
NicoJuicy wrote: post #63

Earlier quoted context omitted.

China's monopolies are far worse than in other countries. Founders there have no choice but to sell or get copied by the platform that you're building on. It's baba or tenscent.

Yes, that's also true, but on the flip side, technological advancement of society as a whole is not stifled by idiotic IP law. I agree, some anti-monopoly regulation is in order in China, and I'm not a fan of the Alibaba/Tencent monopolies on the ecosystem. However, I think there are other ways to go about fostering and encouraging small businesses and protecting creators and their intentions than American-style IP l…

Yes it is, if 90% of the reason for new tech is to get richer, if all protections like copyright, patents, etc are dropped then people will be much less incentivized to invest and stick their necks out for new creations. I wholeheartedly do not agree with your IMHO. That said we could absolutely improve our current patent system and copyright system (especially copyright and those parts which contribute to patent trolls)

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#108

Post by derangedHorse »

dang wrote: post #56
myth_drannon wrote: post #9

Earlier quoted context omitted.

Earlier post 2009: https://news.ycombinator.com/item?id=599852 "Well this is an exceptionally cute idea, but there is absolutely no way that anyone is going to have any faith in this currency." Written by co-founder/CEO of Pachyderm and first employee of RethinkDB none the less.

People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. But if you go back and look at that original 3-comment Bitcoin thread, what you find is that those 3 comments span an entire spectrum: elated, skeptical, and bewildered: https://news.ycombinator.com/item?id=599852 . The first cryptocurrency thread on HN—about 9 months earlier—was even more int…

Not really a superiority thing for me. I just think it’s funny how much perceptions change over time and how reasonable assumptions can be proven wrong

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#109

Post by luckman212 »

clukic wrote: post #62
exdsq wrote: post #7

It’s fun to see Bitcoin posts from 2008-2010. I wonder how many bought and are now retired because of them. Seeing recommendations to use MtGox is pretty funny too https://news.ycombinator.com/item?id=1998144

I read about Bitcoin here over a decade ago. Took the time to setup a miner, and let it run for a week. I think I generated a few hundred coins worth less than a dollar total. Figured the electricity cost way more than that and deleted it all.

I admire your perseverance to continue forging ahead despite deleting $20MM worth of coins.

Re: Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

#110

Post by TigeriusKirk »

exdsq wrote: post #7

It’s fun to see Bitcoin posts from 2008-2010. I wonder how many bought and are now retired because of them. Seeing recommendations to use MtGox is pretty funny too https://news.ycombinator.com/item?id=1998144

The crazy thing is that MtGox really was your best choice back then.

A long way from that mess of a site to Coinbase IPOing tomorrow for a zillion dollars.

Post reply on HN