Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matt Cutts is looking for scraper sites

twitter.com

Post reply on HN
Print view View original

41–50 of 120 posts

Re: Matt Cutts is looking for scraper sites

#41

Post by dangrossman »

danielbarla wrote: post #30
VikingCoder wrote: post #11

Scrapers lift the full content, wholesale, without attribution. You may as will just show http://images.google.com and complain that it's scraping. Or http://news.google.com . In general, do you think Wikipedia gets more traffic because Google exists, or do you think Google gets more traffic because Wikipedia exists? Meaning, which affect is larger? I'm pretty sure the answer to this is obvious. And if more scrapers…

That's one way of looking at it, on the other hand, they link to the original URL, passing traffic back to the original source. Most "scraper" sites take the content, wrap it in their own similar outer layer, and try to take ad revenue. E.g. I've seen my own StackOverflow answers copied, word for word, to a scraper site and presented under a made-up name.

StackOverflow actually allows this; all their data is Creative Commons licensed, and they publish the full database dump on the Internet Archive.

https://archive.org/details/stackexchange

Re: Matt Cutts is looking for scraper sites

#42

Post by dangrossman »

jjoonathan wrote: post #21

Bah, what would I possibly need with a scraped definition that 1) Hasn't been chunked into 20 pieces of varying grammatical structure which are automatically matched to corresponding questions 2) Hasn't been subsequently pasted over a slideshow of completely irrelevant stock photos in bold, white font 3) Isn't accompanied by a grid of ~30 vaguely related questions helpfully linked to similar pages and tastefully deco…

The backlinks and SEO were gained before they did that to the site. Turning every Q&A they used to host as simple text into a terrible 30-click slideshow was a pretty recent change.

Re: Matt Cutts is looking for scraper sites

#43

Post by jbinto »

dangrossman wrote: post #41
danielbarla wrote: post #30

Earlier quoted context omitted.

That's one way of looking at it, on the other hand, they link to the original URL, passing traffic back to the original source. Most "scraper" sites take the content, wrap it in their own similar outer layer, and try to take ad revenue. E.g. I've seen my own StackOverflow answers copied, word for word, to a scraper site and presented under a made-up name.

StackOverflow actually allows this; all their data is Creative Commons licensed, and they publish the full database dump on the Internet Archive. https://archive.org/details/stackexchange

Do the terms of the license allow for this kind of abuse?

Just because something is CC doesn't mean you can do whatever you want with it.

Re: Matt Cutts is looking for scraper sites

#44

Post by dangrossman »

ricg wrote: post #9

Easy. Search for a programming related question. After the result from stackoverflow you'll find dozens of scraper sites.

Those sites didn't actually scrape anything. StackExchange publishes their full database dump, and it's CC licensed.

https://archive.org/details/stackexchange

Re: Matt Cutts is looking for scraper sites

#45

Post by pjc50 »

solve wrote: post #31
pud wrote: post #27

Wikipedia's database is public and used by Google with permission. You can probably use it for your projects, too. So this is neither scraping, nor against the rules. Here are dumps in SQL and XML format: http://dumps.wikimedia.org/enwiki/ Ps- Yes the original post was meant to be funny and it was; I do have a sense of humor. :)

He's talking about outranking the true original source of the content in search results. You most certainly cannot create your own site that consists only of excerpts from Wikipedia, if you wish to remain on Google's search results. Copyrights are irrelevant to this. What's bad though, is that Google isn't just lowering the rankings of non-original content pages now (including any kind of legitimate curation sites.)…

Could you show an example of such a legitimate new curation site please?

Re: Matt Cutts is looking for scraper sites

#46

Post by solve »

troels wrote: post #35
solve wrote: post #19

Earlier quoted context omitted.

Google recently flagged my content-curation startup as "Pure Spam", even though it only takes small snippets from the original sources, is 100% human curated, and always links back to the true original source. Not only are the curated pages blocked, but the entire domain is blocked as "pure spam". People who use Google to find a domain instead of typing the full URL now can't find it anywhere. These assholes are just…

Curious - Which site is that?

You won't find it on Google :) We're making a products recommendation site, focusing on goal-oriented decisions. We'll make the full announcement within the next couple of weeks.

This reminds me. Google is really killing the "release early and release often" approach, if people will now have to do a ton of SEO learning and tweaking to avoid having your MVP permanently banned at launch day.

Re: Matt Cutts is looking for scraper sites

#48

Post by smoyer »

tobehonest wrote: post #33
danielbarla wrote: post #30

Earlier quoted context omitted.

That's one way of looking at it, on the other hand, they link to the original URL, passing traffic back to the original source. Most "scraper" sites take the content, wrap it in their own similar outer layer, and try to take ad revenue. E.g. I've seen my own StackOverflow answers copied, word for word, to a scraper site and presented under a made-up name.

By having a tl;dr about the actual Wikipedia page, there is no need for the user to click on the link. Following what you're saying, Google as wrapped it in their own layer, and trying to take ad revenue.

Actually, I find that having a tl;dr will rarely answer the question(s) I have on a topic, but it will commonly show me whether I've found the right wikipedia page. I usually either click-through or refine my search.

Re: Matt Cutts is looking for scraper sites

#49

Post by josefresco »

7952 wrote: post #29
smoyer wrote: post #15

There are lots of places where Google decides to "help" me, but sometimes I just want search results. Other times, I actually like getting the curated content (e.g. search for "delta 3810"). Is there a way to disable this? EDIT: I should also note that I'm one of those who switched over to DuckDuckGo for privacy reasons, so I don't see these results as often now.

The content they do provide is often so bad it is almost embarrassing. Search for "Russia" and you get a completely useless map and a list of random facts. It may be useful to a child researching geography but for me it is just annoying. I want content that is curated by people who actually understand the subject. I would pay for a search engine designed by someone who understands my industry. The Google algorithm on…

I'm curious, what should they show you if you type on "Russia" the term by itself seems pretty generic and open ended.

"I would pay for a search engine designed by someone who understands my industry"

What industry is that? And how would Google guess or know your industry unless you tell them?

Re: Matt Cutts is looking for scraper sites

#50

Post by dangrossman »

jbinto wrote: post #43
dangrossman wrote: post #41

Earlier quoted context omitted.

StackOverflow actually allows this; all their data is Creative Commons licensed, and they publish the full database dump on the Internet Archive. https://archive.org/details/stackexchange

Do the terms of the license allow for this kind of abuse? Just because something is CC doesn't mean you can do whatever you want with it.

Yes, they do; it's not abuse when you're given explicit permission. CC BY-SA means you can do whatever you want with it as long as you attribute the source as specified.
Post reply on HN