Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrape like the big boys

incolumitas.com

Post reply on HN
Print view View original

191–196 of 196 posts

Re: Scrape like the big boys

#191

Post by runnerup »

fergie wrote: post #21
joekrill wrote: post #14

A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.

Unintroduced acronyms should always be avoided.

Perhaps a stroll through your own comment history (or the comments of any other HN (hacker news) user) would illuminate a lot of places where acronyms are used without introductions. TBH (to be honest) though, I'm not sure if every one of those should always have one or sometimes not.

Re: Scrape like the big boys

#192

Post by KuhlMensch »

Doing a bit of low-stakes monitoring of webpages lately. It started (as I'm assuming it often does) with right-clicking a network request in Chrome and selecting "copy as curl"

Then graduated to JavaScript for surrounding logic e.g. data transformation

I had assumed I'd quickly give up and move to a headless browser, BUT I can't bring myself to move away from tiny CPU utilization of curl.

Throwing together a "plugin" probably takes me less than 20 minutes normally.

I'll probably have a look at using prowl to ping my phone.

And if I get more serious I'll look at auto authenticate options on npm. But I'm not sure if the overhead of maintaining a bunch of spoofy requests will be worth it.

Re: Scrape like the big boys

#193

Post by account42 »

simondotau wrote: post #131
matheusmoreira wrote: post #128

Earlier quoted context omitted.

Do I need to explain that copyright is practically unenforceable in the 21st century? Data is trivially copied and there's nothing you can do to fight that, no amount of laws will ever make it non-trivial again. Even if you successfully sue somebody for this, it won't stop them. At some point people are gonna have to accept this.

I was responding to a question of whether it was theft, not whether such theft is morally grey or unstoppable in practice. Yours is a somewhat orthogonal point and one I don’t entirely disagree with.

Well if you want to stick to the hard facts then it's even simpler: copyright infringement is not theft - those things are covered by entirely separate laws.

Re: Scrape like the big boys

#194

Post by achillesheels »

Grimm1 wrote: post #184
achillesheels wrote: post #183

Earlier quoted context omitted.

Well actually, isn’t Google improving the value of the content ergo property itself by improving its accessibility? I was inferring a one-way street with the accumulated data that can lead to server crashes - which I don’t believe Google’s web crawl does at all (in fact that would be counter-productive).

I'd say most crawlers are looking to provide enriched value for content at their end use. Google is just an aggregator (the biggest by far) but other aggregators are looking to provide similar value.

So then an aggregator is different than a scraping service with respect to the value given to the rest of humanity? In that, in principle one adds value to the content creation and the other deducts through potential harmful interference with its reciprocity?

Re: Scrape like the big boys

#195

Post by Grimm1 »

achillesheels wrote: post #194
Grimm1 wrote: post #184

Earlier quoted context omitted.

I'd say most crawlers are looking to provide enriched value for content at their end use. Google is just an aggregator (the biggest by far) but other aggregators are looking to provide similar value.

So then an aggregator is different than a scraping service with respect to the value given to the rest of humanity? In that, in principle one adds value to the content creation and the other deducts through potential harmful interference with its reciprocity?

I'd say many aggregators do offer value to the rest of humanity but I imagine there are probably some exceptions and also not all scraper services offer no value it's just different value to different people.

Some scraping services make their money by offering scraping services to companies for specific information and you could argue they provide value to other businesses that way, but not to the broader "rest of humanity".

So I'm not sure it's as simple as just "aggregator" good "scraping service" bad as value provided takes on many different forms, and that's what makes this difficult.

I guess it may come down to your take on what you think of middlemen, because they are all effectively middlemen in the data economy.

Edit: I was rereading your comment, in respect directly to the value added to the content, then yes maybe it is more clear that aggregators are in principle different because they do add that value where scraping services that sell the data do not offer any enrichment to the content creator. I personally think protecting content aggregators that republish the data to create visibility or other value for the content creator to the extent that they're not worried about being sued for that is probably a worthwhile thing to happen because of the net benefit to our ability to find information/content.

Re: Scrape like the big boys

#196

Post by wilg »

kall wrote: post #162
wilg wrote: post #90

Not the same kind of scraping, but does anyone have thoughts/resources/best practices for doing link previews (like Twitter/iMessage/Facebook)?

You shouldn‘t really need to do any scraping tricks to get that, because it‘s data the websites (usually) want to give to bots. Or are people getting bot block screens from Cloudflare et all for that basic action these days? It should be a matter of a simple GET request to fetch plain html and parse the OpenGraph meta tags out if that. There are many open source libraries to do that for you depending on your language…

Bot blocks are definitely an issue for certain sites, I've implemented it that way currently.

Microlink is a good tip, thanks!

Post reply on HN