Scrape like the big boys • Hacker News Board

Re: Scrape like the big boys

#11

Post by throwaway2993 » Fri, Nov 05, 2021, 1:11 PM UTC

jtdev wrote: post #3

ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?

I wrote a scraper a couple of years ago to get a single data point from a website where my client was already a paying customer. This website had an API, which they were also paying for, but the API didn't cover that data point, so at the time they had one of their admin people populating that missing piece of data manually, which was taking them around ten minutes a day.

I asked them if my customer could pay to access this data point via their API and they quoted 3600 EUR/month! Enter the scraper...

Re: Scrape like the big boys

#12

Post by bryanrasmussen » Fri, Nov 05, 2021, 1:13 PM UTC

1cvmask wrote: post #7

ebbp wrote: post #5

Earlier quoted context omitted.

We do offer an API - the scrapers are trying to circumvent using that, presumably.

What is your site may I ask? Just curious about the difference in value from using your API and web scraping as there is a cost to web scraping as well.

If you make your scraper well, and it counterfeits being a real user believably, you end up with a solution that can be tweaked as needed to handle whatever traps people put in to try to defeat your scrapers.

If you make your api client well, you don't have the problems of a scraper - but if the api owner decides to change rules for api and you can't do what your business is based on being able to do (think of api owner as Twitter) then you need to make a scraper.

Re: Scrape like the big boys

#13

Post by halfmatthalfcat » Fri, Nov 05, 2021, 1:18 PM UTC

ebbp wrote: post #5

jtdev wrote: post #3

Earlier quoted context omitted.

Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?

We do offer an API - the scrapers are trying to circumvent using that, presumably.

Maybe the API terms/cost are prohibitive? I'm sure there's some equilibrium where they would rather pay you than go through the trouble of scraping.

Re: Scrape like the big boys

#14

Post by joekrill » Fri, Nov 05, 2021, 1:18 PM UTC

A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.

Re: Scrape like the big boys

#15

Post by tptacek » Fri, Nov 05, 2021, 1:20 PM UTC

joekrill wrote: post #14

A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.

Yes; a SERP is a Google search result page. It's the most important acronym in SEO.

Re: Scrape like the big boys

#16

Post by daveguy » Fri, Nov 05, 2021, 1:20 PM UTC

joekrill wrote: post #14

A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.

I had to look it up.

SERP: Search Engine Results Page

Re: Scrape like the big boys

#17

Post by taytus » Fri, Nov 05, 2021, 1:32 PM UTC

ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

>where some amateur did real damage to us

If an amateur can do damage to you, then I have some bad news for you...

Re: Scrape like the big boys

#18

Post by aninteger » Fri, Nov 05, 2021, 1:33 PM UTC

ebbp wrote: post #5

jtdev wrote: post #3

Earlier quoted context omitted.

Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?

We do offer an API - the scrapers are trying to circumvent using that, presumably.

Wait, why wouldn't you have rate limiting on your API? Providers like Cloudflare offer this although I guess you could roll your own too since our industry loves to reinvent the wheel.

Re: Scrape like the big boys

#19

Post by purerandomness » Fri, Nov 05, 2021, 1:38 PM UTC

ebbp wrote: post #5

jtdev wrote: post #3

Earlier quoted context omitted.

Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?

We do offer an API - the scrapers are trying to circumvent using that, presumably.

Why do you think are they trying to circumvent it?

Does your API provide all the information that can be found on the site, or are they scraping because the API is incomplete?

We've once had to scrape Amazon product pages because they have a lot of API endpoints, but those didn't contain the data we needed.

Re: Scrape like the big boys

#20

Post by bgroat » Fri, Nov 05, 2021, 1:42 PM UTC

joekrill wrote: post #14

A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.

Not the OP, but I thought it was well known.

That said, I do a lot of SEO work.

Still, it should be best practice to define any acronym or initialism the first time you use it