A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.
Unintroduced acronyms should always be avoided.
Scrape like the big boys
21–30 of 196 posts
Re: Scrape like the big boys
#22Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
Bots are one of those things that are easy to build and hard to get right, and there's really no way of preparing for the chaotic reality of real web pages other than fixing the problems as they show up. Weird and unexpected interactions are going to happen. Crawling the real web involves navigating a fractal of unexpected, undocumented and non-standard corner cases. Nobody gets that right on the first try. Because of that I do think we need to be a bit patient with bots.
At the same time, even as someone who runs a web crawler, I have zero qualms about blocking misbehaving bots.
Re: Scrape like the big boys
#23Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
[deleted]
Re: Scrape like the big boys
#24A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.
The word SERP feels like a bit of a shibboleth for SEO-people. They seem to take it for granted, the rest of the world just looks puzzled when they hear it.
Re: Scrape like the big boys
#25Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
>where some amateur did real damage to us If an amateur can do damage to you, then I have some bad news for you...
If an amateur can do damage to you, then I have some bad news for you...
I believe the point wasn't surprise that damage occurred at all, but frustration that damage can occur just out laziness/ignorance rather than malice.
Re: Scrape like the big boys
#26Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
>where some amateur did real damage to us If an amateur can do damage to you, then I have some bad news for you...
This is nonsense. It's always easier to destroy than to build/mantain. If you got any real advice, by all means...
Re: Scrape like the big boys
#27Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
Bots are one of those things that are easy to build and hard to get right, and there's really no way of preparing for the chaotic reality of real web pages other than fixing the problems as they show up. Weird and unexpected interactions are going to happen. Crawling the real web involves navigating a fractal of unexpected, undocumented and non-standard corner cases. Nobody gets that right on the first try. Because o…
I kinda feel like rate limiting your request to individual domains and IP addresses is an easy thing that goes a long way towards getting it right.
Re: Scrape like the big boys
#28A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.
Yes; a SERP is a Google search result page. It's the most important acronym in SEO.
I don’t remember never ever hearing it and I’ve been in the industry for some time
Re: Scrape like the big boys
#29Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
Why not create api endpoint and charge mild cost for that data? You’ll make money instead of spending it.
Re: Scrape like the big boys
#30A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.
Unintroduced acronyms should always be avoided.
Depends on the audience-acronym pair. I don't think HTTP needs an introduction in a technical article, OTOH (on the other hand ;) ) a general newspaper should probably expand HTTP but not WWW.