Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrape like the big boys

incolumitas.com

Post reply on HN
Print view View original

141–150 of 196 posts

Re: Scrape like the big boys

#141

Post by ebbp »

kulikalov wrote: post #29
ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Why not create api endpoint and charge mild cost for that data? You’ll make money instead of spending it.

My point was more that we can accept with, and live with, scrapers but expect some minimal level of consideration if you’re going to abuse our very expensively gathered dataset. Sending us 10x daily traffic so you can scrape quicker than the fair usage policy of our API allows is just… poor etiquette? Unkind? Not really sure how to phrase it. I’m exhausted after multiple 18 hours days trying to keep our website online for the public.

Re: Scrape like the big boys

#142

Post by ushtaritk421 »

hall0ween wrote: post #103

Basic question, how does one profit from scraping data and what kinda data? Taking a stab at answering it: you scrape the data and build a business around selling it. Stock prices? But that's boring, plus how many others are doing it? I bet a lot.

Anything you might look up or keep track of online that helps your business is probably being scraped by someone who is using it themselves or selling access to the curated data set.

Prices (are yours high, low compared to competition?), reviews, locations of physical stores, search result placement (where does your widget show up when someone searches "widget" on your site?), just to name a few use cases.

Re: Scrape like the big boys

#143

Post by throwaway984393 »

If you want to avoid bot detection, learn how bot detection work. A lot of commercial "webapp firewalls" and the like actually have minimum requirements before they flag certain traffic as a botnet; stay below those limits and you can keep hammering away. Sometimes those limits are quite high.

In the past we've had the most success defeating bots by just finding stupid tricks to use against them. Identify the traffic, identify anything that is correlated with the botnet traffic, and throw a monkey wrench at it. They're only using one User Agent? Return fake results. 90% of the botnet traffic is coming from one network source (country/region/etc)? Cause "random" network delays and timeouts. They still won't quit? During attacks, redirect to captchas for specific pages. During active attacks this is enough to take them out for days to weeks while they figure it out and work around it.

Re: Scrape like the big boys

#144

Post by ebbp »

taytus wrote: post #17
ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

>where some amateur did real damage to us If an amateur can do damage to you, then I have some bad news for you...

To be clear, they did “damage” was to our bottom line. Most sites don’t capacity plan for random cliff walls of 2-10x traffic (clearly we should!). We’re scalable enough to handle that traffic after a period, but a) it caused intermittent periods of low availability (costing us money because we didn’t generate income the way we normally do) and b) cost us money from scaling all our services up.

It’s just selfish. If you’re going to take the product of other people’s work in a manner they don’t consent to, at least do it in a way that doesn’t cost them twice over.

Re: Scrape like the big boys

#145

Post by serf »

scarygliders wrote: post #31
ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Right with you there. I had a particularly bad time not so long ago, when a customer's site - a shop - was brought to its knees because someone, probably a competitor, hired some scraper-company of some sort to scrape every product and price. The scraper would systematically go through every single product page. And by scraper, I mean - 100's of them. All at the same time, using the old trick of 1 scraper requesting…

>Seriously, I hate scrapers. I hate the people who make scrapers. I hate their lack of ethics. Fuck those guys.

if a scraper is effectively DDoSing you, call it what it is -- a denial of service attack.

i've found from experience that most scraping attempts originate against host-sites that are generally user-hostile; no APIs to use, JS tricks to bother user browsing, or groups that profit from first-mover advantage and thus try to obscure data.

So, if your sites are commonly the victim of scrapers that are harvesting publicly available data i've found that it's more useful to ask myself what alternatives I could provide those that feel the need to scrape.

As for a 'lack of ethics' on how publicly available data is wrangled -- well, i'll just say that I feel that it remains the responsibility of the administrator rather than being something to push the blame onto clients for. There are plenty of technical avenues to pursue before appealing to morals and ethics for help.

Re: Scrape like the big boys

#146

Post by ghusbands »

gonzo41 wrote: post #120
rootusrootus wrote: post #65

Earlier quoted context omitted.

I know a guy at Nike that had to deal with a similar problem. As I recall, they basically gave in -- instead of trying to fight the scrapers, they built them an API so they'd quit trashing the performance of the retail site with all the scraping.

I think there's an opportunity for a new JS framework to have something like randomly generated dom that will always display the page and elements the same to a human but constantly break paths for computers. Like displaying a table with semantic elements, then divs, then using an iframe with css grid and floating values over the top. This almost seems like a problem for AI to solve.

This would have huge accessibility issues, breaking screen readers and the like.

Re: Scrape like the big boys

#147

Post by simondotau »

nickff wrote: post #133
matheusmoreira wrote: post #128

Earlier quoted context omitted.

Do I need to explain that copyright is practically unenforceable in the 21st century? Data is trivially copied and there's nothing you can do to fight that, no amount of laws will ever make it non-trivial again. Even if you successfully sue somebody for this, it won't stop them. At some point people are gonna have to accept this.

> "Do I need to explain that copyright is practically unenforceable in the 21st century?" This sentence added nothing substantive to your comment, and made it rude; could you please be a bit more polite in the future? https://news.ycombinator.com/newsguidelines.html

For what it's worth, I didn't think it was rude. And I do think it was a substantive aspect of his response. It's a valid perspective even if I personally think it's somewhat orthogonal and incomplete.

Re: Scrape like the big boys

#148

Post by simondotau »

throwawaygh wrote: post #139
simondotau wrote: post #125

Earlier quoted context omitted.

> You're going to need to explain how scraping publicly available information on a website is theft. Seriously? Do I need to explain why a song doesn’t enter the public domain when it is played on the radio?

OP was talking about price lists. IANAL but AFAIK you can't copyright a list of prices.

That may be true. A list of prices might not be copyrightable in your particular jurisdiction. However I was only responding to the raw assertion made without any such qualification.

On the opposite end of the spectrum might be a photographer's website containing a gallery of their sample work. The fact that the gallery is openly published doesn't represent a relinquishing of copyright over those images.

Re: Scrape like the big boys

#149

Post by rootusrootus »

gonzo41 wrote: post #120
rootusrootus wrote: post #65

Earlier quoted context omitted.

I know a guy at Nike that had to deal with a similar problem. As I recall, they basically gave in -- instead of trying to fight the scrapers, they built them an API so they'd quit trashing the performance of the retail site with all the scraping.

I think there's an opportunity for a new JS framework to have something like randomly generated dom that will always display the page and elements the same to a human but constantly break paths for computers. Like displaying a table with semantic elements, then divs, then using an iframe with css grid and floating values over the top. This almost seems like a problem for AI to solve.

To some extent those already exist and I get annoyed by them when they cause 1Password to be useless on their login page. But it probably would help with algorithmic scraping.

Re: Scrape like the big boys

#150

Post by TheCoelacanth »

ushtaritk421 wrote: post #134
zivkovicp wrote: post #49

Earlier quoted context omitted.

I agree 100%, but it is a fact of life, and sometimes it's better to just minimize the fuzz and focus on the things that matter. Your argument is perfectly valid and applies to offline activities as well (what stops a competitor from walking through the aisles of a Walmart or Costco?), but this is a battle that can't be won, there are too many parasitic actors. It is human nature.

Understanding your competitor's pricing is not "parasitic", it's research. Every company I've ever worked for that sells something online scrapes their competitors in some way (whether with bots or with interns).

I would say it's the opposite of parasitic. It's essential to having a well-functioning free market.
Post reply on HN