Scrape like the big boys

Post by kall » Sat, Nov 06, 2021, 11:42 AM UTC

gonzo41 wrote: post #120

rootusrootus wrote: post #65

Earlier quoted context omitted.

I know a guy at Nike that had to deal with a similar problem. As I recall, they basically gave in -- instead of trying to fight the scrapers, they built them an API so they'd quit trashing the performance of the retail site with all the scraping.

I think there's an opportunity for a new JS framework to have something like randomly generated dom that will always display the page and elements the same to a human but constantly break paths for computers. Like displaying a table with semantic elements, then divs, then using an iframe with css grid and floating values over the top. This almost seems like a problem for AI to solve.

Post by pnt12 » Sat, Nov 06, 2021, 12:38 PM UTC

RandomThrow321 wrote: post #40

palijer wrote: post #10

Earlier quoted context omitted.

That was not the authors main argument against proxies, that was just an additional point. You ignored the primary argument in your judgment. >>Because I could not fully trust the other customers with whom I shared the proxy bandwidth. What if I share proxy servers with criminals that do more malicious stuff than the somewhat innocent SERP scraping?

Can they not call out a secondary point?

Post by Lifelarper » Sat, Nov 06, 2021, 2:59 PM UTC

blueboo wrote: post #130

gonzo41 wrote: post #120

Earlier quoted context omitted.

I think there's an opportunity for a new JS framework to have something like randomly generated dom that will always display the page and elements the same to a human but constantly break paths for computers. Like displaying a table with semantic elements, then divs, then using an iframe with css grid and floating values over the top. This almost seems like a problem for AI to solve.

Even if your DOM is obfuscated, the rendered page remains vulnerable to OCR. Obfuscate the rendered pixels and you’ll annoy your humans and eventually find that the scrapers’ OCR is superhuman. Still, maybe AI comes into it. Maybe poisoning the data is the right way to do it conditioned on ML-juiced anomaly detection.

Post by un-devmox » Sat, Nov 06, 2021, 3:15 PM UTC

scarygliders wrote: post #31

ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Right with you there. I had a particularly bad time not so long ago, when a customer's site - a shop - was brought to its knees because someone, probably a competitor, hired some scraper-company of some sort to scrape every product and price. The scraper would systematically go through every single product page. And by scraper, I mean - 100's of them. All at the same time, using the old trick of 1 scraper requesting…

Post by echelon » Sat, Nov 06, 2021, 4:32 PM UTC

matheusmoreira wrote: post #128

simondotau wrote: post #125

Earlier quoted context omitted.

> You're going to need to explain how scraping publicly available information on a website is theft. Seriously? Do I need to explain why a song doesn’t enter the public domain when it is played on the radio?

Do I need to explain that copyright is practically unenforceable in the 21st century? Data is trivially copied and there's nothing you can do to fight that, no amount of laws will ever make it non-trivial again. Even if you successfully sue somebody for this, it won't stop them. At some point people are gonna have to accept this.