Ah yes, Challenge 4 (anti-bot measures). At Blekko I developed a number of ways to deal with people that tried to scrape the web site for web results. The three most effective ways are blackholing (your web site vanishes as far as these folks are concerned), hang holding (basically using a crafted TCP/IP stack that does the syn/ack sequence but then never sends data so the client hangs forever), and data poisoning (r…
Lessons learned scraping 100B product pages
71–80 of 99 posts
Re: Lessons learned scraping 100B product pages
#72Earlier quoted context omitted.
I'm not sure you understand how the GIL works in Python. If you're using multiprocessing, there's no locking across the code executing on each core. Also, if you're writing to the same file from four processes, you're going to need locking.
What I have last known is that GIL languages work well in multicore scenarios as long as all N tasks have I/O calls that serve as yielding points for the interpreter, and they do not use preemptive scheduling like the BEAM VM (Erlang, Elixir, LFE, Alpaca) do. Am I mistaken?
If you were to use multithreading instead, you would generally have a problem if you were doing non-I/O work.
Re: Lessons learned scraping 100B product pages
#73Earlier quoted context omitted.
What I have last known is that GIL languages work well in multicore scenarios as long as all N tasks have I/O calls that serve as yielding points for the interpreter, and they do not use preemptive scheduling like the BEAM VM (Erlang, Elixir, LFE, Alpaca) do. Am I mistaken?
As far as Python goes, yes. Multicore implies multiple processes, which means that each process will have it's own Python interpreter, each with it's own GIL. If you were to use multithreading instead, you would generally have a problem if you were doing non-I/O work.
It seems that now we are both on the same page. Single process & many threads are problematic for GIL languages and that's why I gave up using Ruby for scrapers. GIL languages can work very well for the URL downloading part though.
Re: Lessons learned scraping 100B product pages
#74> However, our recommendation is to go with a proxy provider who can provide a single endpoint for proxy configuration and hide all the complexities of managing your proxies. Can you provide an example of such service? THanks!
Re: Lessons learned scraping 100B product pages
#75Ah yes, Challenge 4 (anti-bot measures). At Blekko I developed a number of ways to deal with people that tried to scrape the web site for web results. The three most effective ways are blackholing (your web site vanishes as far as these folks are concerned), hang holding (basically using a crafted TCP/IP stack that does the syn/ack sequence but then never sends data so the client hangs forever), and data poisoning (r…
I thought the number one anti-bot measure was a cease and desist letter :) seriously though, some of these websites clearly don't want to be scraped, what's stopping them from sending scrapehub a C&D letter and forcing them to comply?
One of the things we learned at Blekko was that people that run botnets often sell 'proxy service' as a thing, we identified several made out users of the Time Warner "road runner" service. That put us as a web site in a bind in that the proxy service that was running on an infected computer was violating our terms of service but the user might be completely unaware. If they were also a customer and we black holed their IP it would also cut off legitimate traffic. Since we didn't keep a logs that could identify these relations over time (privacy issues) we had to rely on other methods. We never got enough penetration into the search market to make this a huge concern however so the problem remained largely theoretical. We started a program of doing exponential banning where an IP would be banned and then an hour later unbanned, and if it resumed its bad behavior banned for 2 hours then 4 Etc. Once you get to 1024 hours it is pretty safe to assume they are lawfully evil as it were.
These guys fake their user agent, mask their IP addresses, and generally work hard to defeat anti-bot measures. They know they are over the line, but the law has yet to catch up to them.
Re: Lessons learned scraping 100B product pages
#76Ah yes, Challenge 4 (anti-bot measures). At Blekko I developed a number of ways to deal with people that tried to scrape the web site for web results. The three most effective ways are blackholing (your web site vanishes as far as these folks are concerned), hang holding (basically using a crafted TCP/IP stack that does the syn/ack sequence but then never sends data so the client hangs forever), and data poisoning (r…
I thought the number one anti-bot measure was a cease and desist letter :) seriously though, some of these websites clearly don't want to be scraped, what's stopping them from sending scrapehub a C&D letter and forcing them to comply?
Re: Lessons learned scraping 100B product pages
#77As a side note, I have had quite a bit of experience trying to block automated scraping services. And I found that the best way is to quietly attempt to detect scraping. Then, serve up tainted data. In our case, competitors were scraping pricing data in order to competitively price their products without having to do the work. So we just randomly start to give them incorrect prices on every few products. Not only wou…
What's ironic is that most of the sites with anti scraping protection also do scraping of their own. E.g. Amazon and Walmart both do a lot of their own scraping.
Re: Lessons learned scraping 100B product pages
#78Earlier quoted context omitted.
What's ironic is that most of the sites with anti scraping protection also do scraping of their own. E.g. Amazon and Walmart both do a lot of their own scraping.
Really going to call for a [citation needed] on that "most"!
Every major ecommerce site scrapes, it would be a competitive disadvantage if they didn’t.
Re: Lessons learned scraping 100B product pages
#79Re: Lessons learned scraping 100B product pages
#80Earlier quoted context omitted.
I thought the number one anti-bot measure was a cease and desist letter :) seriously though, some of these websites clearly don't want to be scraped, what's stopping them from sending scrapehub a C&D letter and forcing them to comply?
Sure, if you can make a reasonable assumption it is them scraping you. As they point out in the article they invest in proxy networks to make their requests appear to come from a bunch of addresses that don't lead back to them. One of the things we learned at Blekko was that people that run botnets often sell 'proxy service' as a thing, we identified several made out users of the Time Warner "road runner" service. Th…
I'm thinking of RyanAir suing Expedia, United vs wandr.me, Southwest suing SWMonkey.com, I'm sure there's countless others.