91–99 of 99 posts
Earlier quoted context omitted.
My experience, when it comes to scraping airline websites, the airline's legal department usually doesn't wait to have proof that you were the one that actually scraped them. If you have their data on your website, they send you a C&D, and if they continue to find their data on your website, they will happily sue you. In other words, doesn't matter how you got the data, you must've broken the law if you got their dat…
But if you have Amazon turk workers 'scraping' the data is that illegal?
> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…
That sounds like a very creative idea. What I do is call a regression test every x minutes. If it fails, set a flag to save/store the html everytime we crawl pages. Now we can go back and process these saved pages when we fix our crawler
I do the same at $dayJob where I'm parsing results of an internal API. Instead of making a call later that may not have the same data, I store the json and just process that. I feel like treating network requests as an expensive operation, even though they're not really, helped me come up with some clever ideas I've never had before. It's a premature optimization considering I've had like 0.000001% of failure but being able to replay that one breakage made debugging an esoteric problem waaaaaay simpler than it would've been otherwise.
Earlier quoted context omitted.
That sounds like a very creative idea. What I do is call a regression test every x minutes. If it fails, set a flag to save/store the html everytime we crawl pages. Now we can go back and process these saved pages when we fix our crawler
I crawl a specific site somewhere up to 50 unique URLs a day. I store both the unparsed full html as a file and the json I'm looking for as another separate file. The idea is if something breaks instead of taking a hit to make the call again, I have the data and I should just process that. It's come in extremely handy when a site redesign changed the DOM and broke the parser. I do the same at $dayJob where I'm parsin…
Earlier quoted context omitted.
I don't see a GDPR challenge on this page?
That's probably because it's being inconsistently applied and you're not in Europe. If that's the case it's all the more insidious! Or you accepted the Tumblr terms in the past. This is what I get in the UK: https://imgur.com/a/zlWOByh
I'm in Switzerland (Europe), don't get the https://imgur.com/a/zlWOByh
> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…
Earlier quoted context omitted.
I crawl a specific site somewhere up to 50 unique URLs a day. I store both the unparsed full html as a file and the json I'm looking for as another separate file. The idea is if something breaks instead of taking a hit to make the call again, I have the data and I should just process that. It's come in extremely handy when a site redesign changed the DOM and broke the parser. I do the same at $dayJob where I'm parsin…
Off-topic: I so wish I worked for a company where my work involves scraping and storing and analyzing data. :(
As a side note, I have had quite a bit of experience trying to block automated scraping services. And I found that the best way is to quietly attempt to detect scraping. Then, serve up tainted data. In our case, competitors were scraping pricing data in order to competitively price their products without having to do the work. So we just randomly start to give them incorrect prices on every few products. Not only wou…
What do you do if you can't detect the scraping? And if you do detect scraping, how do you ensure the data you provide them is both invalid and consistent?
But once they lose a bunch of money the first time, they tend to stop trying. We tracked down one competitor that was mirroring our prices on an hourly basis. So we waited until late at night, tanked our price on a few expensive items, then placed orders on the competitors site.
The human touch tends to scare off scrapers faster than a technological fence anyway.
Earlier quoted context omitted.
Off-topic: I so wish I worked for a company where my work involves scraping and storing and analyzing data. :(
Now is a good time to work in this field since data science is hot and companies need web scrapers to provide the data for these models. Atleast that has been my experience in finance. Try applying!
Do you have any recommendations for places and/or interview practices?