61–70 of 102 posts
Earlier quoted context omitted.
Good question. However, that would require websites' users to validate a Captcha every time they navigate it, which is not optimal in terms of user experience.
reCaptcha V3 operates behind the scenes though: https://developers.google.com/recaptcha/docs/v3
Congrats! Look forward to trying it out.
how do you avoid getting banned by the companies you scrape? Most ToSs have a clause like: > We prohibit crawling, scraping, caching or otherwise accessing any content on the Service via automated means... [etc]
This may now be moot after the LinkedIn Vs hiq labs case a couple of days ago which appears to have blanket legalised we scraping.
how do you avoid getting banned by the companies you scrape? Most ToSs have a clause like: > We prohibit crawling, scraping, caching or otherwise accessing any content on the Service via automated means... [etc]
Agreed, some websites are really reticent about scraping. But let's think about Google, they are scraping the whole web regardless of the ToS of the websites, so it all boils down to one question : do you create value for the website owner ? That's why we want to focus on use-cases where we create value for both, our users and the website owner. If you think about Yodlee/Plaid in the banking sector, they built partne…
This has been tried many times and it never seems to gain traction to become a relevant concept. Off the top of my head, I remember Kimono Labs that looked quite promising. Then it was acquired by Palantir and shut down. I also have seen many solutions that are similar (basically most scraping companies, like Diffbot which also claims to use machine learning for their extraction techniques) What's the plan here to re…
This has been tried many times and it never seems to gain traction to become a relevant concept. Off the top of my head, I remember Kimono Labs that looked quite promising. Then it was acquired by Palantir and shut down. I also have seen many solutions that are similar (basically most scraping companies, like Diffbot which also claims to use machine learning for their extraction techniques) What's the plan here to re…
In regards to your question about companies' concerns: if the data is made publicly available (i.e. web page is not behind authentication), then why should it matter how it's accessed?
Also might be concerned about inaccuracies from variable pricing models for example. There’s a few reasons why you may not want it accessible - hence one of the reasons why CORS is even a thing.
Has anyone tried this for careers pages? Would be interested in how this performs on a random sample of ~50 crunchbase NYC startups’ careers pages. I dunno how much time would have to be spent training data...
We did :) It works on all kind of pages. You just have to set it up on one page and it will work on all similar pages of the website. Did you have in mind to train a model to recognise careers pages across websites ?
Very cool! Is there a way to authenticate in to a site and then keep a session alive to scrape private content? Can it pass cookies or can you manually set headers?