21–30 of 102 posts
how do you avoid getting banned by the companies you scrape? Most ToSs have a clause like: > We prohibit crawling, scraping, caching or otherwise accessing any content on the Service via automated means... [etc]
how do you avoid getting banned by the companies you scrape? Most ToSs have a clause like: > We prohibit crawling, scraping, caching or otherwise accessing any content on the Service via automated means... [etc]
Very cool, but super slow, especially for an API point, which I would expect you could use directly from a front end. Tested on a site I regularly visit dashblock (3 selectors, ~20 items): 16.911 seconds curl (no scraping): 60 ms chrome: 987 ms edit: added chrome
Have done a lot of scraping in my life, and I'm super excited about what Hugues and Max have built. I tested a super early version and was surprised how well it worked.
Congrats on the launch! Look forward to playing around with it. I remember a similar startup making quite a splash on HN a few years ago — http://www.kimonolabs.com/ . Do you know why they shut it down and what you guys are doing differently?
Thank you! Kimono Labs built an amazing product and has been finally acquired by Palantir. The main difference between us comes from our machine learning model that gives stability to extraction over time and parse webpages in a generic way. We are also working on automating navigation, which is something they didn't do =)
Very cool, but super slow, especially for an API point, which I would expect you could use directly from a front end. Tested on a site I regularly visit dashblock (3 selectors, ~20 items): 16.911 seconds curl (no scraping): 60 ms chrome: 987 ms edit: added chrome
Indeed, we are rendering the whole page with the javascript, that's why it takes longer than a curl. For now, it's especially useful for dynamic pages but we also plan on supporting pages that don't require rendering.
Recommendation: https://github.com/cliqz-oss/adblocker Should be the fastest adblocker library (used by Ghostery, Cliqz and Brave)
Really neat !