Viewing profile — jakubbalada

> A growing catalog of up-to-date scrapers for popular websites would put of lot of freelancers out of work. I would invest in this. Check out Apify store ( https://apify.com/store…

You can use services like Anti-captcha [1] We have a public API on Apify for that [2] [1] https://anti-captcha.com/mainpage [2] https://www.apify.com/petr_cermak/anti-captcha-recap…

Typically depends on what you do with scraped data. There is an interesting recent US court ruling [1] [1] http://www.reuters.com/article/us-microsoft-linkedin-ruling/...

(co-founder here) We have hundreds of customers, half of them on recurring subscriptions, half just one-time customers paying for crawler configurations.

You might try Apifier for that, we've recently scraped more than 150k reviews for 27k restaurants in London. Here's a community crawler you can use: https://www.apifier.com/communi…

It's also hard to get direct access to the data. But you're right it's a hard sell to enterprises although we have some (e.g. real estate developer creating pricing maps)

Both - developers on a free plan using own RSS for sites without one and business people (mainly startups) building their products on top of Apifier. Typical use is an aggregator t…

We see a lot of users who needs data from the web or APIs for sites which doesn't have one. Just not all of them can code and we have to scale custom development.

Disclaimer: I'm a co-founder of Apifier [1]. It's not an open source, but free up to 10k pages per month. And it can handle modern JS web applications (your code runs in a context …

If you want to scrape many websites on a daily basis, have a look at https://www.apifier.com as an alternative. Disclaimer: I'm a cofounder there

At least in our batch we felt we should really only focus on our product. That's why we've got the grant - to be able to work on our startup full-time.

(Apifier co-founder here) It will be less beneficial now than it was in the first YCF batch. Other fellows really motivated us during the group office hours and other events. And s…

You're right, price per request would be easier for estimation. But as you can use JavaScript, you can scrape whole website with just one page request (see the SFO flights example)…

Of course not, API will be available soon - in a week or two. If you have some other feature requests, please let us know, we need to help with prioritization.

Yes, by default we respect robots.txt. There is a switch to disable it - on your own responsibility. We don't fully respect Crawl-delay, but minimum delay between requests for all …