Viewing profile — Ian_Kerins

A lot of the discussion around the /crawl endpoint seems to miss a key detail in the docs. The crawler explicitly identifies itself as a bot, respects robots.txt, and does not bypa…

Interesting take on it. Some people probably wouldn't like to be called soft but there is likely some truth to it. I feel it really comes down to priorities. Scraping has always be…

One of the main ideas, we explored here is how scraping has shifted from being mainly a technical challenge to an economic one: - Infrastructure and proxies have gotten cheaper, bu…

We just dropped the State of Web Scraping 2025 report. TL;DR: scraping is scaling—fast. - Market boom: Web scraping is growing 15% YoY and projected to hit $13B by 2033. Web data i…

The ethics of these free VPNs and hidden proxy SDKs are very questionable. But they are crazy profitable for the proxy providers running them so unlikely to go away. Did a teardown…

this proxy comparison tool shows you the best ones https://scrapeops.io/proxy-providers/comparison/

It is this type of attitude that is why websites are becoming so aggressive in blocking web scrapers. Being an "ethical web scraper" is about your own ethics, not abusing other peo…

Thanks for sharing it. It currently works with Scrapy & Python Request scrapers, will be launching SDKs for Node, Puppeteer, etc. soon.

Good point, wouldn't say archiving is unethical at all...I was thinking more along the lines of someone scraping a entire segment of a websites data and reproducing it 1 for 1 on t…

Some web scraping can be unethical, say for example if you are scraping a site solely to mirror their content and add zero value to the original content owner. However, there are a…

Interesting!...I'm not a lawyer, so the content for this piece was based on commentary in the below article. Was written by their lawyer, but would love to hear your counter point …

Haha, nice hack!

You can do it as a service, but that is highly competitive and basically trading time for money. Best ways are to productize it: - build a on-demand data api for a specific type of…

If anyone has anything else they think was missed or should be included then let me know!

100% agree, when scraping it should always be done respectfully. - If they provide a API, then use it. - Don't slam a website, ideally spread it out over hours of the day when ther…

This has a lot of good info on how to cloudflare and others work, and more creative ways to bypass them if the easier options don't work https://incolumitas.com/2021/05/20/avoid-pu…