21–30 of 30 posts
I think I understand pagination — but can you elaborate on proxy rotation? > combines the practicality of language models with the powerful features of a traditional scraper such as pagination and proxy rotation
I think that you're building an unethical business, and should not get advice or publicity from this community. Doubly so given you already had a very high visibility Show HN for this just a few months ago.
I don't think this community agrees with you that web scraping is inherently unethical. In fact I think many (most?) in this community would argue that web scraping is an almost fundamental feature of the web itself, and that attempts at preventing it are more unethical than scraping.
"proxy rotation" in the first line does not bode well for ethics.
I think that you're building an unethical business, and should not get advice or publicity from this community. Doubly so given you already had a very high visibility Show HN for this just a few months ago.
I think I understand pagination — but can you elaborate on proxy rotation? > combines the practicality of language models with the powerful features of a traditional scraper such as pagination and proxy rotation
Earlier quoted context omitted.
You’re not only sacrificing simplicity for determinism but also stability. What good is being deterministic when the underlying web page keeps changing all the time, breaking your selectors? This seems like a more stable approach.
However if the selectors break you can notice that quite easily.
Earlier quoted context omitted.
I don't think this community agrees with you that web scraping is inherently unethical. In fact I think many (most?) in this community would argue that web scraping is an almost fundamental feature of the web itself, and that attempts at preventing it are more unethical than scraping.
I doubt the objections are about scraping per se, but about unethical scraping where no consideration is given to etiquette. "proxy rotation" in the first line does not bode well for ethics.
Earlier quoted context omitted.
I doubt the objections are about scraping per se, but about unethical scraping where no consideration is given to etiquette. "proxy rotation" in the first line does not bode well for ethics.
What's wrong with proxy rotation? Big Tech attempts to prevent any scraping of their content whatsoever. In the context of that web, proxy rotation is table stakes.
Not every crawled site is "big tech" or even commercial.
Earlier quoted context omitted.
You’re not only sacrificing simplicity for determinism but also stability. What good is being deterministic when the underlying web page keeps changing all the time, breaking your selectors? This seems like a more stable approach.
However if the selectors break you can notice that quite easily.
Earlier quoted context omitted.
What's wrong with proxy rotation? Big Tech attempts to prevent any scraping of their content whatsoever. In the context of that web, proxy rotation is table stakes.
Without any documentation about how etiquette will be respected and sites won't be hammered, it's fair to be sceptical about the ethics. Not every crawled site is "big tech" or even commercial.
Earlier quoted context omitted.
Without any documentation about how etiquette will be respected and sites won't be hammered, it's fair to be sceptical about the ethics. Not every crawled site is "big tech" or even commercial.
I don't disagree, but I also don't think "proxy rotation" immediately implies sites will be hammered either.