11–20 of 28 posts
Is there support for robots.txt so service operators can opt out of your mass scraping?
Fantastic. MCPs are showing promise.
Thanks! And yeah MCP is super promising. We announced this on X and LinkedIn yesterday and the response has been really good. A lot of people with a bunch of use cases. One surprising thing is there’s also a bunch of semi/non-technical people using our MCP server and the installation experience for them rn just absolutely sucks. I think once auth and 1-click install are solved, MCP could become the standard way to in…
It is fine if that is the scope. It is also understandable why Anthropic chose to use a stateful protocol where stateless HTTP would be more than enough. They are catering for the default transport layer which is stdio based where state needs to be established.
There are also other aspects of it that are simply unnecessarily complex and resource intensive for no good reasons.
Is there support for robots.txt so service operators can opt out of your mass scraping?
No, we don't enforce any robots.txt restrictions ourselves. We also don't do any scraping ourselves. We provide browser infrastructure that operates like any normal browser would - what users choose to do with it is up to them. We're building tools that give AI agents the same web access capabilities that humans have, don't think it's our place to impose any additional limitations.
Is there support for robots.txt so service operators can opt out of your mass scraping?
No, we don't enforce any robots.txt restrictions ourselves. We also don't do any scraping ourselves. We provide browser infrastructure that operates like any normal browser would - what users choose to do with it is up to them. We're building tools that give AI agents the same web access capabilities that humans have, don't think it's our place to impose any additional limitations.
This looks cool. 1) I looked at the pricing. Is search included in the price - (you just pay credits/browser time?) 2) Can you tell me more about the source of your residential proxies? I am new to this space, so don’t know how people source these legitimately. Thanks!
Thanks! 1) Yep, you just pay from browser time and proxy usage 2) We use a handful of proxy providers under the hood ourselves. There’s a lot of shady ones but we only work with ones where we’ve vetted the source of. Different providers source proxies in different ways - directly from ISPs, paying end sources for proxies etc
Earlier quoted context omitted.
Thanks! And yeah MCP is super promising. We announced this on X and LinkedIn yesterday and the response has been really good. A lot of people with a bunch of use cases. One surprising thing is there’s also a bunch of semi/non-technical people using our MCP server and the installation experience for them rn just absolutely sucks. I think once auth and 1-click install are solved, MCP could become the standard way to in…
The MCP protocol is really not that good. The stateful nature of it makes it only suitable as a local desktop RPC pipe - certainly not something that will work well on mobile nor anything anyone would like to try to run maintain in a server-to-serve context. It is fine if that is the scope. It is also understandable why Anthropic chose to use a stateful protocol where stateless HTTP would be more than enough. They ar…
This looks cool. 1) I looked at the pricing. Is search included in the price - (you just pay credits/browser time?) 2) Can you tell me more about the source of your residential proxies? I am new to this space, so don’t know how people source these legitimately. Thanks!
Thanks! 1) Yep, you just pay from browser time and proxy usage 2) We use a handful of proxy providers under the hood ourselves. There’s a lot of shady ones but we only work with ones where we’ve vetted the source of. Different providers source proxies in different ways - directly from ISPs, paying end sources for proxies etc
Is there support for robots.txt so service operators can opt out of your mass scraping?
I pointed their scraper at a url on my server to test it's behavior. It made four separate requests to the same page, three with the UA "udici"[2], and one with the UA "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36" and a different IP. The first three requests were all made within the span of 1 second, and the fourth 27 seconds later.
I emailed their published support address asking for an IP range and UA. They gave me the entire IP range of google cloud, and ignored the UA question.
This goes well beyond the "it's up to our users to implement responsible scraping practices" implication from the developer's other comment[3]. Instead, their service behaves maliciously by default, and they have implemented and documented switches that users can toggle for additional malicious scraping behavior. As far as I can tell, it is not even possible to implement a robots.txt-respecting scraper on top of this, because I couldn't find any mechanism for users to set a specific UA string.
[1]: https://docs.hyperbrowser.ai/sessions/advanced-privacy-and-a... (archived: https://web.archive.org/web/20250322045952/https://docs.hype..., http://archive.today/2025.03.22-050029/https://docs.hyperbro...)