151–160 of 228 posts
There's a straw man here. If you want to reorder an item on Amazon: click on 'order history', scroll, and click buy. This is a well-optimized path already and it doesn't require your full attention. I suspect the agent approach takes more effort as you need to type and then monitor what the AI is doing.
Earlier quoted context omitted.
There's no reason not to respect it. If your browser behaves, it's not going to be excluded in robots.txt. If your browser doesn't behave, you should at least respect robots.txt. If your browser doesn't behave, and you continue to ignore robots.txt, that's just... shitty.
> If your browser behaves, it's not going to be excluded in robots.txt. No, it's common practice to allow Googlebot and deny all other crawlers by default [0]. This is within their rights when it comes to true scrapers, but it's part of why I'm very uncomfortable with the idea of applying robots.txt to what are clearly user agents. It sets a precedent where it's not inconceivable that we have websites curating allowl…
I am not sure I agree with an AI-aided browser, that will scrape sites and aggregate that information, being classified as "clearly" a user agent.
If this browser were to gain traction and ends up being abusive to the web, that's bad too.
Where do you draw the line of crawler vs. automated "user agent"? Is it a certain number of web requests per minute? How are you defining "true scraper"?
Earlier quoted context omitted.
>It's meant for automated scrapers that recursively retrieve all pages on your website, _which this browser is not doing at all_ AFAIK this is false, and this browser can do things like "summarize all the cooking recipes linked in this page" and therefore act exactly like a scraper (even if at smaller scale than most scrapers) If tomorrow magically all phones and all computers had an ad-blocking browser installed -an…
https://www.robotstxt.org/faq/what.html > A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. There's nothing recursive about "summarize all the cooking recipes linked on this page". That's a single-level iterative loop. I will grant that I should alter my original statement: if OP wanted to respect rob…
I think there is big value in something that is on the side of the user, in particular in processing user-hostile material. Here are some concrete use cases: * Buying a sofa. You want to filter for sofas of a specific size, with certain features, marketing sites want to feed you a bunch of marketing slop for each sofa before giving you the details . This generalises to many domains. * You have a few friends who are s…
This jumped out to me as well. Even sites like Amazon lack per-item-cost sorting, which can be really helpful when buying in bulk. Historically we've seen people use scraping and data science to build sites like https://diskprices.com/; without using LLMs. If LLMs are useful for those types of tasks, perhaps we'll see a surge in similar sites instead of end users doing prompt engineering in their browser.
> You want to see any new posts except for those couple of guys who are always posting the same thing.
It looks like nextdoor supports blocking users, although other sites may not.
Earlier quoted context omitted.
> If your browser behaves, it's not going to be excluded in robots.txt. No, it's common practice to allow Googlebot and deny all other crawlers by default [0]. This is within their rights when it comes to true scrapers, but it's part of why I'm very uncomfortable with the idea of applying robots.txt to what are clearly user agents. It sets a precedent where it's not inconceivable that we have websites curating allowl…
> clearly user agents I am not sure I agree with an AI-aided browser, that will scrape sites and aggregate that information, being classified as "clearly" a user agent. If this browser were to gain traction and ends up being abusive to the web, that's bad too. Where do you draw the line of crawler vs. automated "user agent"? Is it a certain number of web requests per minute? How are you defining "true scraper"?
> A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.
To me "recursive" is key—it transforms the traffic pattern from one that strongly resembles that of a human to one that touches every page on the site, breaks caching by visiting pages humans wouldn't typically, and produces not just a little bit more but orders of magnitude more traffic.
I was persuaded in another subthread that Nxtscape should respect robots.txt if a user issues a recursive request. I don't think it should if the request is "open these 5 subreddits and summarize the most popular links uploaded since yesterday", because the resulting traffic pattern is nearly identical to what I'd have done by hand (especially if the browser implements proper rate limiting, which I believe it should).
Earlier quoted context omitted.
https://www.robotstxt.org/faq/what.html > A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. There's nothing recursive about "summarize all the cooking recipes linked on this page". That's a single-level iterative loop. I will grant that I should alter my original statement: if OP wanted to respect rob…
The concept of robots.txt was created in a different time, when nobody envisioned that users would one day use commands written in plain English sentences to interact with websites (including interacting with multiple pages with such commands), so the discussion about if AI browsers should respect it or if they should not is senseless, and instead -if this kind of usage takes off- it would probably make more sense to…
Earlier quoted context omitted.
Thanks for raising this - it's a critical concern and you're absolutely right to be cautious. This is exactly why we're going local-first and open source. With cloud agents (like Manus.im), you're trusting a black box with your credentials. With local agents, you maintain control: - Agents only run when you explicitly trigger them - You see exactly what they're doing in real-time and can stop them - You can run tasks…
this sounds LLM generated regardless, you did not answer OPs point, which is that any potentially malicious site can prompt inject you at any point, and trigger an MCP or any other action or whatever before you see them and stop them. The whole point of an AI browser is like self-driving car, being able to de-focus and let it do its thing. If i have to be nervous to watch if im getting hacked at any given second, the…
Earlier quoted context omitted.
Thanks for the honest feedback! Definitely agree there is good amount of competition here. But we do think there is a gap in the market for open-source, community driven and privacy-first AI browser. (Something like Brave?)
remember, gaps in the market sometimes exist for a reason. Forget AI. How many open source, community driven and privacy first browsers have made serious money? Brave is a decent example but their business model is actually complicated, it includes a lot of little stuff. And they dont have the unit cost of LLMs (im assuming at some point you will take the burden of the llms, if not local)
Island browser, chrome enterprise have kinda of validated the need for enterprise version of browser with VPN and DLP engine (data-loss-prevention).
> simple stuff like reordering tide pods from amazon [..] shouldn't need our full attention anymore There's a straw man here. If you want to reorder an item on Amazon: click on 'order history', scroll, and click buy. This is a well-optimized path already and it doesn't require your full attention. I suspect the agent approach takes more effort as you need to type and then monitor what the AI is doing.
I think there is big value in something that is on the side of the user, in particular in processing user-hostile material. Here are some concrete use cases: * Buying a sofa. You want to filter for sofas of a specific size, with certain features, marketing sites want to feed you a bunch of marketing slop for each sofa before giving you the details . This generalises to many domains. * You have a few friends who are s…