A lot of people here seem to think this is somehow for their benefit, or that OpenAI and friends are trying to make something useful for the average person. They aren't spending billions of dollars to give everyone a personal assistant. They are spending billions now to save even more in wages later, and we are paying for the privilege of training their AI to do it. By the time this thing is useful enough to actually be a personal assistant, they will have released that capability in a model that is far too expensive for the average person.
Operator research preview
161–170 of 448 posts
Re: Operator research preview
#162I can already imagine "The Last Question"[1] playing out in real life — it's both fascinating and scary.
[1] Last Question By Isaac Asimov https://users.ece.cmu.edu/~gamvrosi/thelastq.html
Re: Operator research preview
#163[deleted]
Re: Operator research preview
#164I think this opens a new direction in terms of UI for companies like Instacart or Doordash — they can now optimise marketing for LLMs in place of humans, so they can just give benchmarks or quantized results for a product so the LLM can make a decision, instead of presenting the highest converting products first.
If the operator is told to find the most nutritious eggs for weight gain, the agent can refer to the nutrient labels (provided by Instacart) and then make a decision.
Re: Operator research preview
#165Assuming that Operator does become better (as the models have), and the cost of operation goes down, I would pay a monthly subscription to reduce my screentime. I wonder whether a UI for a new company is even needed in the future.
Re: Operator research preview
#166Earlier quoted context omitted.
Those numbers are not the full story. Note that GP specifically says: "Big jumps in benchmarks from _Claude's Computer Use_ though." Claude Computer Use was not SOTA for browser tasks at the time of its release (and is still not.) In WebArena, Operator does 58.1%. Previous SOTA for browser-use agents is 57.1%. In WebVoyager, Operator does 87.0%. Previous SOTA for browser-use agents is the exact same. See here for det…
Those two were two different models (Kura and jace.ai), and one model being SOTA at one benchmark doesn't make it SOTA overall. Moreover, both are specific for browser use, so they don't operate only on raw pixels but can read HTML/DOM, unlike general computer use models which rely on raw screenshots only.
I think I hit all those points in my previous post, except for the fact that it's two different models, as you've noted. That said, neither of them seem to report scores for the other benchmark in each particular case.
Re: Operator research preview
#167Can this open multiple tabs / navigate to different domains? When booking a restaurant I might want to confirm what the prices are on the menu or check google maps for the reviews / location.
Re: Operator research preview
#168Earlier quoted context omitted.
> But probably more difficult to get app developers on board. That's it. The problem is getting Postmates to agree to give away control of their UI. Giving away their ability to upsell you and push whatever makes them more money. Its never going to happen. Netflix still isn't integrated with Apple TV properly because they don't want to give away that access. I'm not convinced this is the path forward for computers ei…
> I'm not convinced this is the path forward for computers either though. With this approach they'll have to contend with the agent running into all the anti-bot measures that sites have implemented to deal with abuse. CAPTCHAs, flagging or blocking datacenter IP addresses, etc. Maybe deals could be struck to allow agents to be whitelisted, but that assumes the agents won't also be used for abuse. If you could get Ch…
The solution is simple, and it's what's already done with search by proprietary LLMs: reasoning happens on the LLM vendor's servers, tool use happens client-side. Whether for search or "computer use", the websites will register activity coming from the user's machine, as it should be, because LLMs act as User Agents here.
Of course, already with LLM-powered search we see growing number of people doing the selfish/idiotic thing and blocking or poisoning user-initiated LLM interactions[0]; hopefully LLM tools following the practice above will spread quickly enough to beat this idea out of peoples' heads.
--
[0] - As opposed to LLM company crawlers that scrape the web for training data - blocking those is fine and follows the cultural best practices on the web, which have been holding for decades now. But guess what, LLM crawlers tend to obey robots.txt. The "bots" that don't are usually the ones performing specific query on behalf of users; such bots act as User Agents, neither have nor ever had any obligation to obey robots.txt.
Re: Operator research preview
#169What is fascinating about this announcement is if you look into future after considerable improvements in product and the model, we will be just chatting with ChatGPT to book dinner tables, flights, buy groceries and do all sort of mundane and hugely boring things we do on the web, just by talking to the agents. I'd definitely love that.
I don't. Chat interface sucks; for most of these things, a more direct interface could be much more ergonomic, and easier to operate and integrate. The only reason we don't have those interfaces is because neither restaurants, nor airlines, nor online stores, nor any other businesses actually want us to have them. To a business, the user interface isn't there to help the user achieve their goals - it's a platform for milking the users as much as possible. To a lesser or greater extent, almost every site actively defeats attempts at interoperability.
Denying interoperability is so culturally ingrained at this point, that it got pretty much baked into entire web stack. The only force currently countering this is accessibility - screen readers are pretty much an interoperability backdoor with legal backing in some situations, so not every company gets to ignore it.
No, we'll have to settle for "chat agents" powered by multimodal LLMs working as general-purpose web scrappers, because those models are the ultimate form of adversarial interoperability, and chat agents are the cheapest, least-effort way to let users operate them.
Re: Operator research preview
#170I sometimes wonder if Rabbit and their LAM (https://www.rabbit.tech/lam-playground) were just a year too early to market.