Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

Post reply on HN
Print view View original

161–170 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#161

Post by HarHarVeryFunny »

HanClinto wrote: post #13

Why not rev the numbers? "3.5" vs. "3.5 New" feels weird -- is there a particular reason why Anthropic doesn't want to call this 3.6 (or even 3.5.1)?

Well, by calling it 3.5, they are telling you that this is NOT the next-gen 4.0 that they presumably have in the works, and also not downplaying it by just calling it 3.6 (and anyways they are not advancing versions by 0.1 increments - it seems 3.5 was just meant to convey "half way from 3.0 to 4.0"). Maybe the architecture is unchanged, and this just reflects more pre and/or post-training?

Also, they still haven't released 3.5 Opus yet, but perhaps 3.5 Haiku is a distillation of that, indicating that it is close.

From a competitive POV, it makes sense that they respond to OpenAI's 4o and o1 without bumping the version to Claude 4.0, which presumably is what they will call their competitor to GPT-5, and probably not release until GPT-5 is out.

I'm a fan of Anthropic, and not of OpenAI, and I like the versioning and competitive comparisons. Sonnet 3.5 still best coder, better than o1, has to hurt, and a highly performant cheap Haiku 3.5 will hit OpenAI in the wallet.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#162

Post by cherioo »

torginus wrote: post #104

Claude's current ability to use computers is imperfect. Some actions that people perform effortlessly—scrolling, dragging, zooming—currently present challenges for Claude and we encourage developers to begin exploration with low-risk tasks. Nice, but I wonder why didn't they use UI automation/accessibility libraries, that have access to the semantic structure of apps/web pages, as well as accessing documents directly…

I think this is to make human /user experience better. If you use accessibility features, then user need to know how to use those features. Similar to another comment in here, the UX they shoot for is “click the red button with cancel on it”, and ship that ASAP.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#163

Post by accrual »

msoad wrote: post #71

I skimmed through the computer use code. It's possible to build this with other AI providers too. For instance you can asks ChatGPT API to call functions for click and scroll and type with specific parameters and execute them using OS's APIs (A11y APIs usually) Did I miss something? Did they have to make changes to the model for this?

> execute them using OS's APIs (A11y APIs usually)

I wonder if we'll end up with a new set of AI APIs in Windows, macOS, and Linux in the future. Maybe an easier way for them to iterate through windows and the UI elements available in each.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#164

Post by KingOfCoders »

accrual wrote: post #139
qup wrote: post #129

Earlier quoted context omitted.

Have it print a summary of you and stick it in your prompt

Yeah, there was an interesting prompt making rounds recently, something like "Summarize everything you know about me" and leveraging ChatGPT's memory feature to provide insights about oneself. My only trouble with the memory feature is it remembers things that aren't important, like "user is trying to write an async function" and other transient tasks, which is more about what I was doing some random Tuesday and not…

Did that too with interesting results.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#165

Post by TacticalCoder »

One suggestion, use the following prompt at a LLM:

    The combination of the words "computer use" is highly confusing. It's also "Yoda speak". For example it's hard for humans to parse the sentences *"Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku"*, *"Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku "* (it literally relies on the comma to make any sense) and *"Computer use for automated interaction"* (in the youtube vid's title: this one is just broken english). Please suggest terms that are not confusing for a new ability allowing an AI to control a computer as if it was a human.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#166

Post by myprotegeai »

accrual wrote: post #150
myprotegeai wrote: post #136

How long until "computer use" is tricked into entering PII or PHI into an attackers website?

I imagine initial computer use models will be kind of like untrained or unskilled computer users today (for example, some kids and grandparents). They'll do their best but will inevitably be easy to trick into clicking unscrupulous links and UI elements. Will an AI model be able to correctly choose between a giant green "DOWNLOAD NOW!" advertisement/virus button and a smaller link to the actual desired file?

Exactly. Personalized ads are now prompt injection vectors.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#167

Post by accrual »

postalcoder wrote: post #25

and i was just planning to go to sleep…

I discovered Mindcraft recently and stayed up a few hours too late trying to convince my local model to play Minecraft. Seems like every time a new capability becomes available, I can't wait to experiment with it for hours, even at the cost of sleep.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#169

Post by 29decibel »

I am surprised it uses macOS as the demo, as I thought it would be harder to control vs Ubuntu. But maybe at the same time, macOS is the most predictable/reliable desktop environment? I noticed that they use virtual environment for the demo, curious how do they build that along with docker, is that leveraging the latest virtualization framework from Apple?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#170

Post by accrual »

m3kw9 wrote: post #48

I suspect they are gonna need some local offload capabilities for Computer Use, the repeated screen reading can definitely be done locally on modern machines, otherwise the cost maybe impractical.

Maybe we need some agent running on the PC to offload some of these tasks. It could scrape the display at 30 or 60 Hz and produce a textual version of what's going on for the model to consume.
Post reply on HN