Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

521–530 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#521

Earlier quoted context omitted.

Operating a computer?

Right. We don’t have a word for that. Like “using a bicycle” has the word “bicycling”. Tho someone here suggested “computering” which is pretty good.

I’ve been saying computering since the 90s but it never took off. Need to get a petition for OED going.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#522

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

When using these models via the official Anthropic API, do I have to do anything to "opt in" to the new Sonnet, or am I switched over automatically?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#523

Earlier quoted context omitted.

From a security standpoint, what's considered the "proper" way of assigning a bot access based on a person's 2FA? Would that be some sort of limited scope expiring token like GitHub's fine-grained personal access tokens?

I don't know, I feel like it has to be some sort of near field identity proof. E.g. as long as you are wearing a piece of equipment to a physical computer near you can run all those automations for you, or similar. I haven't fully thought what the best solution could be or whether someone is already working on it, but I feel like there has to be something like that, which would allow you better UX in terms of access,…

That's a really cool idea.

The main attack vector would then probably be some man-in-the-middle intercepting the signal from your wearable, which leads me to wonder whether you could protect yourself by having the responses valid for only an extremely short duration, e.g. ~1ms, such that there's no way for an attacker to do anything with the token unless they gain control over compute inside your house.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#524

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

FWIW, the refactor benchmark is quite mechanical - it just stresses reliability of LLMs over long context windows:

Questions are variants of:

Refactor the _set_csrf_cookie method in the CsrfViewMiddleware class to be a stand alone, top level function. Name the new function _set_csrf_cookie, exactly the same name as the existing method. Update any existing self._set_csrf_cookie calls to work with the new _set_csrf_cookie function.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#525
post #374

Earlier quoted context omitted.

This has existed for a long time, it's called "RPA" or Robotic Process Automation. The biggest incumbent in this space is UiPath, but there are a host of startups and large companies alike that are tackling it. Most of the things that RPA is used for can be easily scripted, e.g. download a form from one website, open up Adobe. There are a lot of startups that are trying to build agentic versions of RPA, I'm glad to s…

Honestly, this is going to be huge for healthcare. There's an incredible amount of waste due to incumbent tech making interoperability difficult.

Healthcare has the extra complication of HIPAA / equivalent local laws, and institutions being extremely unwilling to process patient data on devices they don't directly control.

I don't think this is going to work in that industry until local models get good enough to do it, and small enoguh to be affordable to hospitals.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#526
post #283

Earlier quoted context omitted.

What a neat bench mark! I'm blown away that o1 absolutely crushes everyone else in this. I guess the chain of thought really hashes out those associations.

Isn't it possible that o1 was also trained on this data (or something super similar) directly? The score seems disproportionately high.

They definitely considered it. Early theinformation articles talked about how high the performance of strawberry was on it.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#527

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

When using these models via the official Anthropic API, do I have to do anything to "opt in" to the new Sonnet, or am I switched over automatically?

That depends on the model ID you are using.

If you use "claude-3-5-sonnet-latest" you'll be upgraded to "claude-3-5-sonnet-20241022" already - I tested that this morning.

If you're on "claude-3-5-sonnet-20240620" you'll need to change that ID to either the -latest one or the -20241022 one.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#528
post #519
post #516

Earlier quoted context omitted.

The Claude web UI cannot generate binary files, it's (currently) restricted to plain text. If you want binary files you'd be better off with ChatGPT with Code Interpreter mode, which can run Python code that generates binary content. Or ask Claude to write you Python code that generates Excel files and then copy and paste that onto your own computer and run it yourself.

> you'd be better off with ChatGPT with Code Interpreter mode Yes, this is what I am saying. Why go to the trouble to build something as capable as Claude and then hamstring it from being as useful as ChatGPT? I have no doubt that Claude could be more useful if the Anthropic team would let it shine.

They've been investing engineering effort in Claude Artifacts instead, which I find incredibly useful: https://simonwillison.net/2024/Oct/21/claude-artifacts/

I'd love to see them produce their own Code Interpreter alternative, but in the meantime it's open for third parties to offer that (and a few do).

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#529

Earlier quoted context omitted.

If they're accessible to screen readers they have one. Accessibility is API for apps in disguise. In this case I doubt they're networked apps so they probably don't have a server API.

> In this case I doubt they're networked apps so they probably don't have a server API. I think it would be very unusual this decade for software used to run either a medical practice or tax accountants to not be networked. Most such practices have multiple doctors/accountants, each with their individual computer, and they want to be able to share files, so that if your doctor/accountant is away their colleague can a…

Being “on the network” doesn’t mean there’s an accessible API. See QuickBooks Desktop. Intuit forces you into using their API, which is XML-based and ranges from slow to timing out.

Is the idea that someone will always reverse engineer it? Yes, but QuickBooks is brittle as is (you can count on at least one database corruption every year or two). I have zero interest in treading into unsupported territory when database corruption is involved and I’m likely going to need Intuit’s help recovering. We can try to restore from backup, but when there’s corruption it doesn’t always restore successfully, or the corruption was lingering silently for some time and rears its head again after a successful restore, and then we’re back to needing Intuit’s help.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#530
post #499

Claude 3.5 Opus is no longer mentioned at all on https://docs.anthropic.com/en/docs/about-claude/models Internet Archive confirms that on the 8th of October that page listed 3.5 Opus as coming "Later this year" https://web.archive.org/web/20241008222204/https://docs.anth... The fact that it's no longer listed suggests that its release has at least been delayed for an unpredictable amount of time, or maybe even cancel…

It's possible that they've determined that Opus no longer makes sense if they're able to focus on continuously optimising Sonnet. That said, Anthropic have been relatively good at setting and managing expectations, so today would have been a good time to make that clear.
Post reply on HN