Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

Post reply on HN
Print view View original

581–590 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#581

Post by anonzzzies »

findjashua wrote: post #574
p1esk wrote: post #573

Earlier quoted context omitted.

What exactly is left when we remove coding from software engineering? Could it be handled by a manager? Or perhaps by a single senior SWE who could now perform the work of an entire team using these rapidly advancing AI coders?

for a lot of tasks that aren't as cut & dry, i often find myself having to provide it pseudo code, which it can then one-shot to working code. don't get me wrong, it's still a massive upgrade from the pre-sonnet era, but i still don't think it can take a high-level requirement and convert it into a working project... yet

> but i still don't think it can take a high-level requirement and convert it into a working project.

It cannot, you need to hand-hold it, as in, to make something larger than a (albeit good looking) to do app, you don't need to write code , but you do need to be able to review and debug code and take the architectural decisions. It'll simply loop forever otherwise.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#582

Post by miyuru »

bambax wrote: post #418
pants2 wrote: post #357

Earlier quoted context omitted.

Basically, if it means companies can introduce automation without changing anything about the tooling/workflow/programs they already use, it's going to be MASSIVE. Just and install and a prompt and you've already automated a lengthy manual process - awesome.

Companies are going to install an AI inside their own proprietary systems full of proprietary and confidential data and PII about their customers and prospects and whatnot, and let it run around and click on random buttons and submit random forms? Really??!? What could possibly go wrong. I'm currently trying to do a large ORC project using Google Vision API, and then Gemini 1.5 Pro 002 to parse and reconstruct the re…

Have you tried any others? From what I have tried Gemini makes the most mistakes out of all.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#583

Post by thenameless7741 »

simonw wrote: post #499

Claude 3.5 Opus is no longer mentioned at all on https://docs.anthropic.com/en/docs/about-claude/models Internet Archive confirms that on the 8th of October that page listed 3.5 Opus as coming "Later this year" https://web.archive.org/web/20241008222204/https://docs.anth... The fact that it's no longer listed suggests that its release has at least been delayed for an unpredictable amount of time, or maybe even cancel…

Before anyone reads too much into this, here's what an Anthropic staff said on Discord:

> i don't write the docs, no clue

> afaik opus plan same as its ever been

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#584

Post by jstummbillig »

usaar333 wrote: post #524
anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

FWIW, the refactor benchmark is quite mechanical - it just stresses reliability of LLMs over long context windows: Questions are variants of: Refactor the _set_csrf_cookie method in the CsrfViewMiddleware class to be a stand alone, top level function. Name the new function _set_csrf_cookie, exactly the same name as the existing method. Update any existing self._set_csrf_cookie calls to work with the new _set_csrf_coo…

Assuming that that is indeed what most of the benchmark does: If the LLMs are as bad as it as the numbers suggest, then it seems like a perfectly good benchmark. I would definitely want them to be able to do stuff like that when I let them write my code.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#585

Post by brabel »

LASR wrote: post #344

This is actually a huge deal. As someone building AI SaaS products, I used to have the position that directly integrating with APIs is going to get us most of the way there in terms of complete AI automation. I wanted to take at stab at this problem and started researching some daily busineses and how they use software. My brother-in-law (who is a doctor) showed me the bespoke software they use in his practice. Runni…

I tried once to integrate with software used by pharmacists in Australia (which costs a fortune and looks straight out of the 90's). Even though they have a SQL database with all information about everything, the DB is encrypted and they provide no way to access it. I tried contacting the company to learn how I could integrate with it but they offered no help. It's not in their interest to help upcoming competitors and they claim they cannot provide the encryption key as that's for protecting the customer's privacy, forgetting that the pharmacists already have access to all data through their software. But without a way to automatically extract information from the DB, there's no way to migrate to new software as no pharmacist would want to use new software if it cannot "import" the data they need.

It's clear that the reason there's no competition in the market is that the companies (I think it's literally one company) selling this software want it that way, and legislation is yet to catch up with the way they ensure their monopoloy.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#586

Post by codebolt »

seriocomic wrote: post #580
stavros wrote: post #491

Earlier quoted context omitted.

I use Claude for coding and it's fantastic. I definitely have outsourced a lot of my coding to it.

What's the (current) best way to integrate it? VS Code extension? Other IDE?

I'll throw this out here as well: Is there any decent alternative to GitHub Copilot when using Visual Studio? (Pretty happy with it to be fair, but would be open to trying others.)

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#587

Post by RobinL »

seriocomic wrote: post #580
stavros wrote: post #491

Earlier quoted context omitted.

I use Claude for coding and it's fantastic. I definitely have outsourced a lot of my coding to it.

What's the (current) best way to integrate it? VS Code extension? Other IDE?

I use cursor (cursor.com) and it's fantastic

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#588

Post by snowe2010 »

aerhardt wrote: post #478
vl wrote: post #468

Earlier quoted context omitted.

Fundamentally there is human with limited brain capacity that got trained to that. It’s just a question of time when there are equally capable, and then exceedingly capable models. There is nothing magical or special about human brain. The only question is how fast it is going to happen. Ie what percentage of jobs is going to be replaced next year and so on.

> There is nothing magical or special about human brain. There is a lot about the human brain that even the world's top neuroscientists don't know. There's plenty of magic about it if we define magic as undiscovered knowledge. There's also no consensus among top AI researchers that current techniques like LLMs will get us anywhere close to AGI. Nothing I've seen on current models (not even o1-preview) suggests to me…

Defining AGI as “can reason about 5MLOC” is ridiculous. When do the goal posts stop moving? When a computer can solve time travel? Babies have behavior all the time that is no more differentiable from what an LLM does on a normal basis (including terrible logic and hallucinations).

The majority of people on the planet can barely reason about how any given politician will affect them, even when there’s a billion resources out there telling them exactly that. No reasonable human would ever define AGI as having anything to do with coding at all, since that’s not even “general intelligence”… it’s learned facts and logic.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#589

Post by gloosx »

anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

I will repeat my question from one of the previous threads:

Can someone explain these Aider benchmarks to me? They pass same 113 tests through llm every time. Why they then extrapolate ability of llm to pass these 113 basic python challenges to the general ability to produce/edit code? Couldn't LLM provider just fine-tune their model for these tasks specifically - since they are static - to get ad value?

Did anyone ever try to change them test cases or wiggle conditions a bit to see if it will still hit the same %?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#590

Post by maeil »

seriocomic wrote: post #580
stavros wrote: post #491

Earlier quoted context omitted.

I use Claude for coding and it's fantastic. I definitely have outsourced a lot of my coding to it.

What's the (current) best way to integrate it? VS Code extension? Other IDE?

Aider, created by the originator of this very comment thread.
Post reply on HN