Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

Post reply on HN
Print view View original

671–680 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#671

Post by pona-a »

IncreasePosts wrote: post #438
unshavedyak wrote: post #364

Earlier quoted context omitted.

I feel pain for the people who will be employed to "prompt engineer" the behavior of these things. When they inevitably hallucinate some insane behavior a human will have to take blame for why it's not working.. and yea, that'll be fun to be on the receiving end of.

That won't even be a real job. How exactly will there be this complex intelligence that can solve all these real world problems, but can't handle some ambiguity in some inputs it is provided? Wouldn't the ultra smart AI just ask clarifying questions so that literally anyone can "prompt engineer"?

As long as there is liability, there must be a human to blame, no matter how irrational. Every system has a failure mode, and ML models, especially the larger ones, often have the most odd and unique ones.

For example, we can mostly agree CLIP does a fine job classifying images, except if you glue a sticky note saying "iPod" onto an apple, it would say classify it as such.

No matter the performance, these are categorically statistical machines reaching for the most immediately useful representations, yielding an incoherent world model. These systems will be proposed as replacement to humans, they will do their best to pretend to work, they will inevitably fail over a long enough time horizon, and a human accustomed to rubber-stamping its decisions, or perhaps fooled by the shape of a correct answer, or simply tired enough to let it slip by, will take the blame.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#672

Post by ryandvm »

hobofan wrote: post #85
diggan wrote: post #61

Earlier quoted context omitted.

> Opus hasn't yet gotten an update from 3 to 3.5, and if you line up the benchmarks, the Sonnet "3.5 New" model seems to beat it everywhere Why isn't Anthropic clearer about Sonnet being better then? Why isn't it included in the benchmark if new Sonnet beats Opus? Why are they so ambiguous with their language? For example, https://www.anthropic.com/api says: > Sonnet - Our best combination of performance and speed fo…

> I don't understand why this seems purposefully ambiguous? I wouldn't attribute this to malice when it can also be explained by incompetence. Sonnet 3.5 New > Opus 3 > Sonnet 3.5 is generally how they stack up against each other when looking at the total benchmarks. "Sonnet 3.5 New" has just been announced, and they likely just haven't updated the marketing copy across the whole page yet, and maybe also haven't figu…

Jesus, maybe they should let the AIs run the product naming.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#673

Post by inquisitor27552 »

diggan wrote: post #9

I still feel like the difference between Sonnet and Opus is a bit unclear. Somewhere on Anthropic's website it says that Opus is the most advanced, but on other parts it says Sonnet is the most advanced and also the fastest. The UI doesn't make the distinction clear either. Then on Perplexity, Perplexity says that Opus is the most advanced, compared to Sonnet. And finally, in the table in the blogpost, Opus isn't eve…

yes it baffles they cant semver the shit out of them properly (anthtopic, meta, openai, lol)

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#674

Post by ryandvm »

sfink wrote: post #439
jimbokun wrote: post #362

Earlier quoted context omitted.

So it's behaving like our brains. Yet it's not AGI. Does that mean our brains do not implement General Intelligence?

When I take a dump, I do it exactly like a violin virtuoso does. I am not a violin virtuoso. Trust me, I'm really not.

Absolutely bonkers analogy. Ironically, I have no doubt that GPT4 could come up with a better one. What does that mean?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#675

Post by jumping_frog »

sensanaty wrote: post #670
jumping_frog wrote: post #604

Earlier quoted context omitted.

Imagine a Banking website. It has lot of predefined flows of what can be achieved. These flows have steps arranged in a dependency graph. From the server side, a llm can ask users for inputs to satisfy the dependencies of the task which user wants to accomplish. We will have intuitive UI interfaces in all languages of the world.

But if it's a predefined list of flows, you can just throw Selenium/Puppeteer/Playwright/whatever other automation tool at it, rather than relying on an unstable AI that will do different things every time. This is just a solution in search of a problem. AIs aren't reliable enough if the content changes constantly since it'll just click on the "close my account" button for no reason half the time, while a legacy webs…

I think you are confusing client side automation with server side customization (out of the box internationalization, if a user can't understand complex technical terms, llm can explain them in simple terms with examples built on the fly to illustrate the point) which was my point. I was talking about the future of UI itself.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#676

Post by distalx »

stavros wrote: post #497
dimitri-vs wrote: post #479

Earlier quoted context omitted.

Out of curiosity, how are high risk liability enviroments like yours coming to terms with the non-deterministic nature of models like these? Eg. the non-zero chance that it might click a button it *really* shouldn't as demonstrated in the failure demo.

Technical director at another company here: We have humans double-check everything, because we're required by law to. We use automation to make response times faster, or to do the bulk of the work and then just have humans double-check the AI. To do otherwise would be classed as "a software medical device", which needs documentation out the wazoo, and for good reason. I'm not sure you could even have a medical device…

I'm curious, what does your human verification process look like? Does it involve a separate interface or a generated report of some kind? I'm currently working on an tool for personal use, that records actions and triggers them at later stage on when specified event occurs. For verification, generating a CSV report after the process is complete and backing it up with screen recordings.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#678

Post by stavros »

distalx wrote: post #676
stavros wrote: post #497

Earlier quoted context omitted.

Technical director at another company here: We have humans double-check everything, because we're required by law to. We use automation to make response times faster, or to do the bulk of the work and then just have humans double-check the AI. To do otherwise would be classed as "a software medical device", which needs documentation out the wazoo, and for good reason. I'm not sure you could even have a medical device…

I'm curious, what does your human verification process look like? Does it involve a separate interface or a generated report of some kind? I'm currently working on an tool for personal use, that records actions and triggers them at later stage on when specified event occurs. For verification, generating a CSV report after the process is complete and backing it up with screen recordings.

It's a separate interface where the output of the LLM is rated for safety, and anything unsafe opens a ticket to be acted upon by the medical professionals.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#679

Post by 015a »

accrual wrote: post #178
015a wrote: post #148

Why on god's green earth is it not just called Claude 3.6 Sonnet. Or Claude 4 Sonnet. I don't actually care what the answer is. There's no answer that will make it make sense to me.

The best answer I've seen so far is that "Claude 3.5 Sonnet" is a brand name rather than a specific version. Not saying I agree, just a way to visualize how the team is coming up with marketing.

It was certainly named by some nerd: "(pushes glasses up) well, we only updated the quantized diffraction sorter, the 3.5 version number refers to iterations on both that and the field matrix array interpreter, so technically because the interpreter hasn't changed so we shouldn't upgrade the version". This engineer has never seen a dollar from a customer in their life.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#680

Post by _9omd »

LASR wrote: post #344

This is actually a huge deal. As someone building AI SaaS products, I used to have the position that directly integrating with APIs is going to get us most of the way there in terms of complete AI automation. I wanted to take at stab at this problem and started researching some daily busineses and how they use software. My brother-in-law (who is a doctor) showed me the bespoke software they use in his practice. Runni…

This is pretty similar to the argument for why humanoid robots will be a big deal. The physical world is also built mostly for humans, so having an autonomous agent that can interact with it is huge.
Post reply on HN