Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

211–220 of 723 posts

Re: Claude 3 model family

#211

Post by FergusArgyll »

vermorel wrote: post #58
ethbr1 wrote: post #31

Earlier quoted context omitted.

How would that work technically, from a cost of goods sold perspective? (honestly asking, curious)

The "cost" is storing the state of the LLM after processing the input. My back-of-the-envelop guesstimate gives me 1GB to capture the 8bit state of 70B parameters model (I might be wrong though, insights are welcome), which is quite manageable with NVMe storage for fast reload. The operator would charge per pay per "saved" prompt, plus maybe a fix per call fee to re-load the state.

That's a great idea! It would also open up the possibility for very long 'system prompts' on the side of the company, so they could better fine-tune their guardrails

Re: Claude 3 model family

#212

Post by HarHarVeryFunny »

bugglebeetle wrote: post #196
hobofan wrote: post #95

Earlier quoted context omitted.

How so? Because they have taken large investments from Amazon and Google? Or would you also characterize OpenAI as "Microsoft in a trench coat"?

100% OpenAI is Microsoft in a trenchcoat.

They are funded mostly by Microsoft, and dependent on them for compute (which is what this funding is mostly buying), but I'd hardly characterize that as meaning they are "Microsoft in a trenchcoat". It's not normal to identify startups as being their "VC in a trenchcoat", even if they are dependent on the money for growth.

Re: Claude 3 model family

#213

Post by sigmoid10 »

uoaei wrote: post #136
Workaccount2 wrote: post #125

Earlier quoted context omitted.

I'd almost say anyone not doing that is being foolish. The goal of the service is to answer complex queries correctly, not to have a pure LLM that can do it all. I think some engineers feel that if they are leaning on an old school classically programed tool to assist the LLM, it's somehow cheating or impure.

Of course. But we must acknowledge that many have blinders on, assuming that scale is all you need to beat statistical errors.

Well, these people are not wrong per se. Scale is what drove what we have today and as hardware improves, the models will too. It's just that in the very short term it turns out to be faster to just code around some of these issues on the backend of an API rather than increase the compute you spend on the model itself.

Re: Claude 3 model family

#215

Post by appplication »

zurfer wrote: post #168
appplication wrote: post #143

Earlier quoted context omitted.

What this really says to me is the indefensibility of any current advances. There’s really cool stuff going on right now, but anyone can do it. Not to say anyone can push the limits of research, but once the cat’s out of the bag, anyone with a few $B and dozen engineers can replicate a model that’s indistinguishably good from best in class to most users.

Barrier to entry with "few $B" is pretty high. Especially since the scaling laws indicate that it's only getting more expensive. And even if you manage to raise $Bs, you still need to be clever on how to deploy it (talent, compute, data) ...

You’re totally right, a few $B is not something any of us are bootstrapping. But there is no secret sauce (at least none that stays secret for long), no meaningful patents, no network/platform effect, and virtually no ability to lock in customers.

Compare to other traditional tech companies… think Uber/AirBnB/Databricks/etc. Their product isn’t an algorithm that a competitor can spin up in 6 months. These companies create real moats, for better or worse, which significantly reduce the ability for competitors to enter, even with tranches of cash.

In contrast, essentially every product we’ve seen in the AI space is very replicable, and any differentiation is largely marginal, under the hood, and the details of which are obscured from customers.

Re: Claude 3 model family

#217

Post by lmeyerov »

vermorel wrote: post #28

Does any of those LLM-as-a-service companies provide a mechanism to "save" a given input? Paying only for the state storage and the extra input when continuing the completion from the snapshot? Indeed, at 1M token and $15/M tokens, we are talking of $10+ API calls (per call) when maxing out the LLM capacity. I see plenty of use cases for such a big context, but re-paying, at every API call, to re-submit the exact sam…

Yes: That's essentially their fine-tuning offerings. They rewrite some weights in the top layers based on your input, and save+serve that for you.

It sounds like you would like a wrapped version tuned just for big context.

(As others write, RAG versions are also being supported, but they're less fundamentally similar. RAG is about preprocessing to cut the input down to relevant bits. RAG + an agent framework does get closer again tho by putting this into a reasoning loop.)

Re: Claude 3 model family

#218

Post by tempusalaria »

skepticATX wrote: post #24

The results really aren’t striking enough that it’s clear that this model blows GPT-4 away. It seems roughly equivalent, give or take a bit. Why can we still not easily surpass a (relatively) ancient model?

Once you’ve taken all the data in the world and trained a sufficiently large model on it, it’s very hard to improve on that base. It’s possible that GPT-4 basically represents that benchmark, and improvements will require better parsing/tokenization, clever synthetic data methods, building expert datasets. Much harder than just scraping the internet and doing next token after some basic data cleaning.

Re: Claude 3 model family

#219

Post by ethbr1 »

SubiculumCode wrote: post #147
ethbr1 wrote: post #142

Earlier quoted context omitted.

I have a suspicion that greenfield science will be the last thing automated, at least the non-brute-force kind. AI assistants to do the drugery (smart search agents), but not pick the directions to proceed in. Too little relevant training data in niche, state of the art topics. But to the broader point, isn't this progress in a nutshell? (1) Figure out a thing can be done, (2) figure out how to manufacture with human…

I think that picking a direction is not that hard, and I don't know that AI couldn't do it better. I'm not sure mid-tier CEO's won't be on their way out, just like middle management.

I was talking more about science.

On the people-direction side, I expect the span of control will substantially broaden, which will probably lead to fewer manager/leader jobs (that pay more).

You'll always need someone to do the last 5% that it doesn't make sense to data engineer inputs/outputs into/from AI.

Re: Claude 3 model family

#220

Post by tempusalaria »

FergusArgyll wrote: post #216
sidcool wrote: post #6

Wow. 1 million token length.

How did everyone solve it at the same time and there is no published paper (that I'm aware of) describing how to do it? It's like every AI researcher had an epiphany all at once

Firms are hiring from each other all the time. Plus there’s the fact that the base pertaining is being done at higher context lengths, so then the context extending fine tuning is working from a larger base
Post reply on HN