Live data from Hacker News

Claude 3 model family

anthropic.com

241–250 of 723 posts

Re: Claude 3 model family

#241
post #227

Earlier quoted context omitted.

You’re totally right, a few $B is not something any of us are bootstrapping. But there is no secret sauce (at least none that stays secret for long), no meaningful patents, no network/platform effect, and virtually no ability to lock in customers. Compare to other traditional tech companies… think Uber/AirBnB/Databricks/etc. Their product isn’t an algorithm that a competitor can spin up in 6 months. These companies c…

Every big tech in the beginning looked fragile/no moats. I think we'll see that data, knowledge and intelligence compound and at some point it will be as hard to penetrate as Meta's network effects.

Maybe consolidate as well as compound. There's a tendency for any mature industry (which may initially have been bustling with competitors) to eventually consolidate into three players, and while we're currently at the point where it seems a well-funded new entrant can catch up with the leaders, that will likely become much harder in the future as tech advances.

Never say never though - look at Tesla coming out of nowhere to push the big three automakers around! Eventually the established players become too complacent and set in their ways, creating an opening for a smaller more nimble competitor with a better idea.

I don't think LLMs are the ultimate form of AI/AGI though. Eventually we'll figure out a better brain-inspired approach that learns continually from it's own experimentation and experience. Perhaps this change of approach will be when some much smaller competitor (someone like John Carmack, perhaps) rapidly come from nowhere and catch the big three flat footed as they tend to their ginormous LLM training sets, infrastructure and entrenched products.

Re: Claude 3 model family

#242

Just added Claude 3 to Chat at https://double.bot if anyone wants to try it for coding. Free for now and will push Claude 3 for autocomplete later this afternoon. From my early tests this seems like the first API alternative to GPT4. Huge!

How do I change GPT4 to Claude 3 in double.bot?

Re: Claude 3 model family

#243
post #219

Earlier quoted context omitted.

I think that picking a direction is not that hard, and I don't know that AI couldn't do it better. I'm not sure mid-tier CEO's won't be on their way out, just like middle management.

I was talking more about science. On the people-direction side, I expect the span of control will substantially broaden, which will probably lead to fewer manager/leader jobs (that pay more). You'll always need someone to do the last 5% that it doesn't make sense to data engineer inputs/outputs into/from AI.

Yeah. Right now, its been helping me be more productive in my science by writing code quicker...mainly on the data management side of things.

I do however wonder, at what point do I just describe the hypothesis, point to the data files, and have it design an analysis pipeline, produce the results, interpret the results, then suggest potential follow-up hypotheses, do a literature search on that, then have it write up the grant for it.

Re: Claude 3 model family

#244
post #6

Wow. 1 million token length.

How did everyone solve it at the same time and there is no published paper (that I'm aware of) describing how to do it? It's like every AI researcher had an epiphany all at once

A paper describing how you might do it published in December last year. The paper was "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". To be clear I don't know if Claude and Gemini actually use this technique but I would not be surprised if they did something similar:

https://arxiv.org/abs/2312.00752

https://github.com/state-spaces/mamba

Re: Claude 3 model family

#245
The APPS benchmark result of Claude 3 Opus at 70.2% indicates it might be quite useful for coding. The dataset measures the ability to convert problem descriptions to Python code. The average length of a problem is nearly 300 words.

Interestingly, no other top models have published results on this benchmark.

Claude 3 Model Card: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb...

Table 1: Evaluation results (more datasets than in the blog post) https://twitter.com/karinanguyen_/status/1764666528220557320

APPS dataset: https://huggingface.co/datasets/codeparrot/apps

APPS dataset paper: https://arxiv.org/abs/2105.09938v3

Re: Claude 3 model family

#246

I just tried one prompt for a simple coding task involving DB and frontend, and Claude 3 Sonnet (the free and less powerful model) gave a better response than ChatGPT Classic (GPT-4). It used the correct method of a lesser-known SQL ORM library, where GPT-4 made a mistake and used the wrong method. Then I tried another prompt to generate SQL and it gave a worse response than ChatGPT Classic, still looks correct but m…

Are you aware you're using GPT-3 or weaker in those chats? The green icon indicates that you're using the first generation of ChatGPT models, and it is likely to be GPT-3.5 Turbo. I'm unsure but it's possible that it's an even further distilled or quantized optimization than is available via API.

Using GPT-4, I get the result I think you'd expect: https://chat.openai.com/share/da15f295-9c65-4aaf-9523-601bf4...

This is a good PSA that a lot of content out on the internet showing ChatGPT getting things wrong is the weaker model.

Green background OpenAI icon: GPT 3.5

Black or purple icon: GPT 4

GPT-4 Turbo, via API, did slightly better though perhaps just because it has more Drizzle knowledge in the training set, and skips the SQL command and instead suggests modifying only db.ts and page.tsx.

Re: Claude 3 model family

#247

Earlier quoted context omitted.

So double is like copilot, but free? What's the catch?

I guess your data is the catch.

Probably not data so much as growth numbers to appease investors. Such offerings typically don’t last forever. Might as well take advantage while it lasts.

Re: Claude 3 model family

#249

Earlier quoted context omitted.

Hey Wesley, I just checked Double. Do you plan to support open source models hosted locally or on a cloud instance? Asking out of curiosity as I am building a product in the same space and have had a few people ask this. I guess since Double is an extension in IDEs, it can connect to AI models running anywhere.

it's an interesting idea. We asked our users this as well but at least for those we talked to, running their own model wasn't a big priority. What actually mattered to them is being able to try different (but high performance) models, privacy (their code not being trained on), and latency. We have some optimizations around time-to-first-token latency that would be difficult to do if we didn't have information about t…

I see. Thanks Wesley for sharing and great to know it is not a priority. Also, the Mistral situation kinda makes me feel that big corps will want to host models.

Although, I feel Apple will break this trend and bring models to their chips rather than run them on the cloud. "Privacy first" will simply be a selling point for them but generally speaking cloud is not a big sell for them.

I am not at the level to do much optimizations, plus my product is a little more generic. To get to MVP, prompt engineering will probably be my sole focus.

Re: Claude 3 model family

#250

Just signed up for Claude Pro to try out the Opus model. Decided to throw a complex query at it, combining an image with an involved question about SDXL fine tuning and asking it to do some math comparing the cost of using an RTX 6000 Ada vs an H100. It made a lot of mistakes. I provided it with a screenshot of Runpod's pricing for their GPUs, and it misread the pricing on an RTX 6000 ADA as $0.114 instead of $1.14.…

How many uses do you get per day of Opus with the pro subscription?

Hmm, not seeing it anywhere on my profile or in the chat interface, but I might be missing it.
Post reply on HN