Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 3 model family

anthropic.com

Post reply on HN
Print view View original

661–670 of 723 posts

Re: Claude 3 model family

#661

Post by MeImCounting »

auggierose wrote: post #502
yaj54 wrote: post #455

Earlier quoted context omitted.

GPT4 API and ChatGPT both get it wrong: "Sally has 2 sisters. Each of her brothers has 2 sisters because Sally and her 2 sisters make up the group of siblings each brother has." GPT4 w/ CoT system prompting gets it right: SYS: "You are a helpful assistant. Think through your work step by step before providing your answer." USER: "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally h…

Is w/ supposed to mean "with". And if so, why not just write "with"? Also, I guess CoT means Chain of Thought?

Why not just write you are instead of youre? Why shorten any word ever? Yes CoT is chain of thought

Re: Claude 3 model family

#662

Post by BoorishBears »

renewiltord wrote: post #660
BoorishBears wrote: post #614

Earlier quoted context omitted.

Why did you bait the model into a disclaimer with that comical wording?

To determine its ability to disambiguate. And to test its sensitivity on safety. Even Mixtral 8x7b running on my laptop correctly disambiguates. That test is a heuristic for how likely I am to encounter the mistaken safety procedure.

I guess if you want to deny yourself top of the line capabilities because you can't ask it poorly worded questions or might occasionally need to clarify intent for the model that's a fair strategy.

I'm in the camp that this safety pearl clutching is overblown in both directions: it's embarrassingly easy to overcome their disclaimers.

Re: Claude 3 model family

#663

Post by CamperBob2 »

montecarl wrote: post #348
nopinsight wrote: post #283

Earlier quoted context omitted.

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitte…

I really wanted to read the questions, but they make it hard because they don't want the plaintext to be visible on the internet. Below is a link toa python script I wrote, that downloads the password protected zip and creates a decently formatted html document with all the questions and answers. Should only require python3. Pipe the output to a file of your choice. https://pastebin.com/REV5ezhv

Thanks for that. I did have to append .encode("utf-8") to the strings in the print statements before it would let me pipe the output to an .htm file under Windows, but other than that it worked great.

(Edit: better to use the original script but set PYTHONUTF8=1 before running)

Re: Claude 3 model family

#664

Post by arendtio »

eliya_confiant wrote: post #224
simonw wrote: post #189

I just released a plugin for my LLM command-line tool that adds support for the new Claude 3 models: pipx install llm llm install llm-claude-3 llm keys set claude # paste Anthropic API key here llm -m claude-3-opus '3 fun facts about pelicans' llm -m claude-3-opus '3 surprising facts about walruses' Code here: https://github.com/simonw/llm-claude-3 More on LLM: https://llm.datasette.io/

Hi Simon, Big fan of your work with the LLM tool. I have a cool use for it that I wanted to share with you (on mac). First, I created a quick action in Automator that recieves text. Then I put together this script with the help of ChaptGPT: escaped_args="" for arg in "$@"; do escaped_arg=$(printf '%s\n' "$arg" | sed "s/'/'\\\\''/g") escaped_args="$escaped_args '$escaped_arg'" done result=$(/Users/XXXX/Library/Python/…

Does someone know a similar tool like Automator on Linux for this particular use case?

Re: Claude 3 model family

#665

Post by fragmede »

johnfn wrote: post #600
CorpOverreach wrote: post #583

This part continues to bug me in ways that I can't seem to find the right expression for: > Previous Claude models often made unnecessary refusals that suggested a lack of contextual understanding. We’ve made meaningful progress in this area: Opus, Sonnet, and Haiku are significantly less likely to refuse to answer prompts that border on the system’s guardrails than previous generations of models. As shown below, the…

It's not about you. It's about Joe Drugdealer who wants to use it to learn how to make meth, or do other nefarious things.

Why is the knowledge on how to make meth the most dangerous knowledge you can think of? The difficulty in making meth is that, due to the war on drugs, the chemical precursors, specifically methylamine, are illegal and hard to procure as an ordinary citizen. This was popularized by the show Breaking Bad but as far as I've read, is actually true. It seems there would be other bits of knowledge/ideas that would be more poisonous that corporations don't want to promulgate. Ideas like the Jews secretly control everything or that white people are better, are probably not views that corporations or society want an LLM to reinforce and radicalize people into believing, among others.

Re: Claude 3 model family

#666

Post by Nevermark »

littlestymaar wrote: post #626
Nevermark wrote: post #607

Earlier quoted context omitted.

In the sense that less people are needed to do many kinds of work, they chat AI’s are now reducing people. Which is not quite the same as replacing them.

It's not even sure it will reduce the workforce for all of the aforementioned jobs: it's making the same amount of work cost less so it can also increase the demand for the said work to the point it is actually increasing the amount of workers. Like how github and npm increased the developers' productivity so much it drove the developer market up.

Most jobs have a limited demand. Because internal jobs are not the same as products in the marketplace.

Products and services typically require a mix of many kinds of internal parts or tasks to be created or supplied. Most of them are not the majority cost drivers.

You don’t increase the amount of software created by responding to cheaper documentation by increasing the documentation to keep your staff busy, or hiring more document staff, to create even more of the cheaper documentation.

You hire fewer documentation people and shift resources elsewhere.

Making one tasks easier is more likely to reduce internal demand for employees in that area. Very unlikely to somehow increase demand for it.

Unless all tasks get cheaper, or the task is a majority cost driver, and directly spills into obviously lower prices for customers for the product or service.

Re: Claude 3 model family

#667

Post by CamperBob2 »

lukev wrote: post #313
nopinsight wrote: post #283

Earlier quoted context omitted.

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitte…

This doesn't pass the sniff test for me. Not sure if these models are memorizing the answers or something else, but it's simply not the case that they're as capable as a domain expert (yet.) I do not have a PhD, but in areas I do have expertise, you really don't have to push these models that hard to before they start to break down and emit incomplete or wrong analysis.

The idea is that they aren't as capable as a domain expert, but they are more capable than an expert from a different domain.

E.g., if you ask a chemistry PhD something about genetics or astrophysics, you're more likely to get a correct answer from the model. Which is pretty interesting IMO.

Re: Claude 3 model family

#668

Post by airstrike »

giantrobot wrote: post #643
smokel wrote: post #484

Earlier quoted context omitted.

> These things are supposed to have PhD level smarts Whoever told you that?

Anthropic's marketing claiming high scores on supposed intelligence measurements.

Having a PhD is not a requirement for being intelligent

Re: Claude 3 model family

#669

Post by yaj54 »

furyofantares wrote: post #658
dweekly wrote: post #599

Earlier quoted context omitted.

What are your custom instructions?

Well, that's a little embarrassing. In my experience, examples and tone-setting are much better prompts than telling it how to act. Completion-based prompting over instruct-based prompting. With that out of the way, I'll fess up to my stupid custom instructions: > Fuck yeah! Let's fuckin' go! Let's write all the code and solve all the things. > Let's speak directly and make creative guesses! Hell yea motherfucker let…

gold

Re: Claude 3 model family

#670

Post by hmottestad »

hmottestad wrote: post #646
woadwarrior01 wrote: post #495

Earlier quoted context omitted.

Yeah Mixtral is between GPT3.5 and GPT4 in perf. Better than 3.5, but trailing behind 4. > One weakness of Mixtral for me is its support for Norwegian. I recently added grammar correction and summarization feature to my app (which uses different system prompts based on the language). And one of the Norwegian speaking users on discord told me the same thing. He also told me that the smaller Nous-Hermes-2-SOLAR-10.7B s…

Thanks for the heads up :) I will try it out!

Tried it out a bit this evening and I must say that I’m astounded. I asked it to summarise some news articles in a list with 5 bullet points and it did an amazing job. I’m sure GPT4 is better, but this is more than good enough and leagues ahead of the other models I’ve tried locally. Thanks again for the tip!
Post reply on HN