Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

github.com

Post reply on HN
Print view View original

71–80 of 158 posts

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#71

Post by narrator »

thorum wrote: post #32

o1’s innovation is not Chain-of-Thought. It’s teaching the model to do CoT well (from massive amounts of human feedback) instead of just pretending to. You’ll never get o1 performance just from prompt engineering.

If the training data for these LLMs is from humanity in general, and it is trying to imitate humanity, wouldn't its IQ tend to be the average of all of humanity? Perhaps the only people who talk about STEM topics are people of higher IQ generally, including a lot of poor students asking homework questions. Thus, the way to get to higher IQ output is to critique the lower IQ answers, which may be more numerous by rejecting their flaws in favor of the higher IQ answers. That, or just training more heavily on textbooks, and so forth. How to reject errors, and maybe train on synthetic data generated without reasoning with errors.

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#72

Post by undecisive »

arnaudsm wrote: post #64
punnerud wrote: post #62

I changed it into running 100% locally with ollama:8b: https://github.com/punnerud/g1 Not updated the Readme yet

You should also try phi-3-small 7B, seems much better at reasoning according to https://livebench.ai

I just tried it with phi3.5:3.8b-mini-instruct-fp16 - it didn't work with the base question, though interestingly the reasoning decided that strawberry was spelt s-t-r-a-w-b-e-r - which explains why the AIs have such a hard time with this question. I also tried it with my current favourite programming question too - What programming language is this whole line of code using? `def obfuscated_fibonacci(x)` - and like all the AIs, it was convinced the answer was python (the correct answer is ruby - python needs a trailing colon - but most LLMs will swear blind that it's python). It didn't even consider ruby as a possibility. Nobody uses ruby anyway :D

Thanks for the fork and the suggestions though - looks like I'll be having fun with this over the week!

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#73

Post by punnerud »

arnaudsm wrote: post #64
punnerud wrote: post #62

I changed it into running 100% locally with ollama:8b: https://github.com/punnerud/g1 Not updated the Readme yet

You should also try phi-3-small 7B, seems much better at reasoning according to https://livebench.ai

Worked, bud did not see a great improvement over llama:8b

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#74

Post by qeternity »

astrange wrote: post #63
nell wrote: post #59

Earlier quoted context omitted.

> I wouldn't call o1 a "system". It's a model, but unlike previous models, it's trained to generate a very long chain of thought before returning a final answer https://x.com/polynoamial/status/1834641202215297487

That answer seems to conflict with "in the future we'd like to give users more control over the thinking time". I've gotten mini to think harder by asking it to, but it didn't make a better answer. Though now I've run out of usage limits for both of them so can't try any more…

I'm not convinced there isn't more going on behind the scenes but influencing test-time compute via prompt is a pretty universal capability.

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#75

Post by aktuel »

Let's just assume for a moment that the hype is real and that these LLMs are incredibly intelligent and will replace us all soon. Then the model shouldn't be any less intelligent if we remove facts like Uma Thurman's measurements and other vapid information. If the model already has the capability to use tools than all of that crap is redundant anyway. And while we are at it let's remove a ton of other junk like languages I will never use and which also doesn't make the model any smarter. So how small can this kernel get while still being clearly intelligent, able to communicate flawlessly in english and apply logical reasoning. That would be a worthwile endeavor and maybe even possible without boiling the oceans.

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#76

Post by anonzzzies »

astrange wrote: post #67
4ad wrote: post #12

This is the system prompt it uses: You are an expert AI assistant that explains your reasoning step by step. For each step, provide a title that describes what you're doing in that step, along with the content. Decide if you need another step or if you're ready to give the final answer. Respond in JSON format with 'title', 'content', and 'next_action' (either 'continue' or 'final_answer') keys. USE AS MANY REASONING…

That second prompt is interesting. Not magic though. I tried it with every other model I know and they're still basically unable to do: * give me three sentences that end in "is" * tell me the line of Star Spangled Banner that comes before "gave proof through the night" But they did some good thinking before failing at it…

> Not magic though

It's just a pile on of trial and error instructions (maybe learned from previous 'projects', but). There is no magic or skill to prompt 'engineering' anywhere.

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#77

Post by visarga »

thorum wrote: post #32

o1’s innovation is not Chain-of-Thought. It’s teaching the model to do CoT well (from massive amounts of human feedback) instead of just pretending to. You’ll never get o1 performance just from prompt engineering.

> from massive amounts of human feedback

It might be the 200M user base of OpenAI that provided the necessary guidance for advanced CoT, implicitly. Every user chat session is also an opportunity for the model to get feedback and elicit experience from the user.

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#78

Post by wkat4242 »

astrange wrote: post #65
wkat4242 wrote: post #22

Earlier quoted context omitted.

Yeah and some of the other prompts were misspelled and of doubtful use: > In order to make the draft response nicer and complete, a set of question [sic] and its answer are provided," reads one prompt. "Please write a concise and natural reply by modify [sic] the draft response," it continues. This really sounds like a placeholder made up by one engineer until a more qualified team sits down and defines it.

That's not a big problem since it will understand it, and if they already fine tuned the model to work with that prompt it'd get harder to change.

I just don't think Apple would release something like this. They're the company that laser engraves their screws because of their attention to detail.

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#79

Post by kkzz99 »

zozbot234 wrote: post #70

How does this benchmark against Reflection, which was fine-tuned to do the same thing-- provide a detailed Chain of Thought with self-corrections, then write out a final answer?

Pretty sure Reflection-70B was a complete scam. They did the ole bait and switch. The model that they uploaded was completely under-performing compared to their own benchmarks and the "secret API" was just a GPT-4 & Claude wrapper.

Re: g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains

#80

Post by cubefox »

segmondy wrote: post #30

This is not even remotely close and very silly. A ChainOfThought in a loop. TreeOfThoughts is a more sophisticated method, see - https://arxiv.org/pdf/2305.10601 The clue we all had with OpenAI for a long time that this was a search through a tree, they hired Noam Brown, and his past work all hinted towards that. Q , is obviously a search on a tree like A . So take something like CoT, build out a tree, search for the…

It's interesting that DeepMind still publishes this stuff. OpenAI doesn't publish anything of that sort anymore. DeepMind is more research/publication focused, but this is a disadvantage in a competitive landscape where OpenAI and Anthropic can just apply the results of your paper without giving anything back to the research community.
Post reply on HN