Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llemma: An Open Language Model for Mathematics

arxiv.org

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Llemma: An Open Language Model for Mathematics

#41

Post by schoen »

jychang wrote: post #36
aSanchezStern wrote: post #26

Earlier quoted context omitted.

Oh, I think you might misunderstand what I'm comparing it to. The other tools, like Proverbot9001, are exactly the NNUE scenario you describe, where a small neural network guides a search procedure to find proofs; they are more effective at finding formal proofs than Llemma. For other tasks, like non-formal proof generation, Llemma has novel results as far as I know; it's just in terms of producing formal proofs that…

When do you think we’ll have something that can do “verify this proof of the ABC conjecture” and it would check the proof?

If the proof is written in a formal language, we have that now! Even several directly competing efforts (for better and worse).

https://www.cs.ru.nl/~freek/100/

Some people greatly hope that fully formal proofs become a routine part of math research and communication in the future.

Re: Llemma: An Open Language Model for Mathematics

#44

Post by nxobject »

classified wrote: post #42

So we get hallucinations in math, too. This was inevitable.

For what it's worth, I hallucinated a hell of a lot on paper when taking 4xx-level math classes as an undergrad. My homework was marked with so many "no"s in red ink... although I got away with some handwaving.

Re: Llemma: An Open Language Model for Mathematics

#46

Post by pk-protect-ai »

>> Before training L LEMMA 7B, we contract the RoPE (Su et al., 2022) base period of the Code Llama 7B initialization from θ = 1, 000, 000 to θ = 10, 000. This is so that the long context finetuning procedure described in Peng et al. (2023)and Rozière et al. (2023) can be repeated on the trained LLEMMA 7B (we leave actually doing so to future work).

This does not explain what was the reason for RoPE contraction in the first place. Can anyone elaborate what could be the reason?

Re: Llemma: An Open Language Model for Mathematics

#47

Post by tempusalaria »

pk-protect-ai wrote: post #46

>> Before training L LEMMA 7B, we contract the RoPE (Su et al., 2022) base period of the Code Llama 7B initialization from θ = 1, 000, 000 to θ = 10, 000. This is so that the long context finetuning procedure described in Peng et al. (2023)and Rozière et al. (2023) can be repeated on the trained LLEMMA 7B (we leave actually doing so to future work). This does not explain what was the reason for RoPE contraction in th…

In the absence of a very strong justification, my assumption for any random technique in an AI paper is that they tried a bunch of different things and whatever gave the highest evals made it into the paper (even though that performance is likely random not a genuine improvement)

Re: Llemma: An Open Language Model for Mathematics

#48

Post by YetAnotherNick »

WizardMath[1] clearly has better benchmark results and they conveniently ignored it. They knew about the model because they have used their dataset.

This has 51.5% in GSM8K compared to 81.6% for WizardMath.

[1]: https://huggingface.co/WizardLM/WizardMath-70B-V1.0

Re: Llemma: An Open Language Model for Mathematics

#49

Post by joaogui1 »

YetAnotherNick wrote: post #48

WizardMath[1] clearly has better benchmark results and they conveniently ignored it. They knew about the model because they have used their dataset. This has 51.5% in GSM8K compared to 81.6% for WizardMath. [1]: https://huggingface.co/WizardLM/WizardMath-70B-V1.0

One of the main points of Llemma is being an open-source reproduction of Minerva, so basically that's the only comparison they "need" to make. Besides, isn't WizardMath trained on GPT-4 output? They may want to be comparing only among models that can be used commercially

Re: Llemma: An Open Language Model for Mathematics

#50

Post by Zuiii »

keirp1 wrote: post #37
Zuiii wrote: post #35

This seems to be as open as proprietary game engines as no mention of model license was present in the paper, code, model card. I wonder whether their incorrect and misleading use of "open" will be caught during peer review. Maybe the authors also believe that Windows is open. Its source code is also publicly available after all.

The license is in the model card at the top - it's the same as the Llama 2 license.

> The license is in the model card at the top - it's the same as the Llama 2 license.

So the license is indeed proprietary. I hope peer review corrects the authors' misuse of the term.

Post reply on HN