Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hello OLMo: A truly open LLM

blog.allenai.org

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Hello OLMo: A truly open LLM

#31

Post by 23B1 »

simonw wrote: post #8
sgu999 wrote: post #6

Earlier quoted context omitted.

For those also wondering: https://pile.eleuther.ai > The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. By what's the legal complication with it?

It is absolutely absolutely packed with unlicensed, copyrighted data. Books3 is the most notable example - nearly 200,000 pirated ebooks - but a lot of the rest of it is (unlicensed) scraped web data. The legal questions over whether this is a problem are currently still unresolved. Many people are also bothered by the ethical implications, which is a separate issue from the legal questions.

Ironic that even our everyday governance has little 'Alignment' between ethics and law.

Re: Hello OLMo: A truly open LLM

#33

Post by yk »

simonw wrote: post #13
blackeyeblitzar wrote: post #2

This is the only LLM that is exciting to me. Clearly, LLMs are powerful tools that may end up replacing search and even go much further than simple searches by performing the research for you and producing final answers. Closed models like those from Open AI (ironically) or Anthropic cannot be audited. When most users will end up blindly hitting Microsoft’s Copilot button, which they are forcing OEMs to adopt, who’s…

Pet peeve: Google's Gemini LLM model was not to blame for the image generation weirdness. That would be like blaming DALL-E weirdness on GPT-4. Unfortunately, Google marketing decided to slap the "Gemini" brand on both the end-user interface used to interact with the model AND the actual model itself, hence people constantly calling out Gemini-the-model for weird decisions made as part of Gemini-the-user-interface.

Did anybody manage to get the entire prompt out of gemini, or what are you basing your claim on?

Re: Hello OLMo: A truly open LLM

#34

Post by lolinder »

whimsicalism wrote: post #29
vjeux wrote: post #25

If I read the license correctly, it seems that if you want to use the LLM, you need to tell the authors what you are doing with it. Am I reading this correctly? https://allenai.org/licenses/impact-mr “Derivative Impact Reports. AI2 seeks to encourage transparency around Derivatives through the use of Derivative Impact Reports, available here. Before releasing a Model Derivative or Data Derivative, You will share with…

no, this is apache license-d. yes it is confusing that AI2 has custom licenses but they aren't using them here

It looks like the weights [0] and code [1] are Apache licensed, but the training data [2] is using the license that OP is quoting from.

[0] https://huggingface.co/allenai/OLMo-7B

[1] https://github.com/allenai/OLMo

[2] https://huggingface.co/datasets/allenai/dolma

Re: Hello OLMo: A truly open LLM

#35

Post by kikoreis »

What does the risk classification applied to the dataset actually mean? The licensing page [1] AI2 provides for their datasets is really nice but it doesn't really explain [2] what risk means in the context.

Does it mean "risk that the items contained in this set are licensed in a manner incompatible with its use in a training dataset"?

[1] https://allenai.org/impact-license

[2] "the AI2 ImpACT Licenses are artifact-agnostic and are instead structured according to the risk level we’ve assigned a given artifact"

Re: Hello OLMo: A truly open LLM

#36

Post by 6gvONxR4sf7o »

lolinder wrote: post #34
whimsicalism wrote: post #29

Earlier quoted context omitted.

no, this is apache license-d. yes it is confusing that AI2 has custom licenses but they aren't using them here

It looks like the weights [0] and code [1] are Apache licensed, but the training data [2] is using the license that OP is quoting from. [0] https://huggingface.co/allenai/OLMo-7B [1] https://github.com/allenai/OLMo [2] https://huggingface.co/datasets/allenai/dolma

Is the license not transitive? Like could your impact report be “i want to remove this part of the license?”

Re: Hello OLMo: A truly open LLM

#37

Post by jacobn »

23B1 wrote: post #31
simonw wrote: post #8

Earlier quoted context omitted.

It is absolutely absolutely packed with unlicensed, copyrighted data. Books3 is the most notable example - nearly 200,000 pirated ebooks - but a lot of the rest of it is (unlicensed) scraped web data. The legal questions over whether this is a problem are currently still unresolved. Many people are also bothered by the ethical implications, which is a separate issue from the legal questions.

Ironic that even our everyday governance has little 'Alignment' between ethics and law.

Ethics are a lot more nuanced and change a lot faster than laws.

Heck, a large fraction of ethics seem to be so fickle that they’re subject to potential revision by every generation.

In fact, I’d argue that those revisions are a significant portion of how one generation distinguishes itself from their parents.

Yet strangely every generation feels like they have arrived at a set of “universal laws” in their ethics.

Re: Hello OLMo: A truly open LLM

#38

Post by pksebben »

It's odd. Running inference on this (and other models in its class) and I keep running into a "repeating token" situation with moderate-to-long context windows.

It feels almost as if, during inference, the model hits some format of local minimum that it careens around, and while temperature seems to affect this - it doesn't really fix it.

at temp 0.2:

> [{'generated_text': 'What follows is a transcript of a talk between a mysterious man and an agent of a bureau dedicated to investigating things which is typically referred to by some assortment of letters in the alphabet. The identity, origins, and motivations of the man were not known then and remain so. This transcript is not meant to scare, but provided simply to enlighten the concerned citizen of all the various and sundry things that may or may not go bump in the night. AGENT: Please state your name for the record. MYSTERIOUS STRANGER: I am the man. AGENT: Thank you. I am an agent of the Bureau of Investigation. I am here to investigate the following: 1. The following: 2. The following: 3. The following: 4. The following: 5. The following: 6. The following: 7. The following: 8. The following: 9. The following: 10. The following: 11. The following: 12. The following: 13. The following: 14. The following: 15. The following: 16. The following: 17. The following: 18. The following: 19. The following: 20. The following: 21. The following: 22. The following: 23. The following: 24. The following'}]

...and at temp 0.4:

> [{'generated_text': 'What follows is a transcript of a talk between a mysterious man and an agent of a bureau dedicated to investigating things which is typically referred to by some assortment of letters in the alphabet. The identity, origins, and motivations of the man were not known then and remain so. This transcript is not meant to scare, but provided simply to enlighten the concerned citizen of all the various and sundry things that may or may not go bump in the night. AGENT: Please state your name for the record. MYSTERIOUS STRANGER: My name is not important. AGENT: My name is Agent Cyanide. MYSTERIOUS STRANGER: Agent Cyanide. AGENT: I am an agent of the Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations'}]

Re: Hello OLMo: A truly open LLM

#39

Post by KarlKemp »

23B1 wrote: post #31
simonw wrote: post #8

Earlier quoted context omitted.

It is absolutely absolutely packed with unlicensed, copyrighted data. Books3 is the most notable example - nearly 200,000 pirated ebooks - but a lot of the rest of it is (unlicensed) scraped web data. The legal questions over whether this is a problem are currently still unresolved. Many people are also bothered by the ethical implications, which is a separate issue from the legal questions.

Ironic that even our everyday governance has little 'Alignment' between ethics and law.

In this case, both ethics and the law are murky.

Pretty excellent alignment, for once?

Re: Hello OLMo: A truly open LLM

#40

Post by pksebben »

pksebben wrote: post #38

It's odd. Running inference on this (and other models in its class) and I keep running into a "repeating token" situation with moderate-to-long context windows. It feels almost as if, during inference, the model hits some format of local minimum that it careens around, and while temperature seems to affect this - it doesn't really fix it. at temp 0.2: > [{'generated_text': 'What follows is a transcript of a talk betw…

... this can get a little goofy even with do_sample=False and no temp:

| [{'generated_text': "DAUGHTER: tell me a story FATHER: but it's late DAUGHTER: please? FATHER: okay, once upon a time there was a little girl who lived in a little house with her mother and father and her brother and sister and her dog and her cat and her hamster and her fish and her bird and her rabbit and her horse and her cow and her sheep and her goat and her pig and her chicken and her duck and her turkey and her goose and her llama and her alpaca and her camel and her zebra and her giraffe and her elephant and her hippopotamus and her rhinoceros and her kangaroo and her koala and her panda and her bear and her wolf and her fox and her cat and her dog and her bird and her fish and her hamster and her cat and her dog and her bird and her fish and her hamster and her cat and her dog and her bird and her fish and her hamster and her cat and her dog and her bird and her fish and her hamster and"}]

Post reply on HN