Hello OLMo: A truly open LLM
blog.allenai.org
Hello OLMo: A truly open LLM
1–10 of 76 posts
Re: Hello OLMo: A truly open LLM
#2This is the only LLM that is exciting to me. Clearly, LLMs are powerful tools that may end up replacing search and even go much further than simple searches by performing the research for you and producing final answers. Closed models like those from Open AI (ironically) or Anthropic cannot be audited. When most users will end up blindly hitting Microsoft’s Copilot button, which they are forcing OEMs to adopt, who’s to say how the information a user gets is being curated or manipulated by OpenAI or Microsoft or whoever?
We’ve already seen real world examples of severe bias injected into LLMs. For example, Google’s Gemini had secret meta prompts that biased it towards certain types of answers and also caused it to produce hallucinated images that were funny but also dystopian (https://arstechnica.com/information-technology/2024/02/googl...). I don’t think we can just let closed AI systems take over society when they can easily be manipulated by the model owners without transparency.
What I like about AI2’s approach with OLMo is that they are actually open, not just trading on the marketing benefits of the word “open”. Most “open” models are just open weights not open source. That’s like sharing an executable and not the source code. In my view, being open means that others have to be able to reproduce the final product (the model) if they wanted to and had the means (in terms of training hardware). It also means that they should be able to use whatever is provided freely for any purpose, rather than being subject to proprietary licensing. AI2 shares the training source code, training data, evaluation suite, and the model weights that they’ve produced by running the training process. It all uses the Apache license. And it’s also interesting that they used AMD hardware to train this LLM rather than Nvidia/CUDA.
Open weight models like Llama keep repeatedly catching up to the best closed models from OpenAI or Anthropic or others. My hope is that truly open models like OLMa keep developing quickly enough to also keep up. Lastly, I hope that regulation does not block open source private development of AI systems. These systems will be the vehicle for speech for much of society in the future, so blocking private AI systems is a lot like restricting speech. But leaving that aside, open development will also drive innovation and reducing competitive pressure will hurt innovation.
Re: Hello OLMo: A truly open LLM
#3This is the only LLM that is exciting to me. Clearly, LLMs are powerful tools that may end up replacing search and even go much further than simple searches by performing the research for you and producing final answers. Closed models like those from Open AI (ironically) or Anthropic cannot be audited. When most users will end up blindly hitting Microsoft’s Copilot button, which they are forcing OEMs to adopt, who’s…
One thing I wanted to add and call attention to is the importance of licensing in open models. This is often overlooked when we blindly accept the vague branding of models as “open”, but I am noticing that many open weight models are actually using encumbered proprietary licenses rather than standard open source licenses that are OSI approved (https://opensource.org/licenses). As an example, Databricks’s DBRX model has a proprietary license that forces adherence to their highly restrictive Acceptable Use Policy by referencing a live website hosting their AUP (https://github.com/databricks/dbrx/blob/main/LICENSE), which means as they change their AUP, you may be further restricted in the future. Meta’s Llama is similar (https://github.com/meta-llama/llama/blob/main/LICENSE ). I’m not sure who can depend on these models given this flaw.
Re: Hello OLMo: A truly open LLM
#4Has their site been hugged-to-death or is it my hotel wifi?
Re: Hello OLMo: A truly open LLM
#5Notably “The Pile” doesn’t seem to be part of the training data. So this might be more sound legally than many other “open” LLMs
Re: Hello OLMo: A truly open LLM
#6Notably “The Pile” doesn’t seem to be part of the training data. So this might be more sound legally than many other “open” LLMs
For those also wondering: https://pile.eleuther.ai
> The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together.
By what's the legal complication with it?
Re: Hello OLMo: A truly open LLM
#7Notably “The Pile” doesn’t seem to be part of the training data. So this might be more sound legally than many other “open” LLMs
For those also wondering: https://pile.eleuther.ai > The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. By what's the legal complication with it?
It received DMCA takedowns: https://en.wikipedia.org/wiki/The_Pile_(dataset)
> The Books3 component of the dataset contains copyrighted material compiled from Bibliotik, a pirate website. In July 2023, the Rights Alliance took copies of The Pile down through DMCA notices. Users responded by creating copies of The Pile with the offending content removed.
Re: Hello OLMo: A truly open LLM
#8Notably “The Pile” doesn’t seem to be part of the training data. So this might be more sound legally than many other “open” LLMs
For those also wondering: https://pile.eleuther.ai > The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. By what's the legal complication with it?
It is absolutely absolutely packed with unlicensed, copyrighted data.
Books3 is the most notable example - nearly 200,000 pirated ebooks - but a lot of the rest of it is (unlicensed) scraped web data.
The legal questions over whether this is a problem are currently still unresolved. Many people are also bothered by the ethical implications, which is a separate issue from the legal questions.
Re: Hello OLMo: A truly open LLM
#9Is this one of the first LLMs of note that was successfully trained on AMD GPUs? I wonder how seamless the process was and if they faced any issues there.
Re: Hello OLMo: A truly open LLM
#10This is the only LLM that is exciting to me. Clearly, LLMs are powerful tools that may end up replacing search and even go much further than simple searches by performing the research for you and producing final answers. Closed models like those from Open AI (ironically) or Anthropic cannot be audited. When most users will end up blindly hitting Microsoft’s Copilot button, which they are forcing OEMs to adopt, who’s…
> For example, Google’s Gemini had secret meta prompts that biased it towards certain types of answers and also caused it to produce hallucinated images that were funny but also dystopian (https://arstechnica.com/information-technology/2024/02/googl...).
Such a bizarre take to call this "dystopian".
The model happened to create some out-there pictures. I mean, it's no more outlandish then giant dragons and snakes and such being created yet the thought of a person of color being something historically inaccurate is this massive outcry against revisionism? Who cares?
Besides, the article identifies the probable goal which was to eliminate very known biases in existing models (i.e. when generating "angry person" you mainly got black people). Clearly this one wasnt tuned well for that goal, but the objective is not only noble but absolutely should be required for anyone producing LLM models.