51–60 of 76 posts
> 1. No biases. Following LLaMA, PaLM, and others, we exclude all bias terms from our architecture in order to improve training stability. What does this mean? What is a "bias term"?
It's odd. Running inference on this (and other models in its class) and I keep running into a "repeating token" situation with moderate-to-long context windows. It feels almost as if, during inference, the model hits some format of local minimum that it careens around, and while temperature seems to affect this - it doesn't really fix it. at temp 0.2: > [{'generated_text': 'What follows is a transcript of a talk betw…
Too bad they did not put any comparison tables into the blog post.
They're on Hugging Face. Interestingly enough they don't compare it against Mistral 7B. https://huggingface.co/allenai/OLMo-7B
Earlier quoted context omitted.
It looks like the weights [0] and code [1] are Apache licensed, but the training data [2] is using the license that OP is quoting from. [0] https://huggingface.co/allenai/OLMo-7B [1] https://github.com/allenai/OLMo [2] https://huggingface.co/datasets/allenai/dolma
Is the license not transitive? Like could your impact report be “i want to remove this part of the license?”
Earlier quoted context omitted.
... this can get a little goofy even with do_sample=False and no temp: | [{'generated_text': "DAUGHTER: tell me a story FATHER: but it's late DAUGHTER: please? FATHER: okay, once upon a time there was a little girl who lived in a little house with her mother and father and her brother and sister and her dog and her cat and her hamster and her fish and her bird and her rabbit and her horse and her cow and her sheep an…
That's seems a perfect story to put a little child to bed :D. I have used a similar recursive story in the past. My son still jokes about it.
Basically one generated story per day. Absurd in places.
The hype around LLMs won't last past 2030 I suppose. LLMs - we have statistical inference soup that gets outdated like stagnant pond water and by each passing day, becoming less accurate. I am curious how long the hype wave lasts. Ones I have recently seen was K8S. It settled down and won TBH.
The hype around LLMs won't last past 2030 I suppose. LLMs - we have statistical inference soup that gets outdated like stagnant pond water and by each passing day, becoming less accurate. I am curious how long the hype wave lasts. Ones I have recently seen was K8S. It settled down and won TBH.
This is the only LLM that is exciting to me. Clearly, LLMs are powerful tools that may end up replacing search and even go much further than simple searches by performing the research for you and producing final answers. Closed models like those from Open AI (ironically) or Anthropic cannot be audited. When most users will end up blindly hitting Microsoft’s Copilot button, which they are forcing OEMs to adopt, who’s…
Pet peeve: Google's Gemini LLM model was not to blame for the image generation weirdness. That would be like blaming DALL-E weirdness on GPT-4. Unfortunately, Google marketing decided to slap the "Gemini" brand on both the end-user interface used to interact with the model AND the actual model itself, hence people constantly calling out Gemini-the-model for weird decisions made as part of Gemini-the-user-interface.
The way I read the Gemini technical report, it seemed like, unlike GPT-4 vs DALL-E, Gemini was pretrained with multimodal outputs. Is that not the case?
Earlier quoted context omitted.
> For example, Google’s Gemini had secret meta prompts that biased it towards certain types of answers and also caused it to produce hallucinated images that were funny but also dystopian ( https://arstechnica.com/information-technology/2024/02/googl ...). Such a bizarre take to call this "dystopian". The model happened to create some out-there pictures. I mean, it's no more outlandish then giant dragons and snakes a…
Right, "who cares" about the truth in our dystopian world? 1984 is apparently too long ago for people to remember the ministry of truth...