Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Olmo 3: Charting a path through the model flow to lead open-source AI

allenai.org

Post reply on HN
Print view View original

71–80 of 135 posts

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#71

Post by observationist »

latexr wrote: post #51
embedding-shape wrote: post #34

Earlier quoted context omitted.

How many times did you retry (so it's not just up to chance), what were the parameters, specifically for temperature and top_p?

> How many times did you retry (so it's not just up to chance) If you don’t know the answer to a question, retrying multiple times only serves to amplify your bias, you have no basis to know the answer is correct.

https://en.wikipedia.org/wiki/Monte_Carlo_method

If it's out of distribution, you're more likely to get a chaotic distribution around the answer to a question, whereas if it's just not known well, you'll get a normal distribution, with a flatter slope the less well modeled a concept is.

There are all sorts of techniques and methods you can use to get a probabilistically valid assessment of outputs from LLMs, they're just expensive and/or tedious.

Repeated sampling gives you the basis to make a Bayesian model of the output, and you can even work out rigorous numbers specific to the model and your prompt framework by sampling things you know the model has in distribution and comparing the curves against your test case, giving you a measure of relative certainty.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#72

Post by comp_raccoon »

weregiraffe wrote: post #69

Is the training data open-source? And can you validate that the model was trained on the claimed training data alone? Without this, all benchmarks are useless.

Olmo author here! we release all training data and all our training scripts, plus intermediate checkpoints, so you could take a checkpoint, reproduce a few steps on the training data, and check if loss matches.

it’s no cryptography proof, and you can’t get perfect determinism on nvidia GPUs, but it’s pretty close.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#73

Post by lukan »

turnsout wrote: post #59

I used the Ai2 Playground and Olmo 3 32GB Think, and asked it to recommend a language for a green-field web app based on a list of criteria. It gave me a very good and well-reasoned answer (Go, with Rust as a backup), formatted like a high-quality ChatGPT or Claude response. I then had it show the "OlmoTrace" for its response, which seems like it finds exact matches for text strings in its training data that end up i…

"What could be interesting is if the user could dynamically exclude certain training sources before the response is generated."

Yes and also add new ressources on the fly. Unfortunately that requires retraining every time you do, so not really possible, but if you find a way .. I guess many will be interested.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#74

Post by comp_raccoon »

thot_experiment wrote: post #11

Qwen3-30B-VL is going to be fucking hard to beat as a daily driver, it's so good for the base 80% of tasks I want an AI for, and holy fuck is it fast. 90tok/s on my machine, I pretty much keep it in vram permanently. I think this sort of work is important and I'm really glad it's being done, but in terms of something I want to use every day there's no way a dense model can compete unless it's smart as fuck. Even dumb…

Olmo author here! Qwenmodels are in general amazing, but 30B is v fast cuz it’s an MoE. MoEs very much on the roadmap for next Olmo.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#75

Post by krzysiek »

dotancohen wrote: post #12
krzysiek wrote: post #5

To see OlmoTrace go to https://playground.allenai.org/ and after you get the responce to your prompt, click the secod icon from the right on the top (at least on mobile). It took me a while to find it because there is another icon under the responce that looks the same but does something else.

In the 35 minute since you've posted, maybe the interface has changed. Upon scrolling to the bottom of the answer I got a popup asking "Curious about how this response matches the model's training data? Click this to dig deeper." Clicking it did nothing. When I check the top of the answer I see a speech bubble icon and a downarrow icon. Clicking either opens the thought process. This is on Firefox running on Android.

It didn't change for me (Also on Firefox/Android). I meant at the top of the screen in the menu section. An icon that resembles a piece of paper. On desktop it explicitly says "Show Olmo Trace" and both "piece of paper" icons work.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#76

Post by fnbr »

littlestymaar wrote: post #64
fnbr wrote: post #50

Earlier quoted context omitted.

(I’m a researcher on the post-training team at Ai2.) 7B models are mostly useful for local use on consumer GPUs. 32B could be used for a lot of applications. There’s a lot of companies using fine tuned Qwen 3 models that might want to switch to Olmo now that we have released a 32B base model.

May I ask why you went for a 7B and a 32B dense models instead of a small MoE like Qwen3-30B-A3B or gpt-oss-20b given how successful these MoE experiments were?

MoEs have a lot of technical complexity and aren't well supported in the open source world. We plan to release a MoE soon(ish).

I do think that MoEs are clearly the future. I think we will release more MoEs moving forward once we have the tech in place to do so efficiently. For all use cases except local usage, I think that MoEs are clearly superior to dense models.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#77

Post by comp_raccoon »

andai wrote: post #60
thot_experiment wrote: post #11

Qwen3-30B-VL is going to be fucking hard to beat as a daily driver, it's so good for the base 80% of tasks I want an AI for, and holy fuck is it fast. 90tok/s on my machine, I pretty much keep it in vram permanently. I think this sort of work is important and I'm really glad it's being done, but in terms of something I want to use every day there's no way a dense model can compete unless it's smart as fuck. Even dumb…

I'm out of the loop... so Qwen3-30B-VL is smart and Qwen3-30B is dumb... and that has to do not with the size but architecture?

Olmo author here, but I can help! First release of Qwen 3 left a lot of performance on the table bc they had some challenges balancing thinking and non-thinking modes. VL series has refreshed posttrain, so they are much better!

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#78

Post by comp_raccoon »

stavros wrote: post #2

> the best fully open 32B-scale thinking model It's absolutely fantastic that they're releasing an actually OSS model, but isn't "the best fully open" a bit of a low bar? I'm not aware of any other fully open models.

Olmo author here… would be nice to have some more competition!! I don’t like that we are so lonely either.

We are competitive with open weights models in general, just a couple points behind best Qwen.

Fully open models are important for research community; a lot of fundamental discoveries are made when you have access to training data. We call out we are the best fully open model because researchers would want to know about that.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#79

Post by riazrizvi »

littlestymaar wrote: post #64
fnbr wrote: post #50

Earlier quoted context omitted.

(I’m a researcher on the post-training team at Ai2.) 7B models are mostly useful for local use on consumer GPUs. 32B could be used for a lot of applications. There’s a lot of companies using fine tuned Qwen 3 models that might want to switch to Olmo now that we have released a 32B base model.

May I ask why you went for a 7B and a 32B dense models instead of a small MoE like Qwen3-30B-A3B or gpt-oss-20b given how successful these MoE experiments were?

7B runs on my Intel Macbook Pro - there is a broad practical application served here for developers who need to figure out a project on their own hardware, which improves time/cost/effort economy. Before committing to a bigger model for the same project.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#80

Post by stavros »

comp_raccoon wrote: post #78
stavros wrote: post #2

> the best fully open 32B-scale thinking model It's absolutely fantastic that they're releasing an actually OSS model, but isn't "the best fully open" a bit of a low bar? I'm not aware of any other fully open models.

Olmo author here… would be nice to have some more competition!! I don’t like that we are so lonely either. We are competitive with open weights models in general, just a couple points behind best Qwen. Fully open models are important for research community; a lot of fundamental discoveries are made when you have access to training data. We call out we are the best fully open model because researchers would want to kn…

Makes sense, thanks!
Post reply on HN