Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Olmo 3: Charting a path through the model flow to lead open-source AI

allenai.org

Post reply on HN
Print view View original

51–60 of 135 posts

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#51

Post by latexr »

embedding-shape wrote: post #34
Y_Y wrote: post #26

I asked it if giraffes were kosher to eat and it told me: > Giraffes are not kosher because they do not chew their cud, even though they have split hooves. Both requirements must be satisfied for an animal to be permissible. HN will have removed the extraneous emojis. This is at odds with my interpretation of giraffe anatomy and behaviour and of Talmudic law. Luckily old sycophant GPT5.1 agrees with me: > Yes. They h…

How many times did you retry (so it's not just up to chance), what were the parameters, specifically for temperature and top_p?

> How many times did you retry (so it's not just up to chance)

If you don’t know the answer to a question, retrying multiple times only serves to amplify your bias, you have no basis to know the answer is correct.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#52

Post by fnbr »

tcsenpai wrote: post #20

Also: I simply asked to the 7B model "hi, who are u": Thought for 1 seconds Okay, the user is asking "Who are u?" with a casual tone ("u" instead of "you", exclamation mark). Let me break this down. First, I need to address the question itself. They want to know who I am as an AI assistant. My standard response is that I'm DeepSeek R1, designed to be helpful and handle reasoning step by step. But maybe they're not fa…

(I’m a researcher on the post-training team at Ai2.)

Where did you try this? On the Ai2 playground?

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#53

Post by embedding-shape »

latexr wrote: post #51
embedding-shape wrote: post #34

Earlier quoted context omitted.

How many times did you retry (so it's not just up to chance), what were the parameters, specifically for temperature and top_p?

> How many times did you retry (so it's not just up to chance) If you don’t know the answer to a question, retrying multiple times only serves to amplify your bias, you have no basis to know the answer is correct.

Well, seems in this case parent did know the answer, so I'm not sure what your point is.

I'm asking for the sake of reproducibility and to clarify if they used the text-by-chance generator more than once, to ensure they didn't just hit one out of ten bad cases since they only tested it once.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#54

Post by latexr »

tcsenpai wrote: post #20

Also: I simply asked to the 7B model "hi, who are u": Thought for 1 seconds Okay, the user is asking "Who are u?" with a casual tone ("u" instead of "you", exclamation mark). Let me break this down. First, I need to address the question itself. They want to know who I am as an AI assistant. My standard response is that I'm DeepSeek R1, designed to be helpful and handle reasoning step by step. But maybe they're not fa…

Reminds me of an old joke where a guy is walking down the street and another person says “good morning”. The guy starts deconstructing what “good morning” means until he finally reaches the conclusion “that bastard was calling me an asshole”.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#55

Post by Y_Y »

embedding-shape wrote: post #34
Y_Y wrote: post #26

I asked it if giraffes were kosher to eat and it told me: > Giraffes are not kosher because they do not chew their cud, even though they have split hooves. Both requirements must be satisfied for an animal to be permissible. HN will have removed the extraneous emojis. This is at odds with my interpretation of giraffe anatomy and behaviour and of Talmudic law. Luckily old sycophant GPT5.1 agrees with me: > Yes. They h…

How many times did you retry (so it's not just up to chance), what were the parameters, specifically for temperature and top_p?

Sorry I lost the chat, but it was default parameters on the 32B model. It cited some books saying that they had three stomachs and didn't ruminate, but after I pressed on these points it admitted that it left out the fourth stomach because it was small, and claimed that the rumination wasn't "true" in some sense.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#57

Post by refulgentis »

embedding-shape wrote: post #45
refulgentis wrote: post #41

Earlier quoted context omitted.

Unlikely to see more VRAM in the short term, memory prices are thru the roof :/ like, not subtly, 2-4x.

Well, GPUs are getting more VRAM, although it's pricey. But we didn't used to have 96GB VRAM GPUs at all, now they do exist :) But for the ones who can afford it, it is at least possible today. Slowly it increases.

Agreed, in the limit, RAM go up. As billg knows, 128KB definitely wasn't enough for everyone :)

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#58

Post by zamadatix »

latexr wrote: post #51
embedding-shape wrote: post #34

Earlier quoted context omitted.

How many times did you retry (so it's not just up to chance), what were the parameters, specifically for temperature and top_p?

> How many times did you retry (so it's not just up to chance) If you don’t know the answer to a question, retrying multiple times only serves to amplify your bias, you have no basis to know the answer is correct.

If you retry until it gives the answer you want then it only serves to amplify your bias. If you retry and see how often it agrees with itself then it serves to show there is no confidence in an answer all around.

It's a bit of a crutch for LLMs lacking the ability to just say "I'm not sure" because doing so is against how they are rewarded in training.

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#59

Post by turnsout »

I used the Ai2 Playground and Olmo 3 32GB Think, and asked it to recommend a language for a green-field web app based on a list of criteria. It gave me a very good and well-reasoned answer (Go, with Rust as a backup), formatted like a high-quality ChatGPT or Claude response.

I then had it show the "OlmoTrace" for its response, which seems like it finds exact matches for text strings in its training data that end up in the response. Some of the matched sources were related (pages about Go, Rust, Python, etc), while others were completely unrelated, but just happened to have the same turn of phrase (e.g. "Steeper learning curve").

It was interesting, but is it useful? It was impossible for me to actually fact-check any of the claims in the response based on the matched training data. At this stage, it felt about as helpful as linking every word to that word's entry in a dictionary. "Yep, that's a word alright." I don't think it's really tracing the "thought."

What could be interesting is if the user could dynamically exclude certain training sources before the response is generated. Like, I want to ask a question about climate change, but I want to exclude all newspapers and focus on academic journals.

Transparency is a good first step, but I think we're missing the "Step 2."

Re: Olmo 3: Charting a path through the model flow to lead open-source AI

#60

Post by andai »

thot_experiment wrote: post #11

Qwen3-30B-VL is going to be fucking hard to beat as a daily driver, it's so good for the base 80% of tasks I want an AI for, and holy fuck is it fast. 90tok/s on my machine, I pretty much keep it in vram permanently. I think this sort of work is important and I'm really glad it's being done, but in terms of something I want to use every day there's no way a dense model can compete unless it's smart as fuck. Even dumb…

I'm out of the loop... so Qwen3-30B-VL is smart and Qwen3-30B is dumb... and that has to do not with the size but architecture?
Post reply on HN