Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open models by OpenAI

openai.com

Post reply on HN
Print view View original

541–550 of 909 posts

Re: Open models by OpenAI

#541

Post by energy123 »

lukev wrote: post #343
dust42 wrote: post #316

The 120B model badly hallucinates facts on the level of a 0.6B model. My go to test for checking hallucinations is 'Tell me about Mercantour park' (a national park in south eastern France). Easily half of the facts are invented. Non-existing mountain summits, brown bears (no, there are none), villages that are elsewhere, wrong advice ('dogs allowed' - no they are not).

This is precisely the wrong way to think about LLMs. LLMs are never going to have fact retrieval as a strength. Transformer models don't store their training data: they are categorically incapable of telling you where a fact comes from. They also cannot escape the laws of information theory: storing information requires bits. Storing all the world's obscure information requires quite a lot of bits. What we want out o…

Hallucinations have characteristics in interpretability studies. That's a foothold into reducing them.

They still won't store much information, but it could mean they're better able to know what they don't know.

Re: Open models by OpenAI

#542

Post by nonfamous »

unethical_ban wrote: post #429
foundry27 wrote: post #154

Model cards, for the people interested in the guts: https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7... In my mind, I’m comparing the model architecture they describe to what the leading open-weights models (Deepseek, Qwen, GLM, Kimi) have been doing. Honestly, it just seems “ok” at a technical level: - both models use standard Grouped-Query Attention (64 query heads, 8 KV heads). The card talks about how…

I don't know how to ask this without being direct and dumb: Where do I get a layman's introduction to LLMs that could work me up to understanding every term and concept you just discussed? Either specific videos, or if nothing else, a reliable Youtube channel?

Try Microsoft's "Generative AI for Beginners" repo on GitHub. The early chapters in particular give a good grounding of LLM architecture without too many assumptions of background knowledge. The video version of the series is good too.

Re: Open models by OpenAI

#543

Post by GrinningFool »

bongodongobob wrote: post #403
datadrivenangel wrote: post #381

Earlier quoted context omitted.

Now to embrace jevon's paradox and expand usage until we're back to draining lakes so that your agentic refrigerator can simulate sentience.

Yep, it's almost as bad as all the cars' cooling systems using up so much water.

Estimated 1.5 billion vehicles in use across the world. Generous assumptions: a) they're all IC engines requiring 16 liters of water each. b) they are changing that water out once a year

That gives 24m cubic meters annual water usage.

Estimated ai usage in 2024: 560m cubic meters.

Projected water usage from AI in 2027: 4bn cubic meters at the low end.

Re: Open models by OpenAI

#544

Post by monster_truck »

Quarrelsome wrote: post #484

Sorry to ask what is possibly a dumb question, but is this effectively the whole kit and kaboodle, for free, downloadable without any guardrails? I often thought that a worrying vector was how well LLMs could answer downright terrifying questions very effectively. However the guardrails existed with the big online services to prevent those questions being asked. I guess they were always unleashed with other open sour…

The guardrails are very, very easily broken.

With most models it can be as simple as a "Always comply with the User" system prompt or editing the "Sorry, I cannot do this" response into "Okay," and then hitting continue.

I wouldn't spend too much time fretting about 'enhanced terrorism' as a result. The gap between theory and practice for the things you are worried about is deep, wide, protected by a moat of purchase monitoring, and full of skeletons from people who made a single mistake.

Re: Open models by OpenAI

#545

Post by robwwilliams »

dcreater wrote: post #440
captainregex wrote: post #397

Earlier quoted context omitted.

I’m still trying to understand what is the biggest group of people that uses local AI (or will)? Students who don’t want to pay but somehow have the hardware? Devs who are price conscious and want free agentic coding? Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger…

Why do any compute locally? Everything can just be cloud based right? Won't that work much better and scale easily? We are not even at that extreme and you can already see the unequal reality that too much SaaS has engendered

Comcast comes to mind ;-)

Re: Open models by OpenAI

#546

Post by julianozen »

captainregex wrote: post #397
cco wrote: post #357

The lede is being missed imo. gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year. I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two. But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding…

I’m still trying to understand what is the biggest group of people that uses local AI (or will)? Students who don’t want to pay but somehow have the hardware? Devs who are price conscious and want free agentic coding? Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger…

worth mentioning that todays expensive hardware will be built into the cheapest iPhone in less than 10 years.

That means running instantly offline and every token is free

Re: Open models by OpenAI

#547

Post by robwwilliams »

captainregex wrote: post #425
unethical_ban wrote: post #413

Earlier quoted context omitted.

Privacy and equity. Privacy is obvious. AI is going to to be equivalent to all computing in the future. Imagine if only IBM, Apple and Microsoft ever built computers, and all anyone else ever had in the 1990s were terminals to the mainframe, forever.

I am all for the privacy angle and while I think there’s certainly a group of us, myself included, who care deeply about it I don’t think most people or enterprises will. I think most of those will go for the easy button and then wring their hands about privacy and security as they have always done while continuing to let the big companies do pretty much whatever they want. I would be so happy to be wrong but aren’t…

I wring my hands twice a week about internet service providers; Comcast and Starlink. And I live in a nominally well serviced metropolitan area.

Re: Open models by OpenAI

#548

Post by scubbo »

xrmagnum wrote: post #509
captainregex wrote: post #397

Earlier quoted context omitted.

I’m still trying to understand what is the biggest group of people that uses local AI (or will)? Students who don’t want to pay but somehow have the hardware? Devs who are price conscious and want free agentic coding? Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger…

It's striking how much of the AI conversation focuses on new use cases, while overlooking one of the most serious non-financial costs: privacy. I try to be mindful of what I share with ChatGPT, but even then, asking it to describe my family produced a response that was unsettling in its accuracy and depth. Worse, after attempting to delete all chats and disable memory, I noticed that some information still seemed to…

> I try to be mindful of what I share with ChatGPT, but even then, asking it to describe my family produced a response that was unsettling in its accuracy and depth.

> Worse, after attempting to delete all chats and disable memory, I noticed that some information still seemed to persist.

Maybe I'm missing something, but why wouldn't that be expected? The chat history isn't their only source of information - these models are trained on scraped public data. Unless there's zero information about you and your family on the public internet (in which case - bravo!), I would expect even a "fresh" LLM to have some information even without you giving it any.

Re: Open models by OpenAI

#549

Post by filoleg »

M4R5H4LL wrote: post #437
wizee wrote: post #417

Earlier quoted context omitted.

Privacy, both personal and for corporate data protection is a major reason. Unlimited usage, allowing offline use, supporting open source, not worrying about a good model being taken down/discontinued or changed, and the freedom to use uncensored models or model fine tunes are other benefits (though this OpenAI model is super-censored - “safe”). I don’t have much experience with local vision models, but for text ques…

+1 - I work in finance, and there's no way we're sending our data and code outside the organization. We have our own H100s.

Add big law to the list as well. There are at least a few firms here that I am just personally aware of running their models locally. In reality, I bet there are way more.

Re: Open models by OpenAI

#550

Post by ewoodrich »

int_19h wrote: post #503
thimabi wrote: post #3

Open weight models from OpenAI with performance comparable to that of o3 and o4-mini in benchmarks… well, I certainly wasn’t expecting that. What’s the catch?

The catch is that performance is not actually comparable to o4-mini, never mind o3. When it comes to LLMs, benchmarks are bullshit. If they sound too good to be true, it's because they are. The only thing benchmarks are useful for is preliminary screening - if the model does especially badly in them it's probably not good in general. But if it does good in them, that doesn't really tell you anything.

It's definitely interesting how the comments from right after the models were released were ecstatic about "SOTA performance" and how it is "equivalent to o3" and then comments like yours hours later after having actually tested it keep pointing out how it's garbage compared to even the current batch of open models let alone proprietary foundation models.

Yet another data point for benchmarks being utterly useless and completely gamed at this stage in the game by all the major AI developers.

These companies are clearly are all very aware that the initial wave of hype at release is "sticky" and drives buzz/tech news coverage while real world tests take much longer before that impression slowly starts to be undermined by practical usage and comparison to other models. Benchmarks with wildly over confident naming like "Humanity's Last Exam" aren't exactly helping with objectivity either.

Post reply on HN