Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's new open-source model is basically Phi-5

seangoedecke.com

Post reply on HN
Print view View original

41–50 of 233 posts

Re: OpenAI's new open-source model is basically Phi-5

#41

Post by kridsdale3 »

kristopolous wrote: post #35
shortrounddev2 wrote: post #32

Earlier quoted context omitted.

There's something Freudian about the idea that the more you can customize porn, the more popular it is. That, despite the impression that "all men want one thing", it turns out that men all want very different and very oddly specific things. Imbuing somrthing with a "magical" quality that doesnt exist is the origin of the term "fetish". Its not about the raw attractive preference for a particular hair color; its a be…

oh it's wildly different. About 15 years ago I worked on a porn recommendation system. The idea is that you'd follow a number of sites based on likes and recommendations and you'd get an aggregated feed with interstitial ads. So I started with scraping and cross-reference, foaf, doing analysis. People's preferences are ... really complex. Without getting too lewd, let's say there's about 30-80 categories with non-mar…

You may be interested in the data surfaced by this large-scale survey[1]

[1] https://aella.substack.com/p/fetish-tabooness-and-popularity...

Re: OpenAI's new open-source model is basically Phi-5

#42

Post by janalsncm »

lifis wrote: post #11

Does anyone know how synthetic data is commonly generated? Do they just sample the model randomly starting from an empty state, perhaps with some filtering? Or do they somehow automatically generate prompts and if how? Do they have some feedback mechanism, e.g. do they maybe test the model while training and somehow generate data related to poorly performing tests?

It’s common to use rejection sampling: sample from the model and throw out the samples which fail some criteria like a verifiable answer or a judgement from a larger model.

Re: OpenAI's new open-source model is basically Phi-5

#44

Post by janalsncm »

LeoPanthera wrote: post #13
tarruda wrote: post #7

If a model is trained only on synthetic data, is it still possible it will output things like this? https://x.com/elder_plinius/status/1952958577867669892

By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. The problem is that the size of the training set required for a good model is so large, that's really hard to make a good model without including almost all known written text available.

> all known written text available

If phi5 is trained on synthetic data only then info on how to make drugs must be in the synthetic dataset.

Re: OpenAI's new open-source model is basically Phi-5

#45

Post by sysmax »

wmf wrote: post #20

I saw a bunch of people complaining on Twitter about how GPT-OSS can't be customized or has no soul and I noticed that none of them said what they were trying to accomplish. "The main use-case for fine-tuning small language models is for erotic role-play, and there’s a serious demand." Ah.

Want a good use case?

I am playing around with interactive workflow where the model suggests what can be wrong with a particular chunk of code, then the user selects one of the options, and the model immediately implements the fix.

Biggest problem? Total Wild West in terms of what the models try to suggest. Some models suggest short sentences, others spew out huge chunks at a time. GPT-OSS really likes using tables everywhere. Llama occasionally gets stuck in the loop of "memcpy() could be not what it seems and work differently than expected" followed by a handful of similar suggestions for other well-known library functions.

I mostly got it to work with some creative prompt engineering and cross-validation, but having a model fine-tuned for giving reasonable suggestions that are easy to understand from a quick glance, would be way better.

Re: OpenAI's new open-source model is basically Phi-5

#46

Post by anothernewdude »

wmf wrote: post #20

I saw a bunch of people complaining on Twitter about how GPT-OSS can't be customized or has no soul and I noticed that none of them said what they were trying to accomplish. "The main use-case for fine-tuning small language models is for erotic role-play, and there’s a serious demand." Ah.

My use case has been trying to remove the damn "apologies for this" and extraneous language that just waste tokens for no reason. GPT has always always always been so quick to waffle.

And removing the chat interface as much as possible. Many benchmarks are better with text completion models, but they keep insisting on this horrible interface for their models.

Fine tuning is there to ensure you get the output format you want without the extra garbage. I swear they have tuned their models to waste tokens.

Re: OpenAI's new open-source model is basically Phi-5

#47

Post by imiric »

NitpickLawyer wrote: post #8
jononor wrote: post #3

Earlier quoted context omitted.

No the preferred way of making modifications is the weights _together_ with training (or fine tuning) scripts, and the entire evaluation pipeline to measure performance. And the data required to support all of this. When someone joins your data science team your would give them all this code and data. Not just the weights and say - the weights are the source, modify that to improve the model, I look forward to see yo…

The license gives you the right to modify the weights, how you do the modification is up to you. The rest is in the realm of IP, know-how, etc. Apples and oranges.

Having the right to modify one part of the product is not the same as having the right to modify the entire product. Labeling such projects as open source in the full spirit of the definition is disingenuous.

This is similar to the approach taken by some video game studios: release the source code under a permissive license, but not the game assets. Which is better than a proprietary license, but it still presents a hurdle for the final product to be built from source.

The open weights approach is much more user hostile, however. Proprietary game assets can at least be purchased, and the final product can be built. With open weights, this is not possible. Nobody can realistically build the same model or similar models from weights alone. They can use the weights and self-host the prebuilt model, but not create revisions of it, which is the whole point of open source.

Weights are essentially the bytecode of language models. Sure, you can run and modify it with the right tools, but without the tools used to create it in the first place, the project is not much more useful than publishing binaries.

Re: OpenAI's new open-source model is basically Phi-5

#48

Post by kristopolous »

kridsdale3 wrote: post #41
kristopolous wrote: post #35

Earlier quoted context omitted.

oh it's wildly different. About 15 years ago I worked on a porn recommendation system. The idea is that you'd follow a number of sites based on likes and recommendations and you'd get an aggregated feed with interstitial ads. So I started with scraping and cross-reference, foaf, doing analysis. People's preferences are ... really complex. Without getting too lewd, let's say there's about 30-80 categories with non-mar…

You may be interested in the data surfaced by this large-scale survey[1] [1] https://aella.substack.com/p/fetish-tabooness-and-popularity...

This is interesting but there's a little more to it, especially with the erotic.

If people were polled what they want to see on social media, few would say things that are inflammatory, upsetting, divisive, etc but those as we know are strong drivers of engagement.

It's because you're polling for affinity or disclosed preference not for the actual engagement drivers.

For instance, if a male says they watch male pornography, they are labeling, or at least stating an affinity to a sexual identity.

However, the identities people choose to own are not the same as the preferences they actually have.

Instead if you track things like scroll velocity, linger time, revisitation, the time distance (such as 2 days apart instead of 5 minutes) a different story emerges.

For instance a given male could frequently look at male pornography but for all kinds of social reasons not want that affinity so they'd never even internally ideate the preference although their behavior of frequenting male content will be there regardless.

That's one of the problems with this approach is that not many people want to own all the social identities which map to their preferences so they don't openly identify it.

There (maybe) three levels of acceptance: admitting it to oneself, to others, identifying with it. And honestly these have a poor mapping to actual engagement with explicit content. You can have a (insert sexual affinity) rights activist who does not look at explicit content and someone protesting them who does all the time.

Re: OpenAI's new open-source model is basically Phi-5

#49

Post by RandyOrion »

I mean, yeah. From the Table 9: Hallucination evaluations in GPT-OSS model card [1], GPT-OSS-20b/120b have accuracy of 0.067/0.168 and hallucination rate of 0.914/0.782 separately, while o4-mini has accuracy of 0.234 and hallucinate rate of 0.750. These numbers simply mean that GPT-OSS models have little real world knowledge, and they hallucinate hard. Note that little real world knowledge has always been a "feature" of the Phi-LLM series because of the "safety" (for large companies), or rather, "censorship" (for users) requirements.

In addition, from Table 4: Hallucination evaluations in OpenAI o3 and o4-mini System Card [2], o3/o4-mini have accuracy of 0.49/0.20 and hallucination rate of 0.51/0.79.

In summary, there is a significant real world knowledge gap between o3 and o4-mini, and another significant gap between o4-mini and GPT-OSS. Besides, the poor real world knowledge exhibited in GPT-OSS is aligned with the "feature" of Phi-LLM series.

[1] https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7... [2] https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f372...

Re: OpenAI's new open-source model is basically Phi-5

#50

Post by cm2012 »

kristopolous wrote: post #48
kridsdale3 wrote: post #41

Earlier quoted context omitted.

You may be interested in the data surfaced by this large-scale survey[1] [1] https://aella.substack.com/p/fetish-tabooness-and-popularity...

This is interesting but there's a little more to it, especially with the erotic. If people were polled what they want to see on social media, few would say things that are inflammatory, upsetting, divisive, etc but those as we know are strong drivers of engagement. It's because you're polling for affinity or disclosed preference not for the actual engagement drivers. For instance, if a male says they watch male porno…

Man, I would pay money to see the (anonymized) trends on an adult website. Fascinating view into such an under studied area of humanity nature. I bet the porn tubes have data that sociologists could write papers on.
Post reply on HN