831–840 of 909 posts
Earlier quoted context omitted.
That access is over a limited API and usually under heavy restrictions on the healthcare org side (e. g., only use a dedicated machine, locked up software, tracked responses and so on). Running a local model is often much easier: if you already have data on a machine and can run a model without breaching any network one could run it without any new approvals.
What? It’s a straight connect to the models api from azure, aws, or gcp. I am literally using Claude opus 4.1 right now.
Is it just me or is this MUCH sturdier against jailbreaks then similar models, or even the ChatGPT ones? I have had problems even making it output nothing. But I guess I'll try some more :D Nice job @openAI team.
thoughts in the field say instead of a model that is pre-trained normally then censored, this is a model pre-trained on filtered data. i.e. it have never seen anything that is unsafe, ever. you can't jailbreak when there is nothing "outside".
I don't think that's true, you can't ask it outright "How do you make a molotov cocktail?" but if you start by talking about what is allowed/disallowed by policies, how examples would look for disallowed policies and eventually ask it for the "general principles" of how to make a molotov cocktail, it'll happily oblige by essentially giving you enough information to build one.
So it does know how to make an molotov cocktail, for example, but (mostly) refuses to share it.
Earlier quoted context omitted.
I don't know how to ask this without being direct and dumb: Where do I get a layman's introduction to LLMs that could work me up to understanding every term and concept you just discussed? Either specific videos, or if nothing else, a reliable Youtube channel?
What I’ve sometimes done when trying to make sense of recent LLM research is give the paper and related documents to ChatGPT, Claude, or Gemini and ask them to explain the specific terms I don’t understand. If I don’t understand their explanations or want to know more, I ask follow-ups. Doing this in voice mode works better for me than text chat does. When I just want a full summary without necessarily understanding…
My first impression is that this model thinks for a _long_ time. It proposes ideas and then says, "no wait, it's actually..." and then starts the same process again. It will go in loops examining different ideas as it struggles to understand the basic process for calculating notes. It seems to struggle with the septatonic note -> Set notation (semitone positions), as many humans do. As I write this it's been going at about 3tok/s for about 25 minutes. If it finishes while I type this up I will post the final answer.
I did glance at its thinking output just now and I noticed this excerpt where it finally got really close to the answer, giving the right name (despite using the wrong numbers in the set notation, which should be: 0,3,4,6,7,9,10:
Check "Lydian #2": 0,2,3,5,7,9,10. Not ours.
Its notes are: 1 ♯2 3 ♯4 5 6 7
I find looking up lesser known changes and asking for a mode is a good experiment. First I can see if an LLM has developed a way to reason about numbers geometrically as is the case with music.
And by posting about it, I can test how fast AIs might memorize the answer from a random comment on the internet, as I can just use a different change if I find that this post was eventually regurgitated.
After letting ollama run for a while, I'm post what it was thinking about in case anybody's interested. [1]
Also copilot.microsoft.com's wrong answer: [2], and chatgpt.com [3]
I do think that there may be an issue where I did it wrong because after trying the new ollama gui I noticed it's using a context length of 4k tokens, which it might be blowing way past. Another test might be to try the question with a higher context length, but at the same time, it seems like if this question can't be figured out in less time than that, that it will never have enough time...
[0] https://edrihan.neocities.org/changedex (bad UX on mobile! - and in general ;)). won't fix, will make new site soon) [1] https://pastebin.com/wESXHwE1 [2] https://pastebin.com/XHD4ARTF [3] https://pastebin.com/ptMiNbq7
Earlier quoted context omitted.
Add healthcare. Cannot send our patients data to a cloud provider
Europe? US? In Finland doctors can send live patient encounters to azure openai for transcription and summarization.
Might be possible with some certain specific regions/environments of Azure tho, because iirc they have a few that support government confidentiality type of stuff, and some that tout HIPAA compliance as well. Not sure about details of those though.
The lede is being missed imo. gpt-oss:20b is a top ten model (on MMLU (right behind Gemini-2.5-Pro) and I just ran it locally on my Macbook Air M3 from last year. I've been experimenting with a lot of local models, both on my laptop and on my phone (Pixel 9 Pro), and I figured we'd be here in a year or two. But no, we're here today. A basically frontier model, running for the cost of electricity (free with a rounding…
I tried 20b locally and it couldn't reason a way out of a basic river crossing puzzle with labels changed. That is not anywhere near SOTA. In fact it's worse than many local models that can do it, including e.g. QwQ-32b.
Earlier quoted context omitted.
The space invaders benchmark proves that the model can implement a working HTML and JavaScript game from a single prompt. That's a pretty fundamental capability for a model. Comparing them between models is also kind of interesting, even if it's not a flawlessly robust comparison: https://simonwillison.net/tags/space-invaders/
Implement or retrieve? That’s an important distinction. When evaluating models, you run a variety of tests, and the benchmarks that aren’t publicly disclosed are the most reliable. Your Space Invaders game isn’t really a benchmark of anything, just Google it, and you’ll find plenty of implementations.
Firstly, 12GB is not enough space to hold a copy of anything that large from the training data and just regurgitate it back out again.
You can also watch the thinking traces on the reasoning models and see them piece together the approach they are going to take. Here's an example from the 20B OpenAI model with reasoning set to medium: https://gist.github.com/simonw/63d7d8c43ae2ac93c214325bd6d60...
Illustrative extract:
> Edge detection: aliens leftmost or rightmost position relative to canvas width minus alien width.
> When direction changes, move all aliens down by step (e.g., 10 px).
The benchmarks that aren't publicly disclosed tend to be way simpler than this: things like "What is the embryological origin of the hyoid bone?" (real example from MMLU, it then provides four choices as a multiple-choice challenge).