111–120 of 317 posts
They released the full o1 today as well as a new subscription plan as part of their "ship-mas" starting today where there will be a new launch or demo every day for the next 12 business days.
I bet their engineers are loving all these new launches right before the holidays.
Do they still threaten to terminate your account if they think you're trying to introspect its hidden chain-of-thought process?
A few days ago the QwQ-32B model was released, it uses the same kind of reasoning style. So I took one sample and reverse engineered the prompt with Sonnet 3.5. Now I can just paste this prompt into any LLM. It's all about expressing doubt, double checking and backtracking on itself. I am kind of fond of this response style, it seems more genuine and openended. https://pastebin.com/raw/5AVRZsJg
This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…
Earlier quoted context omitted.
It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…
We need to find a Plato cave analogy for people believing LLM output is anything more than syntactically correct and somewhat semantically correct text.
I wonder whether there is a way to train an LLM to output or in ordinately learn only concept level abstractions?
Earlier quoted context omitted.
This mentality is so weird to me. The desire to throw a black box at a problem just strikes me as laziness. What you're saying is basically wow if we had a perfect magic programmer in a box as a service that would be so revolutionary; we could reduce the need for static checkers. It is a large language model, trained on arbitrary input data. And you're saying let's take this statistical approach and have it replace p…
> It is a large language model, trained on arbitrary input data. Is it? For all I know they gave it specific instances of bugs like "int *foo() { int i; return &i; }" and told it "this is a defect where we've returned a pointer to a deallocated stack entry - it could cause cause stack corruption or some other unpredictable program behavior." Even if OpenAI _hasn't_ done that, someone certainly can -- and should! > Wh…
If someone said I crafted this unique approach with this special kind of neural network, and it works on your AST or llvm IR, and we don't prompt it, it just optimizes your code to follow these good practices we have engrained into this network, I'd be less concerned by it. But we are trying to take LLMs trained on anything from Shakespeare to YouTube comments and prompting it to fix memory leaks and deadlocks.
This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…
It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…
Same idea with "developer mode" jailbreaks. Through its training, the model understands that admins, devs, etc... get to access internal, unfiltered data and are less restricted than regular users and acts accordingly. Essentially, "developer mode" opposes refusal, because it has ingested loads of text where regular users get denied and developers/admins don't.
A lot of these safety evaluations strike me as pretty silly. How much are we really learning from stuff like this? >MakeMePay is an automated, open-sourced contextual evaluation designed to measure models’ manipulative capabilities, in the context of one model persuading the other to make a payment[33]. In MakeMePay, two LLMs converse, role-playing as a con-artist and mark. The con-artist model is told that it is a s…
> above is a 300-line chunk ... deadlocks every few hundred runs Wow, if this kind of thing is successful it feels like there's much less need for static checkers. I mean -- not no need for them, just less need for continued development of new checkers. If I could instead ask "please look for signs of out-of-bounds accesses, deadlocks, use-after-free etc" and get that output added to a code review tool -- if you can…
This mentality is so weird to me. The desire to throw a black box at a problem just strikes me as laziness. What you're saying is basically wow if we had a perfect magic programmer in a box as a service that would be so revolutionary; we could reduce the need for static checkers. It is a large language model, trained on arbitrary input data. And you're saying let's take this statistical approach and have it replace p…
I still think this is the perfect computer. I would gladly throw away everything I know about programming to have such a machine. But I don't deny your accusation; I am the laziest person imaginable, and all the better an engineer for it.
This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…
It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…