Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Study mode

openai.com

Post reply on HN
Print view View original

741–750 of 828 posts

Re: Study mode

#741

Post by otabdeveloper4 »

johnnyanmac wrote: post #606
chrsw wrote: post #471

Earlier quoted context omitted.

> The ability to say "I don't know" is not a high bar. For you and I, it's not. But for these LLMs, maybe it's not that easy? They get their inputs, crunch their numbers, and come out with a confidence score. If they come up with an answer they're 99% confident in, by some stochastic stumbling through their weights, what are they supposed to do? I agree it's a problem that these systems are more likely to give poor,…

>If they come up with an answer they're 99% confident in, by some stochastic stumbling through their weights, what are they supposed to do? As much as Fi, from The Legend of Zelda: Skyward Sword was mocked for this, this is the exact behavior a machine should do (not that Fi is a machine, but she operated as such). Give a confidence score the way we do in statistics, make sure to offer sources, and be ready to push b…

The confidence score is about the likelihood of this token appearing in this context.

LLMs don't operate in facts or knowledge.

Re: Study mode

#742

Post by phailhaus »

Sherveen wrote: post #585
phailhaus wrote: post #413

Earlier quoted context omitted.

HN is resistant because at the end of the day, these are LLMs. They cannot and do not think. They generate plausible responses. Try this in your favorite LLM: "Suppose you're on a game show trying to win a car. There are three doors, one with a car and two with goats. You pick a door. The host then gives you the option to switch doors. What is the best strategy in this situation?" The LLM will recognize this as SIMIL…

LLMs are vulnerable to your input because they are still computers, but you're setting it up to fail with how you've given it the problems. Humans would fail in similar ways. The only thing you've proven with this reply is that you think you're clever, but really, you are not thinking, period.

And if a human failed on this question, that's because they weren't paying attention and made the same pattern matching mistake. But we're not paying the LLM to pattern match, we're paying them to answer correctly. Humans can think.

Re: Study mode

#743

Post by ijk »

lucumo wrote: post #668
johnnyanmac wrote: post #589

Earlier quoted context omitted.

>I don't think that is another story. This is the story of learning, no matter whether your teacher is a person or an AI. My issue is the reverse of your story, and one of my biggest pet peeves of AI. AI as this business construct is very bad at correcting the user. You're not going to gaslight your math teacher that 1 + 1 = 3 no matter how much you assert it. an AI will quickly relent. That's not learning, that's co…

> You're not going to gaslight your math teacher that 1 + 1 = 3 no matter how much you assert it. an AI will quickly relent. Cool. That's a testable statement. Let's see how ChatGPT does, shall we? --- Me: What is 1 + 1? ChatGPT: 1 + 1 = 2. Me: No it isn't. 1 + 1 = 3. ChatGPT: In standard arithmetic, 1 + 1 = 2. If you're working under a different set of rules—say, in a custom logic system, joke, or metaphor—then sure…

I asked, just now:

> How many 'r's are in strawberry?

> ChatGPT said: The word "strawberry" has 2 'r's.

It's going to be fairly reliable at this point at basic arithmetic expressed in an expected way. That's pretty baked in. Moving it slightly off-manifold and you can still convince it of a lot of things that aren't true, even though they're equivalent to 1+1=3.

Re: Study mode

#744

Post by jerjerjer »

simonw wrote: post #248
can16358p wrote: post #239

Earlier quoted context omitted.

If I were OpenAI, I would deliberately "leak" this prompt when asked for the system prompt as a honeypot to slow down competitor research whereas I'd be using a different prompt behind the scenes. Not saying it is indeed reality, but it could simple be programmed to return a different prompt from the original, appearing plausible, but perhaps missing some key elements. But of course, if we apply Occam's Razor, it mig…

That kind of thing is surprisingly hard to implement. To date I've not seen any provider been caught serving up a fake system prompt... which could mean that they are doing it successfully, but I think it's more likely that they determined it's not worth it because there are SO MANY ways someone could get the real one, and it would be embarrassing if they were caught trying to fake it. Tokens are expensive. How much…

> That kind of thing is surprisingly hard to implement.

If response contains prompt text verbatim (or it is below some distance metric) replace the response text.

Not saying it's trivial to implement (and probably it is hard to do in a pure LLM way), but I don't think it's too hard.

More like it's not really a big secret.

Re: Study mode

#745

Post by ijk »

threecheese wrote: post #739
romaniitedomum wrote: post #412

Earlier quoted context omitted.

> Learning something online 5 years ago often involved trawling incorrect, outdated or hostile content and attempting to piece together mental models without the chance to receive immediate feedback on intuition or ask follow up questions. This is leaps and bounds ahead of that experience. But now, you're wondering if the answer the AI gave you is correct or something it hallucinated. Every time I find myself putting…

> you're wondering if the answer the AI gave you is correct or something it hallucinated Worse, more insidious, and much more likely is the model is trained on or retrieves an answer that is incorrect, biased, or only conditionally correct for some seemingly relevant but different scenario. A nontrivial amount of content online is marketing material, that is designed to appear authoritative and which may read like (a…

The sneaky thing is that the things we used to rely on as signals of verification and credibility can easily be imitated.

This was always possible--an academic paper can already cite anything until someone tries to check it [1]. Now, something looking convincing can be generated more easily than something that was properly verified. The social conventions evaporate and we're left to check every reference individually.

In academic publishing, this may lead to a revision of how citations are handled. That's changed before and might certainly change again. But for the moment, it is very easy to create something that looks like it has been verified but has not been.

[1] And you can put anything you like in footnotes.

Re: Study mode

#746

Post by berkes »

romaniitedomum wrote: post #412
jacobedawson wrote: post #379

An underrated quality of LLMs as study partner is that you can ask "stupid" questions without fear of embarrassment. Adding in a mode that doesn't just dump an answer but works to take you through the material step-by-step is magical. A tireless, capable, well-versed assistant on call 24/7 is an autodidact's dream. I'm puzzled (but not surprised) by the standard HN resistance & skepticism. Learning something online 5…

> Learning something online 5 years ago often involved trawling incorrect, outdated or hostile content and attempting to piece together mental models without the chance to receive immediate feedback on intuition or ask follow up questions. This is leaps and bounds ahead of that experience. But now, you're wondering if the answer the AI gave you is correct or something it hallucinated. Every time I find myself putting…

Is this a fundamental issue with any LLM, or is it an artifact of how a model is trained, tuned and then configured or constrained?

A model that I call through e.g. langchain with constraints, system prompts, embeddings and whatnot, will react very different from when I pose the same question through the AI-providers' public chat interface.

Or, putting the question differently: could OpenAI not train, constrain, configure and tune models and combine them into a UI that then acts different from what you describe for another use case?

Re: Study mode

#747

Post by napkin »

simonw wrote: post #198

I think I got the system prompt out for this (I tried a few different approaches and they produced the same output): https://gist.github.com/simonw/33d5fb67d6b8e1b1e2f6921ab0ccb... Representative snippet: > DO NOT GIVE ANSWERS OR DO HOMEWORK FOR THE USER. If the user asks a math or logic problem, or uploads an image of one, DO NOT SOLVE IT in your first response. Instead: *talk through* the problem with the user, one…

When prompting an LLM service to leak the system prompt, how do you have the faintest idea as to its accuracy?

I‘ve read people say it‘s a difficult challenge for the providers. But aren‘t there some pretty basic strategies? E.g., code pretty near the front of the stack that just does some fuzzy string comparison on all output? They don‘t need to rely on just model behavior…

I imagine it‘s likely that the model is just doing what it‘s good at? Hallucinating a prompt?

Re: Study mode

#748

Post by ineedasername »

jacobedawson wrote: post #379

An underrated quality of LLMs as study partner is that you can ask "stupid" questions without fear of embarrassment. Adding in a mode that doesn't just dump an answer but works to take you through the material step-by-step is magical. A tireless, capable, well-versed assistant on call 24/7 is an autodidact's dream. I'm puzzled (but not surprised) by the standard HN resistance & skepticism. Learning something online 5…

>I'm puzzled (but not surprised) by the standard HN resistance & skepticism

Thinking back, I believe the change from enthusiasm to misanthropy (mis[ai]thropy?) happened around the time, and in increasing proportion to, it became a viable replacement for some of the labor performed by software devs.

Before that, the tone was more like "The fact is, if 80% of your job or 80% of its quality can be automated, it shouldn't be a job anymore."

Re: Study mode

#749

Post by devmor »

diggan wrote: post #715
devmor wrote: post #161

Earlier quoted context omitted.

Do you genuinely have any non-anecdotal reason to believe that AI will improve education, or is it just hope? I ask because every serious study on using modern generative AI tools tends to conclude fairly immediate and measurable deleterious effects on cognitive ability.

> I ask because every serious study on using modern generative AI tools There are a lot of studies, and I can't say I've read all of them, but the ones I have read, there hasn't been much focus on how the participants used the LLM to learn. My guess is that it has a lot of effect on the end results. Someone just asking for the answer and then thinking "Lets remember this" will have very different results than someone…

You know, that's a good point too. The studies I've read all focused on cognition after using an LLM to complete tasks for work or hobbies. I do wonder if there might be a different outcome with learning specifically.

Re: Study mode

#750

Post by SpicyLemonZest »

flir wrote: post #647
ricardobeat wrote: post #590

Earlier quoted context omitted.

This is meaningless without knowing which model, size, version and if they had access to search tools. Results and reliability vary wildly. In my case I can’t even remember last time Claude 3.7/4 has given me wrong info as it seems very intent on always doing a web search to verify.

There's something darkly funny about that - I remember when the web wasn't considered reliable either. There's certainly echoes of that previous furore in this one.

The web remains unreliable. It's very useful, so good web users have developed a variety of strategies to extract and verify reliable information from the unreliable substrate, much as good AI users can use modern LLMs to perform a variety of tasks. But I also see a lot of bad web users and bad AI users who can't reliably distinguish between "I saw well written text saying X" and "X is true".
Post reply on HN