This is all moving so quickly that slight degradation doesn't really matter. I assume the coming DeepMind model, Gemini, will outperform GPT-4. At least in some areas.
GPT-4 is getting worse over time, not better
181–190 of 315 posts
Re: GPT-4 is getting worse over time, not better
#182Earlier quoted context omitted.
If I'm using it from the web UI, this is exactly what I would want—this allows the language model to define the output language so I get correct syntax highlighting, without an error-prone secondary step of language detection. If I'm using it from the API, then all I have to do is strip out the leading backticks and language name if I don't need to check the language, or alternatively parse it to determine what the o…
It is still ignoring an explicit requirement, which is almost always bad. The user should be able to override what the creator/application thinks is 'strictly better'. Exceptions probably exist, but this isn't one of them.
Like I said, I read their requirement differently—the phrase they use is "the code only".
I don't think that including backticks is a violation of this requirement. It's still readily parseable and serves as metadata for interpreting the code. In the context of ChatGPT, which typically will provide a full explanation for the code snippet, I think this is a reasonable interpretation of the instruction.
Re: GPT-4 is getting worse over time, not better
#183Earlier quoted context omitted.
Solving an IQ test is also not useful by itself, but it is a good benchmark of intelligence.
> it is a good benchmark of intelligence. This is not well-established, and is subject to a great deal of dispute amongst experts in the field.
What is incredibly well established is that g is incredibly predictive for many life outcomes from income to educational attainment to drug addiction. This is what IQ test measures.
The only thing disputed is whether g is the same thing as “intelligence”, and the only dispute there is from softer science fields because “intelligence” is a word without a precise definition and a lot of feelings and opinions wrapped up in it.
Re: GPT-4 is getting worse over time, not better
#184The linked twitter account is an AI influencer, so take whatever is written with a grain of salt. Their goal is to get clicks and views by saying controversial things. This topic has come up before, and my hypothesis is still that GPT-4 hasn't gotten worse, it's just that the magic has worn off as we've used this tech. Studies to evaluate it have gotten better and cleaned up mistakes in the past.
I don’t believe this is true. It’s possible I was blinded by the light, but my programming tasks were previously (during the early access program) being handled by GPT-4 regularly and now they aren’t. I’ve also seen many anecdotes from engineers who had exceptionally early access before GPT-4 was public knowledge. The GPT-4 I use now feels like a shadow of the GPT-4 I used during the early access program. GPT-4, back…
> terrifyingly unaligned
Honestly, if people think that a statistical language model is "terrifying" because it can verbalise the concept of a mass killing, they need to give their heads a wobble.
My text editor can be used to write "set off a nuclear weapon in a city, lol". Is Notepad++.exe terrifying? What about the Sum of All Fears? I could get some pointers from that. Is Tom Clancy unaligned? Am I terrifying because I wrote that sentence? I even understand how terrible nuking a city would be and I still wrote it down. I must be, like, super unaligned.
Re: GPT-4 is getting worse over time, not better
#185This seems like something we could all test somewhat easily.
First, I assume this about the web UI version.
Second, there is a history of all of your prompts and responses on the left side.
Couldn't people just re-run their old prompts and see if the results are worse?
Re: GPT-4 is getting worse over time, not better
#186>Having the behavior of an LLM change over time is not acceptable. By now this is actually funny to read. Never rely on another companies product to make your own product, without accepting things can change overnight and shut you down As Llama2 is self hosted, you can choose which iteration to host. Much better developer experience Edit: to be clear OpenAI is unprofitable, so is Reddit, so was Stadia. Building on to…
That's a pretty defeatist take. Surely if you pay for a service you should expect the provider to be making good faith efforts to provide the same quality of service over time? Natural degradation would be fine, but purposefully sandbagging the service so it gets worse because cheaper is unacceptable. That we have become numb to the point that we collectively accept such poor behavior on the part of vendors in concre…
This is why contracts exist
Re: GPT-4 is getting worse over time, not better
#187Earlier quoted context omitted.
> it is a good benchmark of intelligence. This is not well-established, and is subject to a great deal of dispute amongst experts in the field.
What is incredibly well established is that g is incredibly predictive for many life outcomes from income to educational attainment to drug addiction. This is what IQ test measures. The only thing disputed is whether g is the same thing as “intelligence”, and the only dispute there is from softer science fields because “intelligence” is a word without a precise definition and a lot of feelings and opinions wrapped up…
[flagged]
Re: GPT-4 is getting worse over time, not better
#188Has anyone experimented with mixing outputs from LLm's on a per-token basis? Ie. easy tokens can be provided by a cheap-to-run model, and hard tokens are given by an expensive to run model? A model could be used to decide when it is worth running the expensive model, based on the inputs, output so far, and probability distribution of the output of the cheap model. For example, "Q: If I have 3 bananas and eat none, th…
This is the topic of the language models as tool makers paper https://arxiv.org/pdf/2305.17126.pdf
Re: GPT-4 is getting worse over time, not better
#189Earlier quoted context omitted.
I don’t believe this is true. It’s possible I was blinded by the light, but my programming tasks were previously (during the early access program) being handled by GPT-4 regularly and now they aren’t. I’ve also seen many anecdotes from engineers who had exceptionally early access before GPT-4 was public knowledge. The GPT-4 I use now feels like a shadow of the GPT-4 I used during the early access program. GPT-4, back…
> terrifyingly unaligned Honestly, if people think that a statistical language model is "terrifying" because it can verbalise the concept of a mass killing, they need to give their heads a wobble. My text editor can be used to write "set off a nuclear weapon in a city, lol". Is Notepad++.exe terrifying? What about the Sum of All Fears ? I could get some pointers from that. Is Tom Clancy unaligned? Am I terrifying bec…
>Is Tom Clancy unaligned?
Yes, humans are unaligned. This is why alignment is hard: we're trying to produce machines with human-level intelligence but superhuman levels of morality.
Re: GPT-4 is getting worse over time, not better
#190There is a Chatbot AI product called character.ai that has suffered a marked decline in quality since its launch as they battle their users to maintain the AI’s safety protocols (similar to chatGPT “jailbreaks”). I wonder if something similar could be happening here.
These fighting against people using their product in “unauthorized” ways by the ai companies doesn’t make any sense to me. Who cares if character.ai users do some weird stuff with it, or replika creates romantic relationships, or people make off color jokes in gpt. There seems to be a lot of engineering effort driven by some product managers to have the ai do very specific things which makes the product much worse in…
The problem is 'weird stuff' on chai included suicide, which sadly showed it needed better/more guardrails.