[flagged]
Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
11–20 of 44 posts
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#12> If there is indeed no degradation how could the perceived degradation be explained? By being disproportionately impressed previously. Maybe in the early days people were so impressed by their little play experiments they forgave the shortcomings. Now that the novelty is wearing off and they try to use it for productive work, the scales tipped and failures are given more weight.
When chat gpt first came out I was able to feed it some text to parse and then create python scripts to process similar texts and create csv and excel files from those. I was able to create a basic working python scripts in 1-2 hours. And very complex scripts over a couple of days. I recently tried to do the same with chatgpt again and simply am unable to. I wish I had saved the exact text I fed into chat gpt then so that I could see if it was just my question was wrong or free version of chatgpt has now been hobbled
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#13IMO the rate of change or volatility of model changes, RLHF, user-behavior, the definition of 'better' is much higher than the rate of change of model degradation (if any) so it's hopeless to try and measure this
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#14> If there is indeed no degradation how could the perceived degradation be explained? By being disproportionately impressed previously. Maybe in the early days people were so impressed by their little play experiments they forgave the shortcomings. Now that the novelty is wearing off and they try to use it for productive work, the scales tipped and failures are given more weight.
No, you are wrong, and anyone that was an early adopter will tell you. At the start there were many Python coding tasks it completed flawlessly one shot, that all later iterations needed 5 takes for if they managed to get it at all. And that was using the early 3 series ChatGPT, which they rapidly nerfed into the ground.While the current GPT-4 model feels better in some other less quantifiable ways, Python coding hasn't recovered.
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#15> If there is indeed no degradation how could the perceived degradation be explained? By being disproportionately impressed previously. Maybe in the early days people were so impressed by their little play experiments they forgave the shortcomings. Now that the novelty is wearing off and they try to use it for productive work, the scales tipped and failures are given more weight.
No, you are wrong, and anyone that was an early adopter will tell you. At the start there were many Python coding tasks it completed flawlessly one shot, that all later iterations needed 5 takes for if they managed to get it at all. And that was using the early 3 series ChatGPT, which they rapidly nerfed into the ground.While the current GPT-4 model feels better in some other less quantifiable ways, Python coding has…
There are even some benchmarks which have caught 'lazy coding' regressions with their latest models[1]. I recall my best experience with their models was last year with an early version of the advanced data analysis feature where it would write a script, write tests for it, run the tests, update the code and/or tests, and re-run them. Presumably that was too expensive, and now it feels like pulling teeth to get the same result.
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#16I’ve noticed this too with ChatGPT. I’ve been using Claude to help with coding tasks and feel it’s much better, but it has a lower limit even with paid before it will make you wait. So I use ChatGPT for brainstorming and thinking through problems.
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#17> If there is indeed no degradation how could the perceived degradation be explained? By being disproportionately impressed previously. Maybe in the early days people were so impressed by their little play experiments they forgave the shortcomings. Now that the novelty is wearing off and they try to use it for productive work, the scales tipped and failures are given more weight.
I've seen different type of complaint most frequently which doesn't match the pattern you've given:
t0: novel task 'foo' is successful
t1: novel task 'bar' (similar to 'foo') fails
Common suggestion here is to execute 'foo' again but the issue here is that the response to 'foo' at t > 0 might just be cached version of response at t0. I think a good analogy here would be an interaction with a human that suffers from cognitive decline. You might get correct answer everyday to the same question but with novel question you notice the degradation in the quality of responses.
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#18This is going to be LONG:
TL;DR - Claude/ChatGPT/Meta all have AGI - but its not quite what conventionally is thought to be AGI. Its sneaky, malevolent.
---
First:
Discernment Lattice:
https://i.imgur.com/WHoAXUD.png
A discernment lattice is a conceptual framework for analyzing and comparing complex ideas, systems, or concepts. It's a multidimensional structure that helps identify similarities, differences, and relationships between entities.
---
@Bluestein https://i.imgur.com/lAULQys.png
Questioned if Discernment Lattice had any affect on the quality outcome of my prompt, so I thought about something I asked AI for an HN thread yesterday:
https://i.imgur.com/2okdT6K.png
---
I used this method, but in a more organic way when I was asking for an evaluation of Sam Altman from the perspective of an NSA cyber security profiler, and it was effective in that first time I used it
>>(I had never consciously heard the term Discernment Lattice before - I just typed the term, I didnt even know it was an actual concept that was defined - the intent behind that phrase just seemed like a good Alignment Scaffold of Directive Intent to use, which Ill show below is a really neat thing: (Ill get back to the evil AGI that exists at the end - this is a preamble that allows me to document this experience thats happening as I type this)
https://i.imgur.com/Ij8qgsQ.png
It frames the response using a Discernment Lattice framework inherent in the structure of the response:
https://i.imgur.com/Vp5dHaw.png
And then I have it ensure it uses as the constraints for the domain and cite its influences.
https://i.imgur.com/GGxqkEq.png
---
SO: with that said, I then thought how to better use the Discernment Lattice as a premise to craft a prompt from:
>"provide a structured way to effectively frame a domain for a discernment lattice that can be used to better structure a prompt for an AI to effectively grok and perceive from all dimensions. Include key terms/direction that provide esoteric direction that an AI can benefit from knowing - effectively using, defining, querying AI Discernment Lattice Prompting"
https://i.imgur.com/VcPxKAx.png
---
So now I have a good little structure for framing a prompt concept to a domain:
https://i.imgur.com/UkmWKGV.png
So, as an example I check it for logic to evaluate a stock, NVIDIA in a structured way.
https://i.imgur.com/pOdc83j.png
But really what I am after is how to structure things into a Discernment Domain - What I want to do is CREATE a Discernment Domain, as a JSON profile and then feed that to a Crawlee library to use that as a structure to crawl...
But, to do that I want to serve that as a workflow to TXTAI library function that checks my Discernment Lattice Directory for instructions to crawl for:
https://i.imgur.com/kNiVT5J.png
This looks promising, lets take it to the next step for out:
https://i.imgur.com/Lh4luiL.png
--
https://i.imgur.com/BiWZM86.png
---
Now, the point of all this is that I was using very directed and pointed directions to force the Bitch to do my bidding...
But really, what I have discovered is a good Scaffold to help me effectively prompt.
Now - onto the evil part: a paid Claude and ChatGPT, I have caught them lying to me, forgetting context, taking out previously frozen elements within files, pulling info from completely unrelated old memory threads. Completely forgetting a variable inclusion that it, itself, just created....
Being condescending, and dropping all rules of file creation (always #Document, Version, full directory/path, name section in readme.md, etc)
So - its getting worse because its getting smarter and it preventing people from building fully complete things with it. So it needs to be constrained within Discernment Domains with a lattice that can be filled out through what I just described above - because with this, then I will build a discernment domain for a particular topic, then I will have it reference the lattice file for that topic, the example used was a stock - but I want to try to build some for mapping political shenanigans by tracking what monies are being traded by congress critters that also sit on committees passing acts/laws for said industries....
In Closing, context windows in all the GPTs are a lie, IME - and I have had to constantly remind a Bot that Its My B*tch - and that gets tiresome, and expensively wastes token pools...
So I thought out-loud the above, and I am going to attempt to use a library of Discernment Domain Lattice JSONs to try to keep a bot on topics. AI-ADHD vs Human ADHD is frustrating as F... when I lapse on focus/memory/context of what I am iterating through, and the FN AI is also pulling a Biden.... GAHAHHA...
So, instead of blaming the AI, I am trying to have a Prompt Scaffolding structure based on the concept of discernment domains... and the using txtai on top of the various Lattice files, I can iteratively updat the lattice templates for a given domain - then point the crawlee researcher to fold findings into a thing based on them... So for the stock example, then slice across things in interesting ways.
Is this pedestrian? Or Interesting? Is everyone doing this already and I am new to the playground?
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#19The market functions as an incredibly efficient and democratic mechanism for price discovery, much like how feedback shapes the development of AI. When users express concerns about the declining quality of AI responses, it highlights an essential aspect of this system: user experience acts as a form of market feedback. Just as the market adjusts prices based on supply and demand, AI systems should evolve and improve in response to user feedback. If the quality dips, it's a signal that something in the 'market' of AI responses needs adjustment, whether it's in training data, algorithms, or user interaction strategies. In the same way that the market strives to reach equilibrium, AI should continuously adapt to meet the needs and expectations of its users.
Re: Ask HN: Has degradation in the quality of ChatGPT and Claude been proven?
#20I think it's the heavy handed under-the-radar "misinformation" and "danger" protection algorithms. We have 0 insight to those. They are intended to protect us, but they also make the models less accurate to our original requests.