61–70 of 118 posts
A little bit off topic question. When people are talking about costs with GPT, like the following link. Does the cost concern only apply to the API? If you’re using the WebUI and have a Plus account, is it always just the flat $20 amount? https://news.ycombinator.com/item?id=38193978
Earlier quoted context omitted.
Me: What's 6x6? You: 36 Me: You cheated! You just cited the answer you memorized! You should have started from addition. You: ...okay? 6+6=12, 12+6=18, 18+... Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms. You: ...you're being annoying, but okay? First axiom, we define 0 as... Me: You cheated again ! You memorized Peano Axioms! Jesus Christ, is there a…
But in this case it's not like that at all. They only saw the NAME of the problem. Like if I said "Page 23 of Mathbook Y, problem number 3". Which happens to be 6x6.
I have been writing function names and test names, and then telling gpt to fill in the test, which is usually does how I want (maybe with errors, but it tests the correct thing), and then I tell it to fill out the answers.
this is in a thing I'm building that's never been built, with names that I made up (but describe the functionality well)
It cannot have this spot memorized, I just invented it myself
Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…
Earlier quoted context omitted.
> Okay... Funny how forcing it to not CHEAT did not increase apparent ability. The article did the opposite. It forced the models to cheat to solve the problems. Which it did happily. It should have stated "there is no actual problem to solve here, you must supply a problem for me to solve". > It can code and it has memorized some coding questions are not mutually exclusive This I will give you. Many humans try to ch…
Me: What's 6x6? You: 36 Me: You cheated! You just cited the answer you memorized! You should have started from addition. You: ...okay? 6+6=12, 12+6=18, 18+... Me: You cheated again! You just have 6+6=12 memorized! You should make the rule of addition out of Peano axioms. You: ...you're being annoying, but okay? First axiom, we define 0 as... Me: You cheated again ! You memorized Peano Axioms! Jesus Christ, is there a…
I can't remember how many times I've googled, "how do I create a directory in Python?". Now bard often generates an inline answer for me.
GPT -4 Turbo is still in preview, maybe wait until it is fully released before judging?
The point of a preview phase is to test the model in real world use.
Now do a programming task that requires more than 32k of context and see who’s “better”. If you don’t bench mark that you cannot get an overall pic. GitHub copilot for example could benefit big from the increased context
GPT-4 Turbo is dramatically worse at one task I often try: Read the following passage from [new ML article]. Identify their assumptions, and tell me which mathematical operations or procedures they use depend upon these assumptions. GPT-4: Usually correctly identifies the assumptions, and often quotes the correct mathematics in its reply. GPT-4 Turbo: Sometimes identifies the assumptions, and is guaranteed to stop tr…
Earlier quoted context omitted.
All models are, including humain brain.
The human brain is a model?
Our experience of the world is a model executing.
Comparing the latest neuroscience to latest neural networks. They look and behave very similarly.
Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…
> I also notice that the instructions prompt that mentat uses seems to be inspired by the aider benchmark? Glad to see others adopting similar benchmarking approaches.
We were inspired by you to use Exercism as a benchmark, thank you! We will add attribution for that. We switched our original instruction prompts for that benchmark to be similar to Aiders to allow for fair comparison.
> After looking around a bit, there seems to be a bunch of aider code in your repo. Some attribution would be appreciated.
We have an unused implementation of your output response format (https://github.com/AbanteAI/mentat/blob/main/mentat/parsers/...), but I don't know what else you are seeing? We implemented that to compare with our response formats and didn't find much difference in performance.
Earlier quoted context omitted.
When people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem. For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.
Almost 2024 and people still can't accept that LLM can code...