81–90 of 118 posts
In my day job we use GPT4 quite a bit and we shifted to GPT4 Turbo today. We got a 2-5% performance increase, and quite a bit of speed increase as well. Not to say that the parent post is incorrect, of course. Only that its not as cut and dry as a "GPT4 Turbo is distilled (read: watered down) GPT4".
The thing is why does the GPT-4 Turbo and the Updated GPT 3.5 Turbo have only an output of 4,096 tokens? Previous Model: gpt-3.5-turbo-16k, 16385 tokens context and completion (shared) New Model: gpt-3.5-turbo-1106, 16385 tokens context, 4096 tokens completion Previous Model: gpt-4, 8192 tokens context and completion (shared) New Model: gpt-4-1106-preview, 128000 tokens context, 4096 tokens completion Why would the s…
From my local test on a 13B model, output tokens are 20-30x more expensive than input tokens. So OpenAI's pricing structure is based on expectation that there's much more input than output tokens in an average response. It didn't matter too much if a small percentage of requests used all 4k tokens for output, but with 128k it's a different story.
> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
When did natural language became better for expressing development ideas than code? I know – when you don't know how to code in the first place. Then you will have to bet on all of the ambiguities of the language, cultural and meta-physical which words carry in order to hack your thing together instead of expressing yourself directly and explicitly.
Finally what is beautiful about strict code format we are so used to - it is truly the fastest and shortest path to get your thing done, in case you possess the knowledge needed.
> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
The problem is the discussed results are comparing proportions of a relatively small number - 67 questions. If you model this as a binomial distribution, then 62/67 which GPT4-turbo got gives a 95% confidence interval of the 'true' performance of 83.4% to 97.5%, ie it comfortably includes the proportion that GPT4 achieved (64/67=95.5%). I think the evidence from these tests are not strong enough to draw conclusions f…
I've used gpt4 turbo for some coding problems yesterday. It was worse. That's enough to draw conclusions for me.
The thing is why does the GPT-4 Turbo and the Updated GPT 3.5 Turbo have only an output of 4,096 tokens? Previous Model: gpt-3.5-turbo-16k, 16385 tokens context and completion (shared) New Model: gpt-3.5-turbo-1106, 16385 tokens context, 4096 tokens completion Previous Model: gpt-4, 8192 tokens context and completion (shared) New Model: gpt-4-1106-preview, 128000 tokens context, 4096 tokens completion Why would the s…