221–230 of 1001 posts
Earlier quoted context omitted.
Why? This seems like a reasonable task to benchmark on.
Sure, reasonable to benchmark on if your goal is to find out which companies are the best at stealing the hard work of some honest, human souls.
Earlier quoted context omitted.
I would have used OpenSCAD for that purpose.
I started with a lighter weight solution (JSCAD) first and quickly hit the limitations. So I wanted to explore the other side of it - fully complex over the top software (blender). I guess openscad would be a sweet spot in the middle. Good shout, might experiment.
I'm sure this is a very impressive model, but gemini-3-pro-preview is failing spectacularly at my fairly basic python benchmark. In fact, gemini-2.5-pro gets a lot closer (but is still wrong). For reference: gpt-5.1-thinking passes, gpt-5.1-instant fails, gpt-5-thinking fails, gpt-5-instant fails, sonnet-4.5 passes, opus-4.1 passes (lesser claude models fail). This is a reminder that benchmarks are meaningless – you…
and models are still pretty bad at playing tic-tac-toe, they can do it, but think way too much it's easy to focus on what they can't do
Ask it to implement tic-tac-toe in Python for command line. Or even just bring your own tic-tac toe code.
Then make it imagine playing against you and it's gonna be fast and reliable.
I'm sure this is a very impressive model, but gemini-3-pro-preview is failing spectacularly at my fairly basic python benchmark. In fact, gemini-2.5-pro gets a lot closer (but is still wrong). For reference: gpt-5.1-thinking passes, gpt-5.1-instant fails, gpt-5-thinking fails, gpt-5-instant fails, sonnet-4.5 passes, opus-4.1 passes (lesser claude models fail). This is a reminder that benchmarks are meaningless – you…
>>benchmarks are meaningless No they’re not. Maybe you mean to say they don’t tell the whole story or have their limitations, which has always been the case. >>my fairly basic python benchmark I suspect your definition of “basic” may not be consensus. Gpt-5 thinking is a strong model for basic coding and it’d be interesting to see a simple python task it reliably fails at.
Earlier quoted context omitted.
What prompt do you use for that?
I just tried "analyze this audio file recording of a meeting and notes along with a transcript labeling all the speakers" (using the language from the parent's comment) and indeed Gemini 3 was significantly better than 2.5 Pro. 3 created a great "Executive Summary", identified the speakers' names, and then gave me a second by second transcript: [00:00] Greg: Hello. [00:01] X: You great? [00:02] Greg: Hi. [00:03] X: I…
I'm sure this is a very impressive model, but gemini-3-pro-preview is failing spectacularly at my fairly basic python benchmark. In fact, gemini-2.5-pro gets a lot closer (but is still wrong). For reference: gpt-5.1-thinking passes, gpt-5.1-instant fails, gpt-5-thinking fails, gpt-5-instant fails, sonnet-4.5 passes, opus-4.1 passes (lesser claude models fail). This is a reminder that benchmarks are meaningless – you…
>>benchmarks are meaningless No they’re not. Maybe you mean to say they don’t tell the whole story or have their limitations, which has always been the case. >>my fairly basic python benchmark I suspect your definition of “basic” may not be consensus. Gpt-5 thinking is a strong model for basic coding and it’d be interesting to see a simple python task it reliably fails at.
I like to compare them using chathub using the same prompts
Gemini still calls me "the architect" in half of the prompts. It's very cringe.
Just generated a bunch of 3D CAD models using Gemini 3.0 to see how it compares in spatial understanding and it's heaps better than anything currently out there - not only intelligence but also speed. Will run extended benchmarks later, let me know if you want to see actual data.
Just hand sketched what 5 year old would do on the paper - the house, trees, sun. And asked to generate 3d model with tree.js. Results are amazing! 2.5 and 3 seems way way head.
2.5 stands between GPT-5 and GPT-5.1, where GPT-5 is the best of the 3.
In preliminary evals Gemini 3 seems to be way better than all, but I will know when I run extended benchmarks tonight.
Earlier quoted context omitted.
I asked a semi related question in a different thread [0] -- is the basic idea behind your benchmark that you specifically keep it secret to use it as an "actually real" test that was definitely withheld from training new LLMs? I've been thinking about making/publishing a new eval - if it's not public, presumably LLMs would never get better at them. But is your fear that generally speaking, LLMs tend to (I don't want…
> if it's not public, presumably LLMs would never get better at them. Why? This is not obvious to me at all.
I truly do not understand what plan to use so I can use this model for longer than ~2 minutes. Using Anthropic or OpenAI's models are incredibly straightforward -- pay us per month, here's the button you press, great. Where do I go for this for these Google models?