841–850 of 1001 posts
has anyone managed to use any of the AI models to build a complete 3D fps game using web GL or open GL?
I made a webgl copy of wolfenstein with prompt engineering in browser-based "Make a website" tool that was gemini-powered.
I truly do not understand what plan to use so I can use this model for longer than ~2 minutes. Using Anthropic or OpenAI's models are incredibly straightforward -- pay us per month, here's the button you press, great. Where do I go for this for these Google models?
I am paying for AI ultra - no idea how to use it in the CLI. It says i dont‘t have access. The google admin/payment backend is pure evil. What a mess.
My favorite benchmark is to analyze a very long audio file recording of a management meeting and produce very good notes along with a transcript labeling all the speakers. 2.5 was decently good at generating the summary, but it was terrible at labeling speakers. 3.0 has so far absolutely nailed speaker labeling.
My audio experiment was much less successful — I uploaded a 90-minute podcast episode and asked it to produce a labeled transcript. Gemini 3: - Hallucinated at least three quotes (that I checked) resembling nothing said by any of the hosts - Produced timestamps that were almost entirely wrong. Language quoted from the end of the episode, for instance, was timestamped 35 minutes into the episode, rather than 85 minute…
Understanding precisely why Gemini 3 isn't front of the pack on SWE Bench is really what I was hoping to understand here. Especially for a blog post targeted at software developers...
That's a "loss" I can deal with.
Earlier quoted context omitted.
I am paying for AI ultra - no idea how to use it in the CLI. It says i dont‘t have access. The google admin/payment backend is pure evil. What a mess.
My test a few hours ago. Ultra plan got me ~20 minutes with Antigravity using Gemini 3 Pro (Low) before zero out.
Here are my notes and pelican benchmark, including a new, harder benchmark because the old one was getting too easy: https://simonwillison.net/2025/Nov/18/gemini-3/
I updated my benchmark of 30 pelican-bicycle alternatives that I posted here a couple of weeks ago: https://gally.net/temp/20251107pelican-alternatives/index.ht... There seem to be one or two parsing errors. I'll fix those later.
I tried the first one and 5 Pro gives this: https://imgur.com/a/EhYroCE
Earlier quoted context omitted.
I'm not explaining myself right. Stockfish is a superhuman chess program. It's routinely used in chess analysis as "ground truth": if Stockfish says you've made a mistake, it's almost certain you did in fact make a mistake[0]. Also, because it's incomparably stronger than even the very best humans, sometimes the moves it suggests are extremely counterintuitive and it would be unrealistic to expect a human to find the…
I would love to examine Stockfish play that seemed extremely counterintuitive but which ended up winning. How can I do so? (I don't inhabit any of the current chess spaces so have no idea where to look, but my son is approaching the age where I can start to teach him...). That said, chess is such a great human invention. (Go is up there too. And texas no-limit hold'em poker. Those are my top 3 votes for "best human t…
If you want to see this against someone like Magnus, it is rare as super GMs do not spend a lot of time playing engines publicly.
But if you want to see them against a normal chess master somewhere between master and international master, it is every where. For e.g. this guy analyses his every match afterwards and you frequently here "oh I would never see that line":
https://www.youtube.com/playlist?list=PLp7SLTJhX1u6zKT5IfRVm...
(start watching around 1000+ for frequently seeing those moments)
I also asked it for the best play when in backgammon opponent rolls 6-1 (plays 13/7 8/7) and you roll 5-1. It starts alright with mentioning a good move (13/8 6/5) but continues to hallucinate with several alternative but illegal moves. I'm not too impressed.
Earlier quoted context omitted.
That is not entirely true. At least some of these tests (like HLE and ARC) take steps to keep the evaluation set private so that LLMs can’t just memorize the answers. You could question how well this works, but it’s not like the answers are just hanging out on the public internet.
Excuse my ignorance, how do these companies evaluate their models against the evaluation set without access to it?