Gemini 3.8 Flash is top of the Redactle LLM benchmark but so was Gemini 3.7 Flash. Both one shot all puzzles in the evals though 3.8 is just a bit faster. It also does the evals cheaper and faster than almost all the other models I've tried.
Gemini 3.8 Flash and 3.8 Flash Cyber
421–430 of 699 posts
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#422Earlier quoted context omitted.
Do they officially support you using your subscription in other harnesses?
No, and Google actively bans people for using their subscription from other harnesses via various proxies/gateways. To preempt certain replies, yes, I know you can pay API prices and use whatever harness you want.
Shame. Thanks for the info.
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#423I've been using Gemini 3.7 for my personal trip planning app. Across multiple benchmarks, it ranks higher on everything I tried: - Real world knowledge (when a thing opens and closes, the geographic region, historical facts). It's also the best at taking a cluster of places and working out a visiting order. - Photo ranking (which photo should be the hero). Gemini can tell whether a photo is of the thing or of the vie…
I stopped using Gemini a few months ago because it would often just (partially) reply literal nonsense to me.
Think 2023 style ChatGPT. Something like “to open a document on your Mac click File > Open docurrrar” - like it suddenly forgot it had to produce actual words.
Overall I enjoyed its speed and comprehensiveness. But those occurrences of nonsense just made it feel like a great car that once a month just stops in the middle of the highway.
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#424Earlier quoted context omitted.
The Gemini models have openly trained for SVG output, apparently with a specialism on animals in forms of transport! https://twitter.com/JeffDean/status/2024525132266688757
Community effort happening here to build the ideal dataset: https://github.com/scosman/pelicans_riding_bicycles
Come on, don't provide the smoking gun that shows how to draw a pelican riding a bicycle. If it's on the public internet it will end up in training data and invalidate this important LLM capability benchmark.
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#425"The knowledge cutoff date for Gemini 3.8 Flash is March 2026 – users can expect updated information for some domains while in others they may experience the model’s knowledge is limited to January 2025 (in line with the Gemini 3 Model Family)." Kind of wild that they haven't (successfully) pretrained a base model since Jan-25.
I'm curious if the knowledge cutoff is important, when the interface (Gemini app) can search online for recent information. Is there a big advantage to having everything internal?
You don't need everything internal, but having some idea of recent events is useful. If you ask it to implement some local AI there's a decent chance it will try to use qwen 2.5 without wondering if anything better came out since
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#426Earlier quoted context omitted.
As someone who has stubbornly stuck with Claude Code, what's a good harness for Gemini models?
Pi [1] is amazing. Since using it I've felt no need to switch harnesses anymore. Or choose Oh My PI [2] for batteries included [1] https://github.com/earendil-works/pi [2] https://github.com/can1357/oh-my-pi
Can you use Pi with a Google Pro AI sub or do you need to use API billing?
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#427A company with 400+B revenue from software cannot build a usable command line cli for its vital AI model?
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#428So 89.4% on Terminal Bench 2 but only 19.1% on Tbench 4. Opus 5 is 89.1%/51.8%.
I was thinking the same, obvious suspicion is they benchmaxed it on older bench.
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#429Earlier quoted context omitted.
As someone who has stubbornly stuck with Claude Code, what's a good harness for Gemini models?
Antigravity is probably the best of the bunch I've tried. I'd say it's pretty comparable to Claude Code (I use both daily).
Antigravity which lacks an auto approve mode? Not really comparable to Claude Code when you're looking to run a team of agents from my experience.
Re: Gemini 3.8 Flash and 3.8 Flash Cyber
#430Pelicans (thinking effort high, medium, low): https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - high cost 8.9742 cents Here are the 3.7 pelicans for comparison: https://tools.simonwillison.net/markdown-svg-renderer.html?u... - high cost 8.4387 cents (I think thinking level low is a regression on 3.8 compared to 3.7.)
Impressive pelicans!