131–140 of 259 posts
Earlier quoted context omitted.
Meme benchmarks like this and Strawberry are funny but very easy to game, I bet they're all over training sets nowadays.
If you train a model on the SVGs of pelicans on a bicycle that are out there already you're going to get a VERY weird looking pelican on a bicycle: https://simonwillison.net/tags/pelican-riding-a-bicycle/
I started doing some experimentation with this new Deep Think agent, and after five prompts I reached my daily usage limit. For $250 USD/mo that’s what you’ll be getting folks. It’s just bizarrely uncompetitive with o3-pro and Grok 4 Heavy. Anecdotally (from my experience) this was the one feature that enthusiasts in the AI community were interested in to justify the exorbitant price of Google’s Ultra subscription. I…
> It’s just bizarrely uncompetitive with o3-pro and Grok 4 Heavy. In my experience Grok 4 and 4 Heavy have been crap. Who cares how many requests you get with it when the response is terrible. Worst LLM money I’ve spent this year and I’ve spent a lot.
Ladies and Gentlemen, Here's Gemini Deep Think when prompted with: "Create a svg of a pelican riding on a bicycle" https://www.svgviewer.dev/s/5R5iTexQ Beat Simon Willison to it :)
Easily the best one yet!
Earlier quoted context omitted.
"I'm sorry but that wasn't a very interesting question you just asked. I'll spare you the credit and have a cheaper model answer that for you for free. Come back when you have something actually challenging."
Actually why not? Recognizing problem complexity as a fist step is really crucial for such expensive "experts". Humans do the same. And a question to the knowledgeable: does a simple/stupid question cost more in terms of resources then a complex problem? in terms of power consumption.
Even proof mining and the Harrop formula have to exclude disjunction and existential quantification to stay away from intuitionist math.
IID in PAC/ML implies PEM which is also intentionally existential quantification.
This is the most gentle introduction I know of, but remember LLMs are fundamentally set shattering, and produce disjoint sets also.
We are just at reactive model based systems now, much work is needed to even approach this if it ever is even possible.
[0] https://www.cmu.edu/dietrich/philosophy/docs/tech-reports/99...
Ladies and Gentlemen, Here's Gemini Deep Think when prompted with: "Create a svg of a pelican riding on a bicycle" https://www.svgviewer.dev/s/5R5iTexQ Beat Simon Willison to it :)
OK that is recognizably a pelican, pretty great!
Time for a leaderboard?
Earlier quoted context omitted.
It could be that your problem was too simple to justify the use of Deep Think. But yes, Google should have figured that out and used a less expensive mode of reasoning.
"I'm sorry but that wasn't a very interesting question you just asked. I'll spare you the credit and have a cheaper model answer that for you for free. Come back when you have something actually challenging."
Ladies and Gentlemen, Here's Gemini Deep Think when prompted with: "Create a svg of a pelican riding on a bicycle" https://www.svgviewer.dev/s/5R5iTexQ Beat Simon Willison to it :)
Easily the best one yet!
Pelican:
https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd....
Longer thread re gpt5:
https://old.reddit.com/r/OpenAI/comments/1mettre/gpt5_is_alr...
Asking AI to create 3D scenes like the example in the page seems like asking someone to hammer something with a screwdriver, we would need an AI compatible 3D software that either has easier to use voxels built in so it can create similar to pixel art, or easier math defined curves that can be meshed, either way AI just does not currently have the right tools to generate 3D scenes
Approach is analogous to Grok 4 Heavy: use multiple "reasoning" agents in parallel and then compare answers before coming back with a single response, taking ~30 minutes. Great results, though it would be more fair for the benchmark comparisons to be against Grok 4 Heavy rather than Grok 4 (the fast, single-agent model).
What makes you sure of that? From the article, > Deep Think pushes the frontier of thinking capabilities by using parallel thinking techniques. This approach lets Gemini generate many ideas at once and consider them simultaneously, even revising or combining different ideas over time, before arriving at the best answer. This doesn't exclude the possibility of using multiple agents in parallel, but to me it doesn't ne…
Earlier quoted context omitted.
Similar complaints are happening all over reddit with the Claude Code $200/mo plan and Cursor. The companies with deep VC funding have been subsidizing usage for a year now, but we're starting to see that bleed off. I think the primary concern of this industry right now is how, relative to the current latest generation models, we simultaneously need intelligence to increase, cost to decrease, effective context window…
Agreed, and big context windows are key to mass adoption in wider use cases beyond chatbots (random ex: in knowledge management apps, being able to parse the entire note library/section and hook it into global AI search), but those use cases are decidedly not areas where $200 per month subscriptions can work. I'll hazard to say that cost and context windows are the two key metrics to bridge that chasm with acceptable…
For example, a chat bot doing recipe work should have a RAG DB that, by default, returns entire recipes. A vector DB is actually not the solution here, any number of traditional DBs (relational or even a document store) would work fine. Sure do a vector search across the recipe texts, but then fetch the entire recipe from someplace else. Current RAG solutions can do this, but the majority of RAG deployments I have seen don't bother, they just abuse large context windows.
Which looks like it works, except what you actually have in your context window is 15 different recipes all stitched together. Or if you put an entire recipe book into the context (which is perfectly doable now days!), you'll end up with the chatbot mixing up ingredients and proportions between recipes because you just voluntarily polluted its context with irrelevant info.
Large context windows allow for sloppy practices that end up making for worse results. Kind of like when we decided web servers needed 16 cores and gigs of RAM to run IBM Websphere back in the early 2000s, to serve up mostly static pages. The availability of massive servers taught bad habits (huge complicated XML deployment and configuration files, oodles of processes communicating with each other to serve a single page, etc).
Meanwhile in the modern world I've ran mission critical high throughput services for giant companies on a K8 cluster consisting of 3 machines each with .25 CPU and a couple hundred megs of RAM allocated.
Sometimes more is worse.
