651–660 of 692 posts
Earlier quoted context omitted.
This should become the new Apple's hardware and software play. I am hopeful about the new CEO
Nothing new about that play. They have been heading in this direction for a very long time now.
On my Agentic SQL benchmark it scores 19/25. That's... mediocre. It means performs worse than 3.1 Flash Lite Preview (22/25), is slower (367s vs 142s) and is more expensive (75c vs 2c). It is outperformed by Gemma4 26B-A4B in every way(!) https://sql-benchmark.nicklothian.com/?highlight=google_gemi... (Switch to the cost vs performance chart to see how far this is off the Pareto frontier)
I have a SQL agent and my tests with 3.5 are resulting in hitting query budget limits that have never been hit before. On average, to answer the same question, 3.5 is spending 10x more on SQL queries vs gemini-3-flash-preview.
The query patterns can be extremely degenerate too. E.g. the agent will hit the semantic layer tool to pull the schema, then run `SELECT * FROM table LIMIT 1`, which hits the query budget limit and fails.
I've only really been looking this morning, so I need to do a full eval, but the initial results match what your benchmark shows.
---
Side note: your benchmark has an issue. On Q1 medium the model returned gross margin of 0.127 instead of 12.7 (%), and the benchmark failed it. The failures on Q9 and Q21 are the same (I didn't check other questions). Nowhere in the prompt did you specify you wanted the values converted to percentage points and rounded.
If you asked me to write that SQL with that prompt, unless you were throwing it directly into a visualization I would format it the same way gemini-flash did. If I were pulling into a spreadsheet or vis tool this format is preferable because it's easier to format in a client application.
The other failures like Q21 incorrectly averaging the list price are correct failures.
Am I really so old that when someone says "Flash" my immediate response is... "consider HTML5 instead" ??
Earlier quoted context omitted.
Amazon was unprofitable for over a decade, and they were public. Theres no incentive to be profitable as a private company if you can continue to raise money. Ed Zitron and Gary Marcus are... confused.
But I've been told here -- over and over again -- that the cost of inference was going to go down as the technology matured. The trend lines are going in the opposite direction.
Earlier quoted context omitted.
Asking a model to improve its output is not one-shotting tho? My observation was that asking an llm to iterate and improve a response causes it to add more stuff, rather tha repair the broken stuff. And that model progress in general has the same pattern. This new model adds more details to its responses but continues to make mistakes at about the same rate.
The question was whether you were giving it the rendered image and using the model's visual modal capability, or feeding back in the textual SVG. It's hard to "imagine" what the rendered SVG looks like, for both humans and LLMs, so just iterating on text won't really be as useful of a test. But if you show it what it rendered, it might observe the bad-looking bicycle and be able to fix the text that way.
Earlier quoted context omitted.
I think the big 3 are cartelizing and starting to ratchet up costs. GPT5.5 is not easily distinguishable from 5.1. I would it be shocked if we hit the ceiling and everyone is quietly positioning for the exit.
I don't understand why everyone thinks there is a ceiling below human-level intelligence, when we have an existence proof that human-level intelligence is possible.
For those who would like to know the total and active parameter count of this model: even though Google doesn't disclose the model technicals, we can infer them within relatively tight margins based on what we do know. We know they serve the model on TPU 8i, which we have plenty of hard specs for (so we know the key constraints: total memory and bandwidth and compute flops). We can also set a ceiling on the compute c…
We've been really impressed with the performance of ~30B parameter class models and how close they are to the frontier from ~6-12 months ago, which begs the question, are the frontier labs really serving 10T parameter models? Seems unlikely. If these Gemini 3.5 numbers are accurate, then I'd wager GPT 5.5 and Opus 4.7 are a lot smaller than people have speculated, too. It's not that frontier labs can't create a 5T+ p…