The truth is that the vast majority of FAANG engineers making high six figures are only good at deterministic work. They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. Inside these companies, the massive tech systems built, are actually generally terrible, but they pile on legions of engineers to fix the problems. This…
Meta got caught gaming AI benchmarks
71–80 of 171 posts
Re: Meta got caught gaming AI benchmarks
#72The truth is that the vast majority of FAANG engineers making high six figures are only good at deterministic work. They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. Inside these companies, the massive tech systems built, are actually generally terrible, but they pile on legions of engineers to fix the problems. This…
> They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. You haven't been keeping up. Less than 2 weeks ago, Google released a model that has crushed the competition, clearly being SotA while currently effectively free for personal use. Gemini 2.0 was already good, people just weren't paying attention. In fact 1.5 pro was…
Re: Meta got caught gaming AI benchmarks
#73The truth is that the vast majority of FAANG engineers making high six figures are only good at deterministic work. They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. Inside these companies, the massive tech systems built, are actually generally terrible, but they pile on legions of engineers to fix the problems. This…
There is no system to pitch an idea as opening new frontiers - all ideas must be able to optimize some number that leadership has already been tricked into believing is important.
Re: Meta got caught gaming AI benchmarks
#74I think it's most illustrative to see the sample battles (H2H) that LMArena released [1]. The outputs of Meta's model is too verbose and too 'yappy' IMO. And looking at the verdicts, it's no wonder by people are discounting LMArena rankings. [1]: https://huggingface.co/spaces/lmarena-ai/Llama-4-Maverick-03...
Re: Meta got caught gaming AI benchmarks
#75The truth is that the vast majority of FAANG engineers making high six figures are only good at deterministic work. They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. Inside these companies, the massive tech systems built, are actually generally terrible, but they pile on legions of engineers to fix the problems. This…
A whole lot of opinion there, not a whole lot of evidence.
Re: Meta got caught gaming AI benchmarks
#76I tried to make Studio Ghibli inspired images using presumably their new models. It was ass.
Every other player: Black Forest Labs' Flux, Stability.ai's Stable Diffusion, and even closed models like Ideogram and Midjourney, are all on the path to extinction.
Image generation and editing must be multimodal. Full stop.
Google Imagen will probably be the first model to match the capabilities of 4o. I'm hoping one of the open weights labs or Chinese AI giants will release a model that demonstrates similar capabilities soon. That'll keep the race neck and neck.
Re: Meta got caught gaming AI benchmarks
#77The truth is that the vast majority of FAANG engineers making high six figures are only good at deterministic work. They cant produce new things, and so meta and google are struggling to compete when actual merit matters, and they cant just brute force the solutions. Inside these companies, the massive tech systems built, are actually generally terrible, but they pile on legions of engineers to fix the problems. This…
Re: Meta got caught gaming AI benchmarks
#78Next on Matt Levine’s newsletter: Is Meta fudging with stock evaluation-correlated metrics? Is this securities fraud?
Re: Meta got caught gaming AI benchmarks
#79Earlier quoted context omitted.
I don't know about Llama 4. Competition is intense in this field so you can't expect everybody to be number 1. However, I think the performance culture at Meta is counterproductive. Incentives are misaligned, I hope leadership will try to improve it. Employees are encouraged to ship half-baked features and move to another project. Quality isn't rewarded at all. The recent layoffs have made things even worse. Skilled…
I agree. I think of it like a car engine. You can push it up to a certain RPM and it will keep making more and more power. Above that RPM, the engine starts to produce less power and eventually blows a gasket. I think the performance-based management worked for a while because there were some gains to be had by pushing people harder. However, they’ve gone past that and are now pushing people too hard and getting wors…
Re: Meta got caught gaming AI benchmarks
#80tech companies competing over something that is losing them money is the most bizarre spectacle yet.