Viewing profile — timbilt

https://googlecloudplatform.github.io/scion/overview/

Yes, but in a case like this it's a neutral third-party running the benchmark. So there isn't a direct incentive for them to favor one lab over another. With public benchmarks we'r…

> Unlike many public benchmarks, the PR Benchmark is private, and its data is not publicly released. This ensures models haven’t seen it during training, making results fairer and …

anyone else concerned that training models on synthetic, LLM-generated data might push us into a linguistic feedback loop? relying on LLM text for training could bias the next mode…

Twitter thread about this by the author: https://x.com/jonasgeiping/status/1888985929727037514

Until we get real-time learning to work in production, every AI tool feels like it's getting dumber over time. It goes very quick from "wow this is magic" to starting to notice all…

The weirdness of LLMs is that they're so damn good at so many things but then you see these glaring gaps that instantly make them seem dumb. We desperately need benchmarks and eval…