Viewing profile — oofbey

Yeah that was a clever bit of foresight there.

Really curious to see how ByteDance’s 10T model works out.

I think the Top500 benchmark is biased here. It’s a supercomputing benchmark which means it’s going to heavily weight SIMD and float64 performance, which are things that rarely mat…

The core idea of the RLM paper is to make a regular LLM act more like a coding agent - offload context to something external that needs to be explicitly queried instead of filling …

As agentic coding matures, this kind of project is the right direction. The mechanics of writing the code become less important. But having a language framework that naturally resi…

That’s for the consumer app / chatbot. For the api the terms are different: https://platform.kimi.ai/docs/agreement/modeluse

That depends. There are two wry different processes that both get called distillation. One is where you have a fully trained large model and you are converting it to a smaller mode…

These things take months to train. No chance this is a reaction to what just happened.

Correct: can't opt out of training. This is well documented. "Can't use for commercial purposes" - incorrect AFAICT. In what sense do you mean this? The open weight MIT version obv…

Agreed on the likely mechanism. I'm not sure "overfitting" is even the right description. These things are of course absurdly complicated, and evaluating their quality down to a si…

It’s not that. The tumors they create in mice are just really fragile compared to natural tumors. Natural tumors that actually cause problems have probably grown for years and lear…

The blog is highly suspect, but the study is real. That said it’s not a big deal. Curing cancer in a mouse model is not at all uncommon in new therapies. Mouse models like this are…

Another year, and OpenAI comes up with yet another naming scheme for their models. First it was integers (GPT2, GPT3). Then they added friendly names (remember Ada, Babbage, Curie,…

As somebody who has spent a lot more than 10 minutes trying to figure out why CORS was blocking what seemed legitimate, I sympathize with people doing the wrong thing, and disagree…

I think there’s a reasonable argument that a burst bubble will cause prices to drop. Prices are very high because they’re trying to justify these trillion dollar valuations on IP a…

Correct. We used to think that because NN optimization is non-convex there are all these local minima. Now we know that once you get past the very early parts of training from rand…

That’s a lot of negatives!

Nice breakdown. But data centers are increasingly responsible for their own electricity. As in Colossus.

It’s definitely for hardware reasons. They have been aggressively improving the vector math capabilities in their chips, but as anybody who has tried to run a local LLM will tell y…

Omg that is brilliant. I am so using this.

I cannot fathom why he brings up CPU SIMD as a potential comparative weakness on the NVIDIA Spark when it has teraflops of CUDA sitting right there.

Understanding and deciding to rely upon are not the same. Complexity is one of the biggest challenges in real world software systems. Keeping a subsystem’s responsibilities simple …

You could. But then you’re also building from scratch HA failover, backups, replica management, monitoring, etc - cloud vendor managed RDBMS come with lots of niceties. All of whic…

You underestimate the egos involved.