351–360 of 652 posts
Earlier quoted context omitted.
> You at least need a capable machine, so that's not 0 monetary investment It is 0 monetary investment if I already have said machine lying around doing nothing. Which is exactly the story OP talked about.
But most people don't have an RTX 5090 lying around, so the story doesn't apply to them, right?
Earlier quoted context omitted.
What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way
Yeah, here I am sitting deeply deeply deeply regretting not buying couple CMP 170HX at $200 or $350, knowing I could just flip them ethically at purchase price if nothing came of it... I could have just casually built a 128GB dual A100 local AI monster
This makes me wonder if AI companies even have a MOAT in the first place. All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days. And that makes me wonder if the trill…
The first is the compute. OpenAI and Anthropic secured huge amounts of compute, Google, Meta and xAI have their own huge datacenters. Now anyone can rent some cloud machines and start serving Kimi K3, but it's going to be impossible to get to a similar scale as the big 5 above. And inference has economies of scale: the more people you serve in parallel, the more efficient you are.
The second is the data. By now (and maybe even by one year ago), all the data on the internet has been used for training. You need new data. The big AI companies sit on top of trillions or quadrillions of tokens that they have generated over the years. They can use that to train new models. That data is gold, and the proof is that SpaceX was happy to pay $60B to acquire Cursor.
If you want to overtake the frontier labs, you have 2 options: use their models to generate synthetic data, and provide lots of (cheap, maybe below cost) inference to generate your own new data. The frontier labs know about the first, and I'm sure they try to limit how much others milk their models. As for the second, that's the "honest" way to compete, but it's not easy.
This makes me wonder if AI companies even have a MOAT in the first place. All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days. And that makes me wonder if the trill…
> All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.
Greedy decoding a single batch in most libraries will give you mostly deterministic outputs. Higher batch sizes can increase variance.
But all of this is down to CUDA and/or kernel implementation issues.
Earlier quoted context omitted.
Is it? OpenRouter shows DeepInfra being cheaper than DeepSeek directly https://openrouter.ai/deepseek/deepseek-v4-flash-20260731#pr...
DeepInfra's Cache Read is 6.5 times more expensive than DeepSeek.
Earlier quoted context omitted.
Strix Halo is the unified memory platform from AMD. Similar to the DGX Spark from NVIDIA or the M series Macs. I personally have the Framework Desktop, but there's also systems from other brands like Bosgame
You can also get it in a laptop form factor that feels like a MBP with a nicer keyboard if you get an HP Zbook G1A! Huge fan of that thing, it's th e Linux MBP I've always wanted.
Earlier quoted context omitted.
Google figured this out with their paper from 2023, We have no moat and neither does OpenAI . The moat now is the harness and being able to recursively self improve from RLHF, a great example is how Grok used to be pretty bad but since SpaceX bought Cursor, they used that data to train Grok 4.5 which is now very competent at coding and even exceeds frontier models in certain benchmarks. https://www.semianalysis.com/p…
Moat is not the harness. Harness itself is temporary until the models get better and slowly the code in harness will go down. Note that the biggest GPU providers in the world are the hyper scalers and even they couldn’t allocate more if you pay for it. Because the rich companies and well funded ones are gobbling them up to the point where if tomorrow a 5T model that smokes every other model in the world is released y…
It was a matter of time for China to catch up with the US. In terms of infrastructure, manufacturing, and engineering workforce, China has the upperhand and I foresee them becoming the SOTA leaders. Maybe if the US wasn't so busy gatekeeping and keeping things proprietary, they would've had more trust from the open source community.
This makes me wonder if AI companies even have a MOAT in the first place. All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history so that it can process it. LLMs do not learn or remember anything, which makes it super easy for users to switch LLMs on the fly. Most popular AI frameworks, make this a one-liner change these days. And that makes me wonder if the trill…