471–480 of 682 posts
Earlier quoted context omitted.
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
What do you use instead?
https://omlx.ai https://vmlx.net
That said, it's been a few weeks since I've looked so maybe llama.cpp has those features now... they really do move that quickly.
will it run on 2x 5060Ti with 16GB each?
Earlier quoted context omitted.
Native dflash support on day 1 helps a lot! High quality speculative decoding speeds up a lot of agentic work.
You're right. I'm getting ~33tok/sec w/ dflash on it, even bursts up to 60tok/sec, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based) That's pretty respectable. Still working on optimizing and cleaning up before I push it.
Earlier quoted context omitted.
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
What do you use instead?
https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap
Earlier quoted context omitted.
Common phrase.
That commenter you're replying to knows that. The original commenter before them wrote "pulled the plug" which is different and doesn't quite apply here (actually implies the opposite of what they meant to say).
Earlier quoted context omitted.
There are arguments that the brain is quantum, as in parts of it locally using quantum effects. Which if true, might make a counter-argument, as there will be bigger data centers needed if the goal is to simulate the brain classically. On the other side, advancement in quantum computers would make current LLM inference much faster. Because of the extreme cooling needed, i dont think the energy demand would become les…
> the brain is quantum Doesn't that apply to everything that exists in the material world?
With the business model for API based LLMs looking iffy at best it seems like we’re heading back to the “server under your desk” era of IT again.
If you want to host the model on the server under your desk, you can. If you want to build a data center on-prem to host it, you can. If you want to pay a cloud provider to host it at their data center until you figure out how to scale it without their help, you can. It's like when people were first building commercial services to support Linux-based OSes, and people were also still hacking on it on local machines.
APIs may still have their place - maybe you just want to throw your devs a known quantity with all of the management built in - but it's not going to make Sam Altman a trillionaire, which is something anyone outside of the SV echo chamber could have figured out as soon as the first real competition to OpenAI emerged.
Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection - and Nginx collapsed that into a single box overnight? That moment for LLMs is near. It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation. And I think the data center buildout will end in car…
Everyone keeps repeating this who doesn’t understand the underlying technology. Small llms are still way more efficiently server on big GPUs. Sharing server capacity takes advantage of the massive parallel throughput and sharing of memory bandwidth. You are sharing the GPUs with thousands of concurrent users.