21–30 of 149 posts
I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.
Nvidia leads in bandwidth and specialized architecture, but local inference takes off when it's usably fast at much lower cost and power consumption.
> Swiftlet was built in collaboration with Claude Code.
Did this really happen (some sort of working with Anthropic or Claude Code team) or is it some kind of requirement when you develop some software with Claude Code (I see the other author is: https://github.com/claude), or sort of reuse some of its parts?
Is it like someone saying "built in collaboration with VS Code" or ".. in collaboration with autocomplete plugin"?
Or merely a disclaimer about vibe-coding or AI written tool?
At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.
Isn’t it only writes that kill drives?
Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
I see this at the end of the README > Swiftlet was built in collaboration with Claude Code. Did this really happen (some sort of working with Anthropic or Claude Code team) or is it some kind of requirement when you develop some software with Claude Code (I see the other author is: https://github.com/claude ), or sort of reuse some of its parts? Is it like someone saying "built in collaboration with VS Code" or ".. i…
Earlier quoted context omitted.
What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.
The models themselves have far from plateaued. Maybe someone finds a way to get a really capable model down to, say, 12GB of ram. Then we'd be in business.
this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck. half an hour to process 10k tokens on an M5 seems... not great
Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
Earlier quoted context omitted.
Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.
8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…