131–140 of 411 posts
Earlier quoted context omitted.
Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)
I think not much can run without a dedicated GPU
The fact that it runs at 15tk/s in power saving mode, and 30 in perf. mode blows my mind. I can run the model in the background, coding something for me in OpenCode, hosted in LMStudio, while doing something else. What a world we live in.
Having something close to human intelligence (at least for reasoning and code), running on a laptop, is amazing.
Earlier quoted context omitted.
All white collar work will be done by AI soon, there won’t be scaling down, just scaling up. I do food trucks and festivals and I’ve got AI doing so much of my non-meatspace work now that I’m buying $10-$20 a day in tokens. At that rate, hardware starts to look cheap, and I have less of a use case for it than most white collar workers.
I would also be interested in what you use AI for. Is it marketing? bureaucracy?
Marketing definitely. My food truck side is relatively high volume and it manages my kitchen and warehouse side, basically generating all of the instructions my employees follow, managing and updating my PoSes, creating signage assets for specials, etc.
Reels/posts production and managing ad spend.
Here’s a fun one. I switched payroll providers after several years and suddenly my unemployment insurance rate went from 0.8% to 12.75% which is borderline debilitating to me. I knew something was wrong but not what and I work a lot of hours and calling the state takes forever and is usually unhelpful.
ChatGPT figured out that it was a penalty rate and dug in for me. Turns out because I’m seasonal and have no payroll for one quarter of every year, Gusto did not file a quarterly wage report, so even though I owed nothing I was delinquent. Gotta love government, it’s the only place where you can be delinquent for $0.
ChatGPT filed the report and requested a retroactive re-rate, which they granted. I’m sure I would have figured this out eventually but it would have taken hours, or $5 in tokens.
I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought
To me, the amazing thing is that we now have local models that rival the reasoning of high end models from about a year ago. I hope this trend continues.
Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.
Earlier quoted context omitted.
It should just about be usable in 32GB.
On a consumer hardware it would be nicer. With no GPU/iGPU or a 6-8GB VRAM.
Besides, Macbooks with 32GB RAM is consumer hardware, just maybe on the higher end.
Earlier quoted context omitted.
The whole cloud story lies on two aspects: - Hyperscaling “we are going to serve billions of people in our applications”, which is becoming increasing unlikely as regional tech companies become more dominant than than the global one (this one is as much about geopolitics as technology) - Operations is hard, in which case non-frontier models should be increasingly capable. Devops for small-ish deployment is one of the…
I don’t disagree but saying “cloud will change in the next 8-20y-ish” is a bit of a non-argument, you’re not really stating any thesis to speak of; Change is a given over that time frame.
They will die, for some definitions of death -- I don't think they disappear, but they should be a niche, rather than the dominant doctrine.
“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.
I thought the same thing, and I generally do a lot of animation in my work, and the results in motion graphics with Qwen are impressive, I really fell in love with it