Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

github.com

Post reply on HN
Print view View original

111–120 of 149 posts

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#111

Post by JKCalhoun »

croes wrote: post #73
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

So every script kiddie gets its Mythos to hack sides and scammers don’t need AI services anymore. That time won’t be as much fun as you think

If that's the natural outcome then the future you're describing is inevitable.

Oh well, maybe the last one to leave the internet can turn the lights off.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#112

Post by pbronez »

gizmodo59 wrote: post #96

The web and connecting to other services is very important for almost all of my use cases. While I believe we are going to get better and faster models, the web index is certainly not downloadable and maintainable for 99.99% of the folks who are able to use local models. Any good solutions exist?

There are many search APIs available, I like Kagi's.

Microsoft and Amazon both provide web snapshot services that purport to give you a kind of agent-first internet archive. You can approximate something like that using common crawl, but it's a huge amount of data. Downloading the internet is impossible or a bad idea for almost everyone.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#113

Post by dghlsakjg »

Azantys wrote: post #85
apimade wrote: post #20

Earlier quoted context omitted.

8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…

Here you have shown yourself that progress slows down and doesnt speed up. 8.9/0.35 = ~25x more performance in 10 years from 2006 to 2016. 104.8/8.9 = ~12x more performance in 10 years from 2016 to 2026. Growth has dropped 50%.

That isn't deceleration, you've just chosen a very selective way to compare. If you use time as a denominator, which is kind of intrinsic when talking about rates of acceleration, you get a very different result. If you graphed .3, 8.9, and 104.4 on the y axis, with years on the x axis, it would be pretty clear that there was in increase in the rate of progress.

We went from adding 8 teraflops in a decade, to adding almost 100 the next decade. If we add "only" 400 more teraflops in the next decade the graph will make that initial growth look flat in comparison, even though your math would show that we are basically stalled out.

It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#114

Post by morgoo »

gizmodo59 wrote: post #96

The web and connecting to other services is very important for almost all of my use cases. While I believe we are going to get better and faster models, the web index is certainly not downloadable and maintainable for 99.99% of the folks who are able to use local models. Any good solutions exist?

If you're really into self-hosting I've been experimenting with SearXNG and early signs are promising

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#115

Post by dghlsakjg »

dd8601fn wrote: post #41
greggsy wrote: post #21

Earlier quoted context omitted.

The innovation curve doesn’t always reward the innovators. It pays to let others test the waters before you.

Especially if you’re Apple and your expertise and reputation is in making privacy conscious devices and cloud services that are exceptionally user friendly. Then you go to the AI labs and say, “We see you’ve spent a trillion dollars on that model… we’ll write you a check today to let us run it how we want.” Skips the ugly horse race and massive risk while focusing on what they do better than anyone. And they’re the o…

Apple has a real knack for not being first mover but being “best” mover. They didn’t invent tablets or mp3 players or thin laptops or ARM laptops, but once they entered those markets they absolutely became the definitional product for the entire category.

We don’t talk about AVP.

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#117

Post by Hugsbox »

pizza234 wrote: post #80
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

> but this is how progress happens. This is progress in the same way that a man climbing a tree is making progress toward reaching the moon. This project is essentially the MoE-of-the day, with some platform-related optimizations. > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips. That won't happen. Projects like this just give the illus…

"It will never be possible to have a computer in your home. You will always have to buy computer time and access via dumb terminals. Technology will never again advance."

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#118

Post by ac29 »

JKCalhoun wrote: post #110
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

Agree—and I want to see this functionality move to the larger platforms that everyone uses like LM Studio.

llama.cpp has used mmap by default for years

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#119

Post by ac29 »

CircuitSeuss wrote: post #19
Mistletoe wrote: post #9

Earlier quoted context omitted.

What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.

A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at. Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a fa…

> Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]

Unified memory has existed for decades in the PC space, Apple didnt invent it.

And the test you linked to has nothing to do with unified memory, its a web browser benchmark (almost entirely constrained by single threaded CPU performance that Apple better than competitors at).

Re: Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

#120

Post by trymas »

pizza234 wrote: post #80
dghlsakjg wrote: post #6

I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

> but this is how progress happens. This is progress in the same way that a man climbing a tree is making progress toward reaching the moon. This project is essentially the MoE-of-the day, with some platform-related optimizations. > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips. That won't happen. Projects like this just give the illus…

it’s equivalent take to laugh at first transformers 9 years ago, because they were shit and hardware requirements were immense.

IMHO it’s a matter of time until we (consumers) will get the hardware (maybe coupled maybe even more novel techniques). Though I expect it will take another 10 years or more.

Post reply on HN