Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1-Bit Bonsai Image 4B Image Generation for Local Devices

prismml.com

Post reply on HN
Print view View original

41–50 of 214 posts

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#41

Post by c0rruptbytes »

mft_ wrote: post #32

Genuine question: is this solving a real problem? IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on. Oka…

ideally if ternary models work, the math is extremely easy for computers (addition/subtraction vs 16 bit multiplication)

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#42

Post by ArchieScrivener »

moralestapia wrote: post #38
mft_ wrote: post #32

Genuine question: is this solving a real problem? IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on. Oka…

Genuine question: doesn't it blow your mind that there exists a 1 Gigabyte file/program that can generate any image you can think of just from a rough description of it?

[dead]

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#43

Post by jjcm »

neals wrote: post #23
lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

I'm curious how hardware and power cost would stack up to subscription cost

I did an estimate of that if you're interested: https://x.com/pwnies/status/2028831699736637912

The TL;DR though is that a 10-15b param model baked into an ASIC with the latest fab tech would take around 62W of power draw when active. At ~10k+ t/s though it likely would only be active for short bursts of time. It'd fit perfectly fine within the thermal envelope of a laptop.

The approach makes a lot of sense. Once you get to those speeds, latency of the network becomes one of the bigger bottlenecks, so local has a real advantage over a subscription.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#44

Post by jjcm »

bigmadshoe wrote: post #17
lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

Can you give an example of such a problem?

Decompiling a binary and recreating the source, doing a full line-by-line security audit, always-on agents monitoring state minute-by-minute, etc.

I would very easily find ways to hit that level of token usage if it was cheaper/faster.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#45

Post by goofy_lemur »

mft_ wrote: post #32

Genuine question: is this solving a real problem? IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on. Oka…

Yes its a huge deal because these are starting to get bound by memory bandwidth not compute. therefore one bit wirfhts stream way faster leading to substantially better results. At least thats what Id guess!

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#46

Post by goofy_lemur »

lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

Ok heres the thing you will nevwr be able to truly do this due to logic.

Logically five people pooling their resources beats one guy.

therefore datacenters will always win because they get higher time utilization.

so forget it.

I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware.

you will get better hardware though, cutting edge will always be cloud

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#47

Post by jjcm »

c0rruptbytes wrote: post #41
mft_ wrote: post #32

Genuine question: is this solving a real problem? IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on. Oka…

ideally if ternary models work, the math is extremely easy for computers (addition/subtraction vs 16 bit multiplication)

Not quite as I understand it. The ternary approach bonsai uses leverages a FP16 scaling factor that each value in the ternary maps to. You're still using 16 bit multiplication, it's just that the weights are far more compressed.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#48

Post by c-hendricks »

goofy_lemur wrote: post #46
lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

Ok heres the thing you will nevwr be able to truly do this due to logic. Logically five people pooling their resources beats one guy. therefore datacenters will always win because they get higher time utilization. so forget it. I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware. you will get better hardware though, cut…

> cutting edge will always be cloud

Don't think anyone was refuting that?

And of course when you pool resources you have access to more resources.

Post reply on HN