Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1-Bit Bonsai Image 4B Image Generation for Local Devices

prismml.com

Post reply on HN
Print view View original

51–60 of 214 posts

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#51

Post by smallerize »

a1o wrote: post #25
mkl wrote: post #22

Earlier quoted context omitted.

The white paper says "mean-active memory pressure down to 1.95 GB for 1-bit Bonsai Image 4B and 2.38 GB for Ternary Bonsai Image 4B". Storage is on the linked page, and is about half that.

That is very low, looks like it should run in base MacMini M4 with 16GB RAM. I understand it is not released yet? What sort of harness is necessary for this type of model? (I have only used coding agents through GH Copilot in VS Code, the JetBrains AI tool and Pi, this last one was sort of a pain to setup…)

They are released in the Bonsai Studio software and also https://huggingface.co/collections/prism-ml/bonsai-image

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#52

Post by wongarsu »

neals wrote: post #23
lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

I'm curious how hardware and power cost would stack up to subscription cost

For open models, usually not well. You get 5+ providers competing on cost, all with cheaper electricity and better hardware utilization than your local setup

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#53

Post by woadwarrior01 »

moralestapia wrote: post #38
mft_ wrote: post #32

Genuine question: is this solving a real problem? IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on. Oka…

Genuine question: doesn't it blow your mind that there exists a 1 Gigabyte file/program that can generate any image you can think of just from a rough description of it?

Where are you getting the 1 Gigabyte number from?

Their 1-bit quantized Diffusion Transformer is just under 1 GB. You also need the text-encoder (4-bit quantized) and VAE (unquantized) for inference and their combined weight is ~3.42 GB.

TBF, even at that size it's no less mind blowing.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#54

Post by c0rruptbytes »

jjcm wrote: post #47
c0rruptbytes wrote: post #41

Earlier quoted context omitted.

ideally if ternary models work, the math is extremely easy for computers (addition/subtraction vs 16 bit multiplication)

Not quite as I understand it. The ternary approach bonsai uses leverages a FP16 scaling factor that each value in the ternary maps to. You're still using 16 bit multiplication, it's just that the weights are far more compressed.

fair, i think i was referring more to 1.58 bit architecture in general since the original paper (Figure 3) shows that we eliminate FP16 multiplication and addition just for INT8 addition. I need to dive deeper into bonsai overall if it differs

https://arxiv.org/pdf/2402.17764

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#55

Post by kiicia »

mft_ wrote: post #32

Genuine question: is this solving a real problem? IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on. Oka…

It’s like asking how did Memoji generation on iPhone solved a real problem?

It does not need to directly solve any particular problem to be overall good for consumers, by putting pressure to all those subscription based solutions… at least it’s private and does not require you to provide all your data…

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#56

Post by mirekrusin »

barnas2 wrote: post #34
lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

A company called Taalas is working on something like that. Not Opus4.6 quality, but I'm sure they're targeting larger models. Currently they're using a LLama 8B model. It runs at ~17k tokens per second, and you can test it at https://chatjimmy.ai/ .

It starts to be interesting when latency is better than average website.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#57

Post by lumost »

bigmadshoe wrote: post #17
lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

Can you give an example of such a problem?

"Design me a 3d printable rocket engine for a hobby rocket project. Verify it's design in a full simulation. Iterate until it works reliably in simulation based on a verified printable design on a consumer laser sintering device (or substitute contract manufacture for under 1000 dollars)."

This is a hobby version of a project, but you can imagine commercial versions of the same prompt for new databases, genomics studies, material analysis, operating systems etc.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#58

Post by mirekrusin »

goofy_lemur wrote: post #46
lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

Ok heres the thing you will nevwr be able to truly do this due to logic. Logically five people pooling their resources beats one guy. therefore datacenters will always win because they get higher time utilization. so forget it. I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware. you will get better hardware though, cut…

Just like cloud is "cheaper" than colo/metal, right?

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#59

Post by lumost »

goofy_lemur wrote: post #46
lumost wrote: post #10

I actually can’t wait for the future where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription. There are many problems I want to work on which require billions of tokens. These are completely inaccessible without corporate project sponsorship at the moment. An asic generation machine which can pump out a few 10s of thousands of tokens per second at opus4.6 quality is more than…

Ok heres the thing you will nevwr be able to truly do this due to logic. Logically five people pooling their resources beats one guy. therefore datacenters will always win because they get higher time utilization. so forget it. I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware. you will get better hardware though, cut…

Laptops/desktops are cheaper per flop than any datacenter hardware by a good order of magnitude.

The problem is that expectations rise in datacenters, hardware/power/security/availability guarantees cost real money. Then the operator providing these guarantees expects some margin.

You can see this most clearly with "developer desktops", a gcp instance costs about 10x a hetzner instance which costs between 5 and 10x the same hardware sitting in the back of an office somewhere. While all of these premiums matter for 24/7 systems under active development, they don't really matter for ephemeral small scale workloads.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#60

Post by zamadatix »

c-hendricks wrote: post #48
goofy_lemur wrote: post #46

Earlier quoted context omitted.

Ok heres the thing you will nevwr be able to truly do this due to logic. Logically five people pooling their resources beats one guy. therefore datacenters will always win because they get higher time utilization. so forget it. I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware. you will get better hardware though, cut…

> cutting edge will always be cloud Don't think anyone was refuting that? And of course when you pool resources you have access to more resources.

They just mean this part: "where I upgrade hardware in order to upgrade my ai as an alternative to an expensive subscription."

Upgrading local hardware will remain the more expensive alternative to the subscription regardless what the relative cost of running the models themselves are. If the local hardware to do so becomes affordable then the subscription will be even more affordable, not expensive.

At least for these kinds of mega tasks. For more micro task we will always end up with unutilized local compute we already purchased which will be "free" since we already paid for non-AI reasons (e.g. a gaming GPU while not gaming).

Post reply on HN