Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1-Bit Bonsai Image 4B Image Generation for Local Devices

prismml.com

Post reply on HN
Print view View original

61–70 of 214 posts

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#61

Post by IanCal »

jjcm wrote: post #43
neals wrote: post #23

Earlier quoted context omitted.

I'm curious how hardware and power cost would stack up to subscription cost

I did an estimate of that if you're interested: https://x.com/pwnies/status/2028831699736637912 The TL;DR though is that a 10-15b param model baked into an ASIC with the latest fab tech would take around 62W of power draw when active. At ~10k+ t/s though it likely would only be active for short bursts of time. It'd fit perfectly fine within the thermal envelope of a laptop. The approach makes a lot of sense. Once you…

Is latency of the network that noticeable? Aren’t we talking low hundreds of ms at worst here? Much lower for something close regionally.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#62

Post by IanCal »

lumost wrote: post #59
goofy_lemur wrote: post #46

Earlier quoted context omitted.

Ok heres the thing you will nevwr be able to truly do this due to logic. Logically five people pooling their resources beats one guy. therefore datacenters will always win because they get higher time utilization. so forget it. I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware. you will get better hardware though, cut…

Laptops/desktops are cheaper per flop than any datacenter hardware by a good order of magnitude. The problem is that expectations rise in datacenters, hardware/power/security/availability guarantees cost real money. Then the operator providing these guarantees expects some margin. You can see this most clearly with "developer desktops", a gcp instance costs about 10x a hetzner instance which costs between 5 and 10x t…

Doesn’t it flip around for small scale? Paying 100x the cost for something, all in, it’s cheaper to rent for small workloads like 10m/day.

At 10x you have to be at hours per day and 5x you’re at 4h.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#63

Post by SiempreViernes »

lumost wrote: post #57
bigmadshoe wrote: post #17

Earlier quoted context omitted.

Can you give an example of such a problem?

"Design me a 3d printable rocket engine for a hobby rocket project. Verify it's design in a full simulation. Iterate until it works reliably in simulation based on a verified printable design on a consumer laser sintering device (or substitute contract manufacture for under 1000 dollars)." This is a hobby version of a project, but you can imagine commercial versions of the same prompt for new databases, genomics stud…

From the prompt it seems evident the envisioned user doesn't have an interest in designing the motor themselves, so why not simply buy a stock motor?

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#64

Post by SwellJoe »

mft_ wrote: post #32

Genuine question: is this solving a real problem? IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on. Oka…

I think the value of it is currently more academic than useful in the real world. Everything at the frontier is still only marginally Good Enough (in image generation, most of it is shit even from the best models), so things far behind the frontier in terms of capability (as a tiny 1-bit model necessarily must be) are unusable.

But, getting remarkably higher density of capability per unit of compute is a big thing. It means the frontier can get better and cheaper to operate and less resource hungry, and it means what can be accomplished at the edge, on personal laptops or phones, becomes a broader spectrum of tasks.

And, for privacy, there are a lot of things that should run on-device and not everyone has big dedicated GPUs.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#66

Post by goofy_lemur »

lumost wrote: post #59
goofy_lemur wrote: post #46

Earlier quoted context omitted.

Ok heres the thing you will nevwr be able to truly do this due to logic. Logically five people pooling their resources beats one guy. therefore datacenters will always win because they get higher time utilization. so forget it. I always wonder the same but i let logic tell me its a fantasy, on average you cant outspend a whole group of people making better use of the hardware. you will get better hardware though, cut…

Laptops/desktops are cheaper per flop than any datacenter hardware by a good order of magnitude. The problem is that expectations rise in datacenters, hardware/power/security/availability guarantees cost real money. Then the operator providing these guarantees expects some margin. You can see this most clearly with "developer desktops", a gcp instance costs about 10x a hetzner instance which costs between 5 and 10x t…

Actually they wouldnt spend the money if it were cheaper.

HBM has way higher bandwidth and its not all about flops.

Also the FP4 flops (inference) are so mind bogglingly high on these things.

Lastly what you fail to consider is the chip to chip bandwidth which is critical.

the people running these know that networking is just as critical.

all reduce etc.

they wouldnt pay if they could get something better value.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#67

Post by joblessjunkie »

mft_ wrote: post #32

Genuine question: is this solving a real problem? IME, the bottleneck when using diffusion models isn't storage space or memory, it's generation time. Lots of models will run on 8-12 GB 1080-generation GPUs onwards, or on Macs with similar memory, which are probably the bottom end from a GPU power perspective anyway. I also note that these models are marginally slower than the small FLUX.2 model they're based on. Oka…

We are in an era of extreme demand for GPU and limited supply. Every inference we push to the edge frees cloud resources for other tasks. Every efficiency gain increases what we can achieve with existing resources. If images can be rendered with half as much compute, we need half as many GPUs.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#68

Post by wmf »

jjcm wrote: post #43
neals wrote: post #23

Earlier quoted context omitted.

I'm curious how hardware and power cost would stack up to subscription cost

I did an estimate of that if you're interested: https://x.com/pwnies/status/2028831699736637912 The TL;DR though is that a 10-15b param model baked into an ASIC with the latest fab tech would take around 62W of power draw when active. At ~10k+ t/s though it likely would only be active for short bursts of time. It'd fit perfectly fine within the thermal envelope of a laptop. The approach makes a lot of sense. Once you…

You're not counting the capex which could be the same cost as 5-10 years of Claude.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#69

Post by all2 »

mirekrusin wrote: post #56
barnas2 wrote: post #34

Earlier quoted context omitted.

A company called Taalas is working on something like that. Not Opus4.6 quality, but I'm sure they're targeting larger models. Currently they're using a LLama 8B model. It runs at ~17k tokens per second, and you can test it at https://chatjimmy.ai/ .

It starts to be interesting when latency is better than average website.

The immediate load (less than 200ms on my machine through a slow connection) is quite pleasant, tbh.

Re: 1-Bit Bonsai Image 4B Image Generation for Local Devices

#70

Post by tomcam »

lumost wrote: post #57
bigmadshoe wrote: post #17

Earlier quoted context omitted.

Can you give an example of such a problem?

"Design me a 3d printable rocket engine for a hobby rocket project. Verify it's design in a full simulation. Iterate until it works reliably in simulation based on a verified printable design on a consumer laser sintering device (or substitute contract manufacture for under 1000 dollars)." This is a hobby version of a project, but you can imagine commercial versions of the same prompt for new databases, genomics stud…

Stop it, you tease. I'm getting a little tingly
Post reply on HN