Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BitNet: Inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

161–170 of 179 posts

Re: BitNet: Inference framework for 1-bit LLMs

#161

Post by Springtime »

Karrot_Kream wrote: post #160
Springtime wrote: post #157

Earlier quoted context omitted.

Your account posted dense, opinionated and structured paragraphs mere minutes apart—sometimes the same minute—for multiple story submissions. Even with my own sometimes lengthy replies this would be infeasible to both instantly have structured opinions and type them out in time. Two of your posts were posted the same minute, with a combined word count of 146 . It feels like it'd take someone superhuman to come across…

Lol. I know at least a few high karma account who post at the same frequency but they post about anti-AI and anti-tech topics instead on the big social media tech where anti-tech opinions dominate. I guess this exempts them from scrutiny? I love these witch hunts.

They post 146 words per minute across multiple different submissions with similarly structured posts? I know there are users who post frequently in other communities I'm familiar with but not in that kind of timeframe with such paragraph density or structure.

Re: BitNet: Inference framework for 1-bit LLMs

#162

Post by Karrot_Kream »

Springtime wrote: post #161
Karrot_Kream wrote: post #160

Earlier quoted context omitted.

Lol. I know at least a few high karma account who post at the same frequency but they post about anti-AI and anti-tech topics instead on the big social media tech where anti-tech opinions dominate. I guess this exempts them from scrutiny? I love these witch hunts.

They post 146 words per minute across multiple different submissions with similarly structured posts? I know there are users who post frequently in other communities I'm familiar with but not in that kind of timeframe with such paragraph density or structure.

Yes. Any of the really heated political threads on this site are full of posters like this. I don't want to dox anyone, but since at least 2019-ish I've found posters that spend hours a day posting huge amounts of content in large bursts on this site.

COVID was ridiculous as I presume a lot of anxious people were stuck at home able to do nothing but post.

Re: BitNet: Inference framework for 1-bit LLMs

#163

Post by grepfru_it »

joquarky wrote: post #146
observationist wrote: post #74

Earlier quoted context omitted.

So is it finally time for a Beowulf cluster to do something amazing?

Maybe! And it looks like Windows 11 will finally make this the year of the Linux desktop...

I'll believe that when CowboyNeaLLM is released

Re: BitNet: Inference framework for 1-bit LLMs

#164

Post by Springtime »

Karrot_Kream wrote: post #162
Springtime wrote: post #161

Earlier quoted context omitted.

They post 146 words per minute across multiple different submissions with similarly structured posts? I know there are users who post frequently in other communities I'm familiar with but not in that kind of timeframe with such paragraph density or structure.

Yes. Any of the really heated political threads on this site are full of posters like this. I don't want to dox anyone, but since at least 2019-ish I've found posters that spend hours a day posting huge amounts of content in large bursts on this site. COVID was ridiculous as I presume a lot of anxious people were stuck at home able to do nothing but post.

Does this account for them reading the article though? There are pre-existing opinions that could be easy to rapidly post based on topical discussions but here it also has to be considered the time to parse technical submissions.

They read this article, called out a specific discrepancy then commented on a paper on Arxiv in a 70-odd word post then the same minute another 70-odd word post on a different technical post. Maybe like you suggest they're just wired differently.

Re: BitNet: Inference framework for 1-bit LLMs

#165

Post by Karrot_Kream »

Springtime wrote: post #164
Karrot_Kream wrote: post #162

Earlier quoted context omitted.

Yes. Any of the really heated political threads on this site are full of posters like this. I don't want to dox anyone, but since at least 2019-ish I've found posters that spend hours a day posting huge amounts of content in large bursts on this site. COVID was ridiculous as I presume a lot of anxious people were stuck at home able to do nothing but post.

Does this account for them reading the article though? There are pre-existing opinions that could be easy to rapidly post based on topical discussions but here it also has to be considered the time to parse technical submissions. They read this article, called out a specific discrepancy then commented on a paper on Arxiv in a 70-odd word post then the same minute another 70-odd word post on a different technical post…

Yeah, there are plenty of very online people who are great at just posting things and doing the minimum amount of diligence to contribute their thought. Like I said, just hang around in any of the high energy threads on this site and you'll see a lot of it.

Re: BitNet: Inference framework for 1-bit LLMs

#166

Post by Aerroon »

embeddnet wrote: post #117
embedding-shape wrote: post #16

Earlier quoted context omitted.

> Framework is ready. Now we need someone to actually train the model. If Microslop aren't gonna train the model themselves to prove their own thesis, why would others? They've had 2 years (I think?) to prove BitNet in at least some way, are you really saying they haven't tried so far? Personally that makes it slightly worrisome to just take what they say at face value, why wouldn't they train and publish a model the…

Rest assured, all the big players (openai, google, deepseek etc) have run countless experiments with 4,3,2,1.58,1 bits, and various sparse factors and shapes. This barrel has been scraped to the bottom

I have doubts about this. Perhaps the closed models have, but I wouldn't be so sure for the open ones.

GLM 5, for example, is running 16-bit weights natively. This makes their 755B model 1.5TB in size. It also makes their 40B active parameters ~80GB each.

Compare this to Kimi K2.5. 1T model, but it's 4-bit weights (int4), which makes the model ~560 GB. Their 32B active parameters are ~16 GB.

Sure, GLM 5 is the stronger model, but is that price worth paying with 2-3x longer generation times? What about 2-3x more memory required?

I think this barrel's bottom really hasn't been scraped.

Re: BitNet: Inference framework for 1-bit LLMs

#167

Post by Aerroon »

DrBazza wrote: post #100
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

> memory bandwidth is always the bottleneck I'm hoping that today's complaints are tomorrow's innovations. Back when 1Mb hard drive was $100,000, or when Gates said 640kb is enough. Perhaps some 'in the (chip) industry' can comment on what RAM manufacturers are doing at the moment - better, faster, larger? Or is there not much headroom left and it's down to MOBO manufacturers, and volume?

We have faster memory, it's just all used in data center cards you can't buy (and can't afford to buy).

AMD actually used HBM2 memory in their Radeon VII card back in 2019 (!!) for $700. It had 16 GB of HBM2 memory with 1 TB/s throughput.

The RTX 5080 in conversion l comparison also has 16 GB of VRAM, but was released in 2025 and has 960 GB/s throughput. The RTX 5090 does have an edge at 1.8 TB/s bandwidth and 32 GB of VRAM but it also costs several times more. Imagine if GPUs had gone down the path of the Radeon VII.

That being said, the data center cards from both are monstrous.

The Nvidia B200 has 180 GB of VRAM (2x90GB) offering 8.2 TB/s bandwidth (4.1 TB/s x2) released in 2024. It just costs as much as a car, but that doesn't matter, because afaik you can't even buy them individually. I think you need to buy a server system from Nvidia or Dell that will come with like 8 of these and cost you like $600k.

AMD has the Mi series. Eg AMD MI325x. 288 GB of VRAM doing 10 TB/s bandwidth and released in 2024. Same story as Nvidia: buy from an OEM that will sell you a full system with 8x of these (and if you do get your hands on one of these you need a special motherboard for them since they don't do PCIe). Supposedly a lot cheaper than Nvidia, but still probably $250k.

These are not even the latest and greatest for either company. The B300 and Mi355x are even better.

It's a shame about the socket for the Mi series GPUs (and the Nvidia ones too). The Mi200 and Mi250x would be pretty cool to get second-hand. They are 64 GB and 128GB VRAM GPUs, but since they use OAP socket you need the special motherboard to run them. They're from 2021, so in a few years time they will likely be replaced, but as a regular joe you likely can't use them.

The systems exist, you just can't have them, but you can rent them in the cloud at about $2-4 per hour per GPU.

Re: BitNet: Inference framework for 1-bit LLMs

#168

Post by rablackburn »

giancarlostoro wrote: post #14

One of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think.…

I feel like I should say "spoiler alert" but:

> I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers?

It depends what that word "reasonable" means for your specific use-case ;)

Re: BitNet: Inference framework for 1-bit LLMs

#169

Post by 5o1ecist »

janalsncm wrote: post #111

They have a demo video in the readme. I think they are trying to convey that BitNet is fast, which it is. But it is worth taking a moment to pause and actually see what the thing is doing so quickly. It seems to keep repeating that the water cycle is the main source of energy for all living things on the planet and then citing Jenkins 2010. There are also a ton of sentence beginning with “It also…” I don’t even think…

[dead]

Re: BitNet: Inference framework for 1-bit LLMs

#170

Post by WithinReason »

ismailmaj wrote: post #116
WithinReason wrote: post #114

Earlier quoted context omitted.

How do you do ternary matmul with popcnt on 1.58 bit packed data?

Assuming 2 bit per values (first bit is sign and second bit is value). actv = A[_:1] & B[_:1] sign = A[_:0] ^ B[_:0] dot = pop_count(actv & !sign) - pop_count(actv & sign) It can probably be made more efficient by taking a column-first format. Since we are in CPU land, we mostly deal with dot products that match the cache size, I don't assume we have a tiled matmul instruction which is unlikely to support this weird…

This is 11 bit ops and a subtract, which I assume is ~11 clocks, while you can just do:

l1 = dot(A[:11000000],B[:11000000]) l2 = dot(A[:00110000],B[:00110000]) l3 = dot(A[:00001100],B[:00001100]) l4 = dot(A[:00000011],B[:00000011])

result = l1 + l2 * 4 + l3 * 16 + l4 * 64

which is 8 bit ops and 4x8 bit dots, which is likely 8 clocks with less serial dependence

Post reply on HN