Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BitNet: Inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

111–120 of 179 posts

Re: BitNet: Inference framework for 1-bit LLMs

#111

Post by janalsncm »

They have a demo video in the readme. I think they are trying to convey that BitNet is fast, which it is. But it is worth taking a moment to pause and actually see what the thing is doing so quickly.

It seems to keep repeating that the water cycle is the main source of energy for all living things on the planet and then citing Jenkins 2010. There are also a ton of sentence beginning with “It also…”

I don’t even think it’s correct. The sun is the main source of energy for most living things but there’s also life near hydrothermal vents etc.

I don’t know who Jenkins is, but this model appears to be very fond of them and the particular fact about water.

I suppose fast and inaccurate is better than slow and inaccurate.

Re: BitNet: Inference framework for 1-bit LLMs

#112

Post by bottlepalm »

cubefox wrote: post #17
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

LLM account

It's scary, without the em dashes, and the rapid fire commenting of the account - who would ever realize this is a bot? Two easy to fix things, and after that it'd be very difficult to tell that this is a bot.

It's not a question of if there are other bots out there, but only what % of comments on HN right now and elsewhere are bot generated. That number is only going to increase if nothing is done.

Re: BitNet: Inference framework for 1-bit LLMs

#113

Post by ramses0 »

intrasight wrote: post #24
giancarlostoro wrote: post #14

One of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think.…

It's not so much a "minimally viable LLM" but rather an LLM that knows natural language well but knows nothing else. Like me - as an engineer who knows how to troubleshoot in general but doesn't know about a specific device like my furnace (recent example). And I don't think that LLM could just Google or check Wikipedia. But I do agree that this architecture makes a lot of sense. I assume it will become the norm to u…

I asked this question a while back (the "only train w/ wikipedia LLM") and got pointed to the general-purpose "compression benchmarks" page: `https://www.mattmahoney.net/dc/text.html`

While I understand some of the fundamental thoughts behind that comparison, it's slightly wonky... I'm not asking "compress wikipedia really well", but instead "can a 'model' reason its way through wikipedia" (and what does that reasoning look like?).

Theoretically with wikipedia-multi-lang you should be able to reasonably nail machine-translation, but if everyone is starting with "only wikipedia" then how well can they keep up with the wild-web-trained models on similar bar chart per task performance?

If your particular training technique (using only wikipedia) can go from 60% of SOTA to 80% of SOTA on "Explain why 6-degrees of Kevin Bacon is relevant for tensor operations" (which is interesting to plug into Google's AI => Dive Deeper...), then that's a clue that it's not just throwing piles of data at the problem, but instead getting closer to extracting the deeper meaning (and/or reasoning!) that the data enables.

Re: BitNet: Inference framework for 1-bit LLMs

#114

Post by WithinReason »

ismailmaj wrote: post #102
WithinReason wrote: post #94

Earlier quoted context omitted.

yes but this is not 1 bit matmul, it's 1.58 bits with expensive unpacking

The title and the repo uses 1-bit when it means 1.58 bits tertiary values, it doesn't change any of my arguments (still xors and pop_counts).

How do you do ternary matmul with popcnt on 1.58 bit packed data?

Re: BitNet: Inference framework for 1-bit LLMs

#115

Post by Havoc »

lemonish97 wrote: post #99

I wonder when we begin to see the dividends of all the NPU PCs come into play. AMD have been doing some good work with their NPU/iGPU hybrid inference kernels. If these larger models could be scaled down to run on NPUs, you'd see much better power advantages, compared to running them on the CPU.

You can already run some models on the NPUs in the Rockchip RK3588 SBCs which are pretty abundant.

A claude 4.6 they are most certainly not, but if you get through the janky AF software ecosystem they can run small LLMs reasonably well with basically zero CPU/GPU usage

Re: BitNet: Inference framework for 1-bit LLMs

#116

Post by ismailmaj »

WithinReason wrote: post #114
ismailmaj wrote: post #102

Earlier quoted context omitted.

The title and the repo uses 1-bit when it means 1.58 bits tertiary values, it doesn't change any of my arguments (still xors and pop_counts).

How do you do ternary matmul with popcnt on 1.58 bit packed data?

Assuming 2 bit per values (first bit is sign and second bit is value).

actv = A[_:1] & B[_:1]

sign = A[_:0] ^ B[_:0]

dot = pop_count(actv & !sign) - pop_count(actv & sign)

It can probably be made more efficient by taking a column-first format.

Since we are in CPU land, we mostly deal with dot products that match the cache size, I don't assume we have a tiled matmul instruction which is unlikely to support this weird 1-bit format.

Re: BitNet: Inference framework for 1-bit LLMs

#117

Post by embeddnet »

embedding-shape wrote: post #16
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

> Framework is ready. Now we need someone to actually train the model. If Microslop aren't gonna train the model themselves to prove their own thesis, why would others? They've had 2 years (I think?) to prove BitNet in at least some way, are you really saying they haven't tried so far? Personally that makes it slightly worrisome to just take what they say at face value, why wouldn't they train and publish a model the…

Rest assured, all the big players (openai, google, deepseek etc) have run countless experiments with 4,3,2,1.58,1 bits, and various sparse factors and shapes. This barrel has been scraped to the bottom

Re: BitNet: Inference framework for 1-bit LLMs

#118

Post by mattalex »

zozbot234 wrote: post #63
embedding-shape wrote: post #45

Earlier quoted context omitted.

> You could have said same about Transformers, Google released it, but didn't move forward, I don't think you can, Google looked at the research results, and continued researching Transformers and related technologies, because they saw the value for it particularly in translations. It's part of the original paper, what direction to take, give it a read, it's relatively approachable for being a machine learning paper…

What OpenAI did was train increasingly large transformer model instances. which was sensible because transformers allowed for a scaling up of training compared to earlier models. The resulting instances (GPT) showed good understanding of natural language syntax and generation of mostly sensible text (which was unprecedented at the time) so they made ChatGPT by adding new stages of supervised fine tuning and RLHF to t…

There were plenty of models the size of gpt3 in industry.

The core insight necessary for chatgpt was not scaling (that was already widely accepted): the insight was that instead of finetuning for each individual task, you can finetune once for the meta-task of instruction following, which brings a problem specification directly into the data stream.

Re: BitNet: Inference framework for 1-bit LLMs

#120

Post by pxndx »

vova_hn2 wrote: post #34
orbital-decay wrote: post #23

Earlier quoted context omitted.

Funny enough I now involuntarily take RTFA as a slight slop signal, because all these accounts dutifully read the article before commenting, unlike most HNers who often respond to headlines.

First they claimed that if you use em dashes you are not human And I did not speak out Because I was not using em dashes Then they claimed that if you're crammar is to gud you r not hmuan And I did not spek aut Because mi gramar sukcs Then they claimed that if you actually read the article that you are trying to discuss you are not human...

There's obviously an xkcd about this: https://xkcd.com/810/
Post reply on HN