Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BitNet: Inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

51–60 of 179 posts

Re: BitNet: Inference framework for 1-bit LLMs

#51

Post by K0balt »

vova_hn2 wrote: post #34
orbital-decay wrote: post #23

Earlier quoted context omitted.

Funny enough I now involuntarily take RTFA as a slight slop signal, because all these accounts dutifully read the article before commenting, unlike most HNers who often respond to headlines.

First they claimed that if you use em dashes you are not human And I did not speak out Because I was not using em dashes Then they claimed that if you're crammar is to gud you r not hmuan And I did not spek aut Because mi gramar sukcs Then they claimed that if you actually read the article that you are trying to discuss you are not human...

I’ve been rounded up for things I wrote two decades ago because of my em dashes lol. The pitchfork mentality gives me little hope for how things are going to go once we have hive mind AGI robots pervasive in society.

Re: BitNet: Inference framework for 1-bit LLMs

#52

Post by nkohari »

cubefox wrote: post #17
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

LLM account

I would love to understand the thought process behind this. I'm sure it's a fun experiment, to see if it's possible and so on... but what tangible benefit could there be to burning tokens to spam comments on every post?

Re: BitNet: Inference framework for 1-bit LLMs

#54

Post by WhitneyLand »

If they had a big result like, native 1.58-bit quality clearly matches top peers, they would be saying that prominently in the repo.

The engineering/optimization work is nice, but this is not what people have been waiting for, as much as, can’t the Bitnet idea that seemed promise really deliver in a competitive way.

Re: BitNet: Inference framework for 1-bit LLMs

#55

Post by wongarsu »

embedding-shape wrote: post #45
throwaw12 wrote: post #39

Earlier quoted context omitted.

Because this is Microsoft, experimenting and failing is not encouraged, taking less risky bets and getting promoted is. Also no customer asked them to have 1-bit model, hence PM didn't prioritize it. But it doesn't mean, idea is worthless. You could have said same about Transformers, Google released it, but didn't move forward, turns out it was a great idea.

> You could have said same about Transformers, Google released it, but didn't move forward, I don't think you can, Google looked at the research results, and continued researching Transformers and related technologies, because they saw the value for it particularly in translations. It's part of the original paper, what direction to take, give it a read, it's relatively approachable for being a machine learning paper…

On the one hand, not publishing any new models for an architecture in almost a year seems like forever given how things are moving right now. On the other hand I don't think that's very conclusive on whether they've given up on it or have other higher priority research directions to go into first either

Re: BitNet: Inference framework for 1-bit LLMs

#56

Post by silon42 »

cubefox wrote: post #13
nickcw wrote: post #7

> bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU and GPU (NPU support will coming next). One bit or one trit? I am confused!

Yeah, "1.58 bit" is 1 trit with three states, since log2(3)≈1.58. So it's not a inference framework for 1-bit models (two states per parameter) but for 1.58 bit models (three states per parameter). Annoying that they try to mix up the two.

I always hope for "just a bunch of if statements" ... this is not it.

Re: BitNet: Inference framework for 1-bit LLMs

#57

Post by giancarlostoro »

bee_rider wrote: post #49
giancarlostoro wrote: post #14

One of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think.…

Isn’t that sort of what a RAG is? You’d need an LLM “smart” enough to turn natural-user prompts into searches, then some kind of search, then an LLM “smart” though to summarize the results.

Yeah, I think RAG is the idea that will lead us there, though its a little complicated, because for some subjects, say Computer Science, you need a little more than just "This is Hello World in Go" you might need to understand not just Go syntax on the fly, but more CS nuances that are not covered in one single simple document. The idea being having a model that runs fully locally on a phone or laptop with minimal resources. On the other hand, I can also see smaller models talking to larger models that are cheaper to run in the cloud. I am wondering if this is the approach Apple might take with Siri, specifically in order to retain user privacy as much as possible.

Re: BitNet: Inference framework for 1-bit LLMs

#58

Post by giancarlostoro »

embedding-shape wrote: post #18
giancarlostoro wrote: post #14

One of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think.…

Your worry about Wikipedia is that there is "much room for human error and bias", yet earlier you seem to imply that a LLM that has access to the www somehow would have less human error and bias? Personally, I'd see it the other way around.

When GPT 3.5 became a thing, it had crawled a very nuanced set of websites, this is what I mean. You basically curate where it sources data from.

Re: BitNet: Inference framework for 1-bit LLMs

#59

Post by giancarlostoro »

intrasight wrote: post #24
giancarlostoro wrote: post #14

One of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think.…

It's not so much a "minimally viable LLM" but rather an LLM that knows natural language well but knows nothing else. Like me - as an engineer who knows how to troubleshoot in general but doesn't know about a specific device like my furnace (recent example). And I don't think that LLM could just Google or check Wikipedia. But I do agree that this architecture makes a lot of sense. I assume it will become the norm to u…

Correct! I know RAG is a thing, but I wish we could have "DLCs" for LLMs like image generation has LoRa's which are cheaper to train for than retraining the entire model, and provide more output like what you want. I would love to pop in the CS "LoRa or DLC" and ask it about functional programming in Elixir, or whatever.

Maybe not crawl the web, but hit a service with pre-hosted, precurated content it can digest (and cache) that doesn't necessarily change often enough. You aren't using it for the latest news necessarily, but programming is mostly static knowledge a a good example.

Re: BitNet: Inference framework for 1-bit LLMs

#60

Post by vova_hn2 »

K0balt wrote: post #51
vova_hn2 wrote: post #34

Earlier quoted context omitted.

First they claimed that if you use em dashes you are not human And I did not speak out Because I was not using em dashes Then they claimed that if you're crammar is to gud you r not hmuan And I did not spek aut Because mi gramar sukcs Then they claimed that if you actually read the article that you are trying to discuss you are not human...

I’ve been rounded up for things I wrote two decades ago because of my em dashes lol. The pitchfork mentality gives me little hope for how things are going to go once we have hive mind AGI robots pervasive in society.

If I was operating a bot farm, at this point I would probably add some bots that go around and accuse legit human users (or just random users) of being bots.

Created confusion and frustration will make it much harder to separate signal from the noise for most people.

Post reply on HN