Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Steerling-8B, a language model that can explain any token it generates

guidelabs.ai

Post reply on HN
Print view View original

61–70 of 104 posts

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#61

Post by rickydroll »

adebayoj wrote: post #36
great_psy wrote: post #16

Earlier quoted context omitted.

The example on the website shows one to many as well: Wikipedia, axive article, etc along with a ratio how much it influences the chunk of the answer.

Exactly! We will have a future post that shows this more granularly over the coming weeks. Here is a post we wrote on how this works at smaller scale: https://www.guidelabs.ai/post/prism/

Oh, that looks like a wonderful article. I just skimmed it, and I hope to get back to it later today. One thing I would love to see is how much of the training set is substantially similar to each other, especially in the code training set.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#65

Post by xmcqdpt2 »

adebayoj wrote: post #59
pu_pe wrote: post #57

Looks neat and original, congrats! I don't quite grasp how to interpret the training data attribution process. For example, it seems to say that for a given sentence like "They argued that humans tend to weigh losses more heavily than gains, leading to risk aversion", 24% is attributed to Wikipedia and 23% to Arxiv. Does that mean that the concepts used in this sentence are also found in those datasets, and that's wh…

Great questions. We weren't quite explicit about the training data attribution process. We'll discuss this in more detail in future work. We can track down which parts of the training data were interpolated to create that sentence. For those training data sentences, we then compare the concepts between generated and training. We can attribute to exact sentences and chunks in the training data. For the first release,…

That would be great because "I got it from Wikipedia and Arxiv" isn't exactly useful.

From reading your second link (and please tell me if I got it wrong) it sounds like it isn't actually tracking to training data but to prototypes which are then linked a posteriori to likely sections of the training data. The attribution isn't exact, right? It's more like "these are the likely texts that contributed to one of those prototypes that produced the final answer." Specifically the bit in PRISM titled "Nearest neighbour Search" sounds like you could have a prototype that takes from 1000 sources but 3 of them more than the others, so the model identify those 3, but the other ones might matter just as much in aggregate?

It says that the decomposition is linear. Can you remove a given prototype and infer again without it? That would be really cool.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#66

Post by KingOfCoders »

crimsonnoodle58 wrote: post #53

So maybe one day we'll see coding agents like Claude Code create and update an ATTRIBUTION.md, citing all the open source projects and their licenses used to generate code in your project?

Not as long as all developers add an ATTRIBUTION.md citing all open source projects they read the source for, all companies they worked for and trained them and all Stack Overflow answers they have used for write the code.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#67

Post by adebayoj »

whinvik wrote: post #52

Looks very interesting. Can you comment on why you think this model can give comparable performance with less training data?

We train the model with `explanations`. Most training asks the model to predict the next token or group of tokens. Our training says, predict the next group of tokens (causal diffusion), but also these tokens should be about {sports/art/coding/etc}. So in addition to token supervision, the model gets concept level supervision. The model is forced to more quickly learn these high level concepts.

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#68

Post by gchamonlive »

Grimblewald wrote: post #58
gchamonlive wrote: post #49

Earlier quoted context omitted.

I think this was answered before, with the constraints of the architecture of the model. You can't expect something fundamentally different from an LLM, because that's how they work. It's different from other models because they were not designed for this. Maybe you were expecting more, but that's not OP's fault or demerit.

What you're saying fits my understanding/expectations. However the post and the user I am replying to seem to imply different. This makes me wonder, is my understanding incomplete or is this post marketing hype dressed up as insight? So I am asking for transparency.

We are probably at the point where hype and insight aren't that much distinguishable other than what would bear fruit in the future, but I agree with you

Re: Show HN: Steerling-8B, a language model that can explain any token it generates

#69

Post by lelanthran »

KingOfCoders wrote: post #66
crimsonnoodle58 wrote: post #53

So maybe one day we'll see coding agents like Claude Code create and update an ATTRIBUTION.md, citing all the open source projects and their licenses used to generate code in your project?

Not as long as all developers add an ATTRIBUTION.md citing all open source projects they read the source for, all companies they worked for and trained them and all Stack Overflow answers they have used for write the code.

> Not as long as all developers add an ATTRIBUTION.md citing all open source projects they read the source for, all companies they worked for and trained them and all Stack Overflow answers they have used for write the code.

Oh? You are under the impression that software gets the same rights and privileges of humans?

Or maybe you are under the impression that you are so special that you face no danger from having no income because the models already ingested all your work and can launder it effectively?

Post reply on HN