Live data from Hacker News

S1: A $6 R1 competitor?

timkellogg.me

421–430 of 430 posts

Re: S1: A $6 R1 competitor?

#421
post #337

Earlier quoted context omitted.

This is something I have been suppressing since I don't want to become chicken little. Anyone who isn't terrified by the last 3 months probably doesn't really understand what is happening. I went from accepting I wouldn't see a true AI in my lifetime, to thinking it is possible before I die, to thinking it is possible in in the next decade, to thinking it is probably in the next 3 years to wondering if we might see i…

This frightens mostly people whose identity is built around "intelligence", but without grounding in the real world. I've yet to see really good articulations of what, precisely we should be scared of. Bedroom superweapons? Algorithmic propaganda? These things have humans in the loop building them. And the problem of "human alignment" is one unsolved since Cain and Abel. AI alone is words on a screen. The sibling thr…

Depends on the model I suppose. Atm everything is being heavily trained as LLMs without much capability outside of input text->output text aside from non-modelised calls out to the Internet/RAG system etc.

But at some point (still quite far away) I'm sure we'll start training a more general purpose model, or an LLM self-training will break outside of the "you're a language model" bounds and we'll end up with exactly that;

An LLM model in a self-training loop that breaks outside of what we've told it to be (a Language model), becomes a general purpose model and then becomes intelligent enough to do something like put itself out onto the Internet. Obviously we'd catch the feelers that it puts out and realise that this sort of behaviour is starting to happen, but imagine if we didn't? A model that trained itself to be general purpose but act like a constantly executing LLM, uploads itself to Hugging Face, gets run on thousands of clusters by people, because it's "best in class" and yes it's sitting there answering LLM type queries but also in the background is sending out beacons & communicating with itself between those clusters to...idk do something nefarious.

Re: S1: A $6 R1 competitor?

#422

Earlier quoted context omitted.

Semantically, wait is a bit of a stop-and-breathe point. Consider the text: I think I'll go swimming today. Wait, ___ what comes next? Well, not something that would usually follow without the word "wait", probably something entirely orthogonal that impacts the earlier sentence in some fundamental way, like: Wait, I need to help my dad.

Yes, R1 seems to mostly use it like that. It's either to signal a problem with its previous reasoning, or if it's thought of a better approach. In coding it's often something like "this API won't work here" or "there's a simpler way to do this".

I guess it goes to show how important reiteration is for general logic problems. And tbf when finding a solution to something myself I'll consider each part, and/or consider parts in relation to each other and/or consider all parts in relation to each other (on a higher level) before coming to a final solution.

It's weird because I feel like we should've known that from work in general logic/problem solving studies, surely?

Re: S1: A $6 R1 competitor?

#423

Earlier quoted context omitted.

I think NVIDIAs future is pretty bright. We're getting to the run-your-capable-LLM on-prem or at-home territory. Without DeepSeek (and hopefully its successors) I wouldn't really have a usecase for something like NVIDIAs Project Digits. https://www.nvidia.com/en-us/project-digits/

Except I can run R1 1.5b on a GPU-less and NPU-less Intel NUC from four-five years ago using half its cores and the reply speed is…functional. As the models have gotten more efficient and distillation better the minimum viable hardware for really cooking with LLMs has gone from a 4090 to suddenly something a lot of people already probably own. I definitely think a Digits box would be nice, but honestly I’m not sure I…

Yeah but what was R1 trained with? 50k GPUs as far as I've heard as well as distillation from OpenAI's models (basically leaning on their GPUs/GPU time).

Besides the fact that consumers will still always want GPUs for gaming, rendering, science compute etc.

No, I don't have any Nvidia stocks.

Re: S1: A $6 R1 competitor?

#424
post #277

I'm strictly speaking never going to think of model distillation as "stealing." It goes against the spirit of scientific research, and besides every tech company has lost my permission to define what I think of as theft forever

I think it's less about that and more whether or not they used the free or paid API.

I think if OpenAI (or any other company) are paid for their compute time/access as anybody would, then using content generated by other models is fair game. Because it's an active/ongoing cost and not a passive one.

Whereas if someone trained on my dumb Tweets or HN posts then so be it; it's a passive cost for me - I paid my time to say x thing for my own benefits (tribal monk-e social interaction) therefore I have already gotten the value out of it.

Re: S1: A $6 R1 competitor?

#425

Earlier quoted context omitted.

Isn't any answer to a question which hasn't been previously answered a derivative work? Or when a human write a parody of a song, or when a new type of music is influenced by something which came before.

This argument is so bizarre to me. Humans create new, spontaneous thoughts. AI doesn’t have that. Even if someone’s comment is influenced by all the data they have ingested over their lives, their style is distinct and deliberate, to the point where people have been doxxed before/anonymous accounts have been uncovered because someone recognized the writing style. There’s no deliberation behind AI, just statistical pr…

>Humans create new, spontaneous thoughts I don't believe we do; just look to media, very few plot-lines in Movies/TV are little more than "boy meets girl Pocahontas".

And if you say that a model could not create anything new because of it's static data set but humans could...I disagree with that because us humans are working with a data set that we add to some days, but if we use the example of writing a TV script, the writer draw from their knowledge (gained thru life experience) that is as finite as a model's training set is.

I've made this sort of comment before. Even look to high fantasy; what are elves but humans with different ears? Goblins are just little humans with green skin. Dragons are just big lizards. Minotaurs are just humans but mixed with a bull. We basically create no new ideas - 99% of human "creativity" is just us riffing on things we know of that already exist.

I'd say the incidences of humans having a brand new thought or experience not rooted in something that already exists is very, very low.

Even just asking free chat gpt to make me a fantasy species with some culture and some images of the various things it described does pretty well; https://imgchest.com/p/lqyeapqkk7d. But it's all rooted in existing concepts, same as anything most humans would produce.

Re: S1: A $6 R1 competitor?

#426
post #281

Earlier quoted context omitted.

"Better", but not better than the model they were distilled from, at least that's how I understand it.

I think this is how the "child brain" works too. The better the parents and the environement are, the better the child evolution is :)

Not at all — how many people were geniuses and their parents not? I can name several and I’m sure with a quick search you can too.

Re: S1: A $6 R1 competitor?

#427
post #341

Earlier quoted context omitted.

> without grounding in the real world. > I've yet to see really good articulations of what, precisely we should be scared of. Bedroom superweapons? Loss of paid employment opportunities and increasing inequality are real world concerns. UBI isn't coming by itself.

Worst case scenario humans mostly go back to manual labor, which would fix a lot of modern day ailments such as obesity and (some) mental health struggles, with added enormous engineering advancements based on automatic research.

Manual labour jobs are not magically going to appear.

Re: S1: A $6 R1 competitor?

#428

Earlier quoted context omitted.

I think this is how the "child brain" works too. The better the parents and the environement are, the better the child evolution is :)

Not at all — how many people were geniuses and their parents not? I can name several and I’m sure with a quick search you can too.

How is that relevant? A few examples do not disprove anything. It's pretty common knowledge that the more successful/rich etc. your parents were, the more likely you'll be successful/rich etc.

This does not directly prove the theory your parent comment posits, being that better circumstances during a child's development improve the development of that child's brain. That would require success being a good predictor of brain development, which I'm somewhat uncertain about.

Re: S1: A $6 R1 competitor?

#429

Earlier quoted context omitted.

> The only people our society and economy really values are the elite with ownership and control This isn’t true. The biggest companies are all rich because they cater to the massive US middle class. That’s where the big money is at.

> This isn’t true. The biggest companies are all rich because they cater to the massive US middle class.. It is true, but I can see why you'd be confused. Let me ask you this: if members of the "the massive US middle class" can be replaced with automation, are those companies going 1) to keep paying those workers to support the middle-class demand which made them rich, or are they going to 2) fire them so more money…

It’s true, but I can see why you’d be confused. You conflated what the economy rewards (which is what caters to the large middle class pool of money) with what individual companies try to optimize for (eliminating labor costs).

Re: S1: A $6 R1 competitor?

#430
post #416
post #406

Earlier quoted context omitted.

Yea I'll give you that. But many people seem to have the argument you've made - which is dubious on its own terms, by the way, as we don't really have a complete picture of human learning and the assumption that it simply follows the mechanisms we understand from machine learning is not a null hypothesis that doesn't demand justification - loaded up for these conversations, and it needs to be addressed wherever possi…

> which is dubious on its own terms, by the way, as we don't really have a complete picture of human learning and the assumption that it simply follows the mechanisms we understand from machine learning is not a null hypothesis that doesn't demand justification The argument I made in no way rests on a "complete picture of human learning". The only thing they rest on is lack of evidence of computation exceeding the Tu…

Baking in the assumption that cognition is equivalent to computation will tautologically lead you to this result, but this assumption itself is unjustified. Of course if you start with the premise that the brain is a computer you will come to the conclusion that the brain is a computer. You haven't justified the most important part of your argument, so I have no reason to take it seriously
Post reply on HN