Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepseek R1-0528

huggingface.co

Post reply on HN
Print view View original

221–230 of 264 posts

Re: Deepseek R1-0528

#221

Post by addaon »

alfiedotwtf wrote: post #192
desdenova wrote: post #180

Earlier quoted context omitted.

What makes models non-deterministic isn't the training algorithm, but the initial weights being random. Training is reproducible only if, besides the pipeline and data, you also start from the same random weights.

So is there no “introduce randomness” at some step afterwards? If not, I would guess these models would be getting stuck in a local maxima

> If not, I would guess these models would be getting stuck in a local maxima

It sounds like you're referring to something like simulated annealing. Using that as an example, the fundamental requirement is to introduce arbitrary, uncorrelated steps -- there's no requirement that the steps be random, and the only potential advantage of using a random source is that it provides independence (lack of correlation) inherently; but in exchange, it makes testing and reproduction much harder. Basically every use of simulated annealing or similar I've run into uses pseudorandom numbers for this reason.

Re: Deepseek R1-0528

#222

Post by yencabulator »

selfhoster11 wrote: post #159
xnickb wrote: post #151

Earlier quoted context omitted.

I'd argue we don't need a 10 star system. The single bit we have now is enough. And the question is also pretty clear: did $company steal other peoples work? The answer is also known. So the reason one would want an open source model (read reproducible model), would be that of ethics

We use pop-cultural references to communicate all the time these days. Those don't necessarily come from only the most commonly known sections of these works, so the AI would necessarily need the full work (or a functional transformation of the work) to be able to hit the theoretical maximum of the ability to decode about and reason using such references. To exclude copyrighted works from the training set is to expec…

"Your honor, if I hadn't robbed that bank I wouldn't have gotten all that money!"

Re: Deepseek R1-0528

#223

Post by subscribed »

numpad0 wrote: post #99
behnamoh wrote: post #87

Earlier quoted context omitted.

it's got more 'source' than whatever OpenAI provides for their models.

less alcoholic beverages are fully alcoholic beverages

0.5% or 0.03% satisfy my "nonalcoholic" criteria.

> Studies have found ethanol levels in commercial apple juice ranging from 0.06 to 0.66 grams per liter, with an average around 0.26 grams per liter[1]

Even apple juice is an alcoholic drink if you push your criteria to absurdity.

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC5421578/

Re: Deepseek R1-0528

#224

Post by macrolime »

timschmidt wrote: post #171
macrolime wrote: post #170

Earlier quoted context omitted.

Not legally. That's the difference.

Sure you can. It's often legally protected activity. You're just limited to distributing your modifications without the original work.

For some games maybe, but software often has a clause forbidding reverse engineering

Re: Deepseek R1-0528

#225

Post by latchkey »

jacob019 wrote: post #63
latchkey wrote: post #38

Earlier quoted context omitted.

Every prompt is valuable.

I'm not so sure. I have agents that do categorization work. Take a title, drill through a browse tree to find the most applicable leaf category. Lots of other classification tasks that are not particularly sensitive and it's hard to imagine them being very good for training. Also transformations of anonymized numerical data, parsing, etc.

"one man's garbage is another man's treasure"

Re: Deepseek R1-0528

#227

Post by qingcharles »

diggan wrote: post #47
lodovic wrote: post #42

Earlier quoted context omitted.

If you use the excess memory for AI only it's cheaper to rent . A single H100 costs less than $2 per hour. (incl power)

Vast.ai has a bunch of 1x H100 SXM available, right now the cheapest at $1.554/hr. Not affiliated, just a (mostly) happy user, although don't trust the bandwidth numbers, lots of variance (not surprising though, it is a user-to-user marketplace).

Every time someone asks me what hardware to buy to run these at home I show them how many thousands of hours at vast.ai you could get for the same cost.

I don't even know how these Vast servers make money because there is no way you can ever pay off your hardware from the pennies you're getting.

Re: Deepseek R1-0528

#229

Post by ratamacue »

pradn wrote: post #114
jazzyjackson wrote: post #75

Earlier quoted context omitted.

No sign of what source material it was trained on though right? So open weight rather than reproducible from source. I remember there's a project "Open R1" that last I checked was working on gathering their own list of training material, looks active but not sure how far along they've gotten: https://github.com/huggingface/open-r1

Isn't it basically not possible for the input data set list to be listed? It's an open secret all these labs are using immense amounts of copyrighted material. There's a few efforts at full open data / open weight / open code models, but none of them have gotten to leading-edge performance.

My brain was largely trained using immense amounts of copyrighted material as well. Some of it I can even regurgitate almost exactly. I could list the names of many of the copyrighted works I have read/watched/listened to. I suppose my brain isn't open source, although I don't think it would currently be illegal to take a snapshot of my brain and publish it if the technology existed and open-source that. Granted, this would only be "reproducible" from source if you define the "source" as "my brain" rather than all of the material I consumed to make that snapshot.

Re: Deepseek R1-0528

#230

Post by jacob019 »

timschmidt wrote: post #173
reedciccio wrote: post #172

Earlier quoted context omitted.

It's very imperfect analogy though these things can't be rebuilt "from scratch" like a program, the training process doesn't seem to be replicable anyway. Nonetheless, full data disclosure is necessary, according to the result of the years-long consultation led by the Open Source Initiative https://opensource.org/ai

> the training process doesn't seem to be replicable anyway The training process is fully deterministic. It's just an algorithm. Feed the same data in and you'll get the same weights out. If you're speaking about the computational cost, it used to be that way for compilers too. Give it 20 years and you'll be able to train one of today's models on your phone.

A lot of quibbling here, wasn't sure where to reply. If you've built any models in PyTorch, then you know. Conceptually it is deterministic, a model trained using deterministic implementations of low level algorithms will produce deterministic results. And when you are optimizing the pipeline, it is common to do just that:

    torch.manual_seed(0)
    random.seed(0)
    np.random.seed(0)
    torch.use_deterministic_algorithms(True)
But in practice that is too slow, we use nondeterministic implementations that run fast and loose with memory management and don't necessarily care about the order in which parallel operations return.
Post reply on HN