Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deepseek R1-0528

huggingface.co

Post reply on HN
Print view View original

181–190 of 264 posts

Re: Deepseek R1-0528

#181

Post by rurban »

codedokode wrote: post #134
karencarits wrote: post #119

What use cases are people using local LLMs for? Have you created any practical tools that actually increase your efficiency? I've been experimenting a bit but find it hard to get inspiration for useful applications

Anyone who does not want to leak their data? I am actually surprised that people are ok with trusting their secrets to a random foreign company.

A random foreign company is far better than a big 5 eyes country, which syphon everything to the NSA, and use it against you.

Whilst the Chinese intelligence agency will have not much power over you.

Re: Deepseek R1-0528

#182

Post by timschmidt »

desdenova wrote: post #180
timschmidt wrote: post #173

Earlier quoted context omitted.

> the training process doesn't seem to be replicable anyway The training process is fully deterministic. It's just an algorithm. Feed the same data in and you'll get the same weights out. If you're speaking about the computational cost, it used to be that way for compilers too. Give it 20 years and you'll be able to train one of today's models on your phone.

What makes models non-deterministic isn't the training algorithm, but the initial weights being random. Training is reproducible only if, besides the pipeline and data, you also start from the same random weights.

That would fall under "Feed the same data in and you'll get the same weights out." Lots of deterministic algorithms use a random seed.

Re: Deepseek R1-0528

#183

Post by willmarch »

timschmidt wrote: post #173
reedciccio wrote: post #172

Earlier quoted context omitted.

It's very imperfect analogy though these things can't be rebuilt "from scratch" like a program, the training process doesn't seem to be replicable anyway. Nonetheless, full data disclosure is necessary, according to the result of the years-long consultation led by the Open Source Initiative https://opensource.org/ai

> the training process doesn't seem to be replicable anyway The training process is fully deterministic. It's just an algorithm. Feed the same data in and you'll get the same weights out. If you're speaking about the computational cost, it used to be that way for compilers too. Give it 20 years and you'll be able to train one of today's models on your phone.

I’m pretty sure the initial weights are randomized meaning no two models will train in the same way twice. The order in which you feed in training data to the model would also add an element of randomness. Model training is closer to growing a plant than running a compiler.

Re: Deepseek R1-0528

#184

Post by timschmidt »

willmarch wrote: post #183
timschmidt wrote: post #173

Earlier quoted context omitted.

> the training process doesn't seem to be replicable anyway The training process is fully deterministic. It's just an algorithm. Feed the same data in and you'll get the same weights out. If you're speaking about the computational cost, it used to be that way for compilers too. Give it 20 years and you'll be able to train one of today's models on your phone.

I’m pretty sure the initial weights are randomized meaning no two models will train in the same way twice. The order in which you feed in training data to the model would also add an element of randomness. Model training is closer to growing a plant than running a compiler.

That's still a deterministic algorithm. The random data and the order of feeding training data into it are part of the data which determines the output. Again, if you do it twice the same way, you'll get the same output.

Re: Deepseek R1-0528

#185

Post by reedciccio »

timschmidt wrote: post #184
willmarch wrote: post #183

Earlier quoted context omitted.

I’m pretty sure the initial weights are randomized meaning no two models will train in the same way twice. The order in which you feed in training data to the model would also add an element of randomness. Model training is closer to growing a plant than running a compiler.

That's still a deterministic algorithm. The random data and the order of feeding training data into it are part of the data which determines the output. Again, if you do it twice the same way, you'll get the same output.

> if you do it twice the same way, you'll get the same output

Point at the science that says that, please: Current scientific knowledge doesn't agree with you.

Re: Deepseek R1-0528

#186

Post by timschmidt »

reedciccio wrote: post #185
timschmidt wrote: post #184

Earlier quoted context omitted.

That's still a deterministic algorithm. The random data and the order of feeding training data into it are part of the data which determines the output. Again, if you do it twice the same way, you'll get the same output.

> if you do it twice the same way, you'll get the same output Point at the science that says that, please: Current scientific knowledge doesn't agree with you.

> Current scientific knowledge doesn't agree with you.

I'd love a citation. So far you haven't even suggested a possible source for this non-determinism you claim exists.

Re: Deepseek R1-0528

#187

Post by code_biologist »

pdimitar wrote: post #166
ribelo wrote: post #92

Earlier quoted context omitted.

livebench was good, but now it's a joke. Gemini flash is better in coding than pro and sonnet 3.7. And this is only the beginning of weird results.

Flash is better than Pro in coding? Whoa... [makes a note to try a few things later this day] Out of curiosity, how did you gauge that?

I think your parent comment is citing that as an example of why livebench is no longer a good benchmark. That said, the new Flash is very good for what it is, and IMO after the Pro 05-06 nerfs the two models are much closer in performance for many tasks than they really should be — Pro should be / was way better (RIP 03-25 release). That livebench result may be wrong about the specific ranking, but I think it's right that Flash is in the same class of coding strength as Sonnet 3.7.

Re: Deepseek R1-0528

#188

Post by willmarch »

timschmidt wrote: post #184
willmarch wrote: post #183

Earlier quoted context omitted.

I’m pretty sure the initial weights are randomized meaning no two models will train in the same way twice. The order in which you feed in training data to the model would also add an element of randomness. Model training is closer to growing a plant than running a compiler.

That's still a deterministic algorithm. The random data and the order of feeding training data into it are part of the data which determines the output. Again, if you do it twice the same way, you'll get the same output.

If they saved the initial randomized model and released it and there was no random bit flipping during copying, then possibly but it would still be difficult when you factor in the RLHF that comes about through random humans interacting with the model to tweak its workings. If you preserved that data as well, and got all of the initial training correct... maybe. But I'd bet against it.

Re: Deepseek R1-0528

#189

Post by pdimitar »

code_biologist wrote: post #187
pdimitar wrote: post #166

Earlier quoted context omitted.

Flash is better than Pro in coding? Whoa... [makes a note to try a few things later this day] Out of curiosity, how did you gauge that?

I think your parent comment is citing that as an example of why livebench is no longer a good benchmark. That said, the new Flash is very good for what it is, and IMO after the Pro 05-06 nerfs the two models are much closer in performance for many tasks than they really should be — Pro should be / was way better (RIP 03-25 release). That livebench result may be wrong about the specific ranking, but I think it's right…

Thanks, that's very informative.

My ignorance is showing here: why is the Pro 05-06 a nerf?

Re: Deepseek R1-0528

#190

Post by timschmidt »

willmarch wrote: post #188
timschmidt wrote: post #184

Earlier quoted context omitted.

That's still a deterministic algorithm. The random data and the order of feeding training data into it are part of the data which determines the output. Again, if you do it twice the same way, you'll get the same output.

If they saved the initial randomized model and released it and there was no random bit flipping during copying, then possibly but it would still be difficult when you factor in the RLHF that comes about through random humans interacting with the model to tweak its workings. If you preserved that data as well, and got all of the initial training correct... maybe. But I'd bet against it.

So long as the data provided was identical, and sources of error like floating point errors due to hardware implementation details are accounted for, I see no reason output wouldn't be identical.

Where would other non-determinism come from?

I'm open to there being another source. I'd just like to know what it would be. I haven't found one yet.

Post reply on HN