191–200 of 463 posts
At this point we might as well adopt that joke where you blindly throw away half the resumes because you don't want to hire unlucky people.
A person's total luck is constant over a lifetime. The remaining half of the candidates already spent some of their luck in this selection, so they'll be on average less lucky than the discarded half.
Earlier quoted context omitted.
And this lies at the heart of the problem. We expect computers to be consistent despite running programs that are not designed to be consistent. This despite the fact that we have lots of experience of programs running on computers that produces wildly inconsistent outputs. But for some reason some people choose to assume LLMs should act like a calculator instead of any of those programs.
> This despite the fact that we have lots of experience of programs running on computers that produces wildly inconsistent outputs. The average user has very little. A word processor with inconsistent pagination or a spreadsheet with inconsistent totals is rightly seen as faulty.
An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says: > temperature 0.1 — low, supposedly nudging the model toward deterministic outputs This is…
[flagged]
Earlier quoted context omitted.
[flagged]
Of course life isn't fair. But here the result is that companies will ignore potentially great candidates which dedicate all their programming time to their job and instead consider candidates which may be not just worse programmers, but also are more interested in their hobbies (or padding their CV) that doing their job. I'm saying this as somebody who most of the time has some side project going on.
The takeaway from this for me is that, using an LLM to score anything takes multiple (maybe even many) runs and the result you’ll get is, at best, a sane-ish distribution. Which sort of sounds workable until you scale it up to larger datasets, where at some point compute/time/energy costs will render it non-viable. I am sure there’s some reasonable rule of thumb estimation on distribution that could be applied based…
Earlier quoted context omitted.
>in theory theory, temperature 0 doesn't really exist. It does exist very much, even if you go to pure math. Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. I.e. in a discrete setting (like for LLMs with a finite set of output tokens), probability P becomes one for argmax and 0 for everything else. Only in coding practice it is easer to implement T=0 as a simple if check th…
> Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. In pure math, it does not always do that. It becomes a dirac-delta comb with equal weight on every maximum. There can be more than 1 maximum. Setting the temperature to zero turns into greedy sampling, but greedy sampling is not necessarily deterministic as you can have multiple equally optimal options.
Even better Wikipedia lists the abbreviation I am familiar with but give a different interpretation of the same words:
Earlier quoted context omitted.
a studied example is sampling judicial decisions before lunch and after lunch. judges are more lenient on a full stomach.
how did they account for sampling bias? a judge might leave easier cases for after lunch. people with control over their schedules usually ease themselves back into it after breaks.
Earlier quoted context omitted.
Wdym, cant you just litter your applications with buzzwords and other bs to automatically get a high score in these systems?
I have been at both side of the market. And it sucks so bad at both ends. Companies which deeply care about next hire are struggling to hire and actual great people looking out are outcompeted by AI slop and AI bulk applying. It is actually a very hard to solve problem.
Chickens coming home to roost.
An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says: > temperature 0.1 — low, supposedly nudging the model toward deterministic outputs This is…
Provided:
* If it’s MoE we are talking about, that the duplicate inputs are for the whole batch (yes, your batch neighbours can impact your choice of experts. Blergh.)
* Your kernels are deterministic
* There’s no system wide effort switch that responds to, e.g. work load across the cluster (for a thinking model)
Upshot:
Temperature 0 is not deterministic in probably any existing cloud infra, but it could be for edge inference pretty reliably.
To your quibble on 0.1 being more deterministic - I think it’s a pretty fair summary - we’re going to sample much more from the ‘temp 0’ answer at 0.1 than we would at temp 0.9, no?