Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The LLM Lobotomy?

learn.microsoft.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: The LLM Lobotomy?

#51

Post by mr_toad »

thomasmg wrote: post #41
mehdibl wrote: post #30

Since when LLM become deterministic?

LLM are just software + data and can be made deterministic, in the same way a pseudo random number generator can be made deterministic by using the same seed. For an LLM, you typically set temperature to 0, or set the random seed to the same value, run it on the same hardware (or emulation) or otherwise ensure the (floating point) calculations get the exact same results. I think that's it. In reality, yes it's not th…

Unfortunately because floating point addition isn’t always associative, and because GPUs don’t always perform calculations in the same order you won’t always get the same result even with a temperature of zero.

Re: The LLM Lobotomy?

#52

Post by juliangoldsmith »

I've been using Azure AI Foundry for an ongoing project, and have been extremely dissatisfied.

The first issue I ran into was with them not supporting LLaMA for tool calls. Microsoft stated in February that they were working on it [0], and they were just closing the ticket because they were tracking it internally. I'm not sure why they've been unable to do what took me two hours in over six months, but I am sure they wouldn't be upset by me using the much more expensive OpenAI models.

There are also consistent performance issues, even on small models, as mentioned elsewhere. This is with a rate on the order of one per minute. You can solve that with provisioned throughput units. The cheapest option is one of the GPT models, at a minimum of $10k/month (a bit under half the cost of just renting an A100 server). DeepSeek was a minimum of around $72k/month. I don't remember there being any other non-OpenAI models with a provisioned option.

Given that current usage without provisioning is approximately in the single dollars per month, I have some doubts as to whether we'd be getting our money's worth having to provision capacity.

Re: The LLM Lobotomy?

#53

Post by TZubiri »

cjtrowbridge wrote: post #49

This brings up a point many will not be aware of. If you know the random seed and the prompt, and the hash of the model's binary file; the output is completely deterministic. You can use this information to check whether they are in fact swapping your requests out to cheaper models than what you're paying for. This level of auditability is a strong argument for using open-source, commodified models, because you can e…

Pretty sure this is wrong, requests are batched and size can affect the output, also gpus are highly parallel, there can be many race conditions.

Re: The LLM Lobotomy?

#54

Post by maxbond »

fortyseven wrote: post #25
cush wrote: post #8

Is setting temperature to 0 even a valid way to measure LLM performance over time, all else equal?

I'd have assumed a fixed seed was used, but he doesn't mention that. Weird. Maybe he meant that?

I bet they did both. If I'm reading the documentation right you have to supply a seed in order to get "best effort" determinism.

https://learn.microsoft.com/en-us/azure/ai-foundry/openai/re...

Re: The LLM Lobotomy?

#55

Post by lostmsu »

vintermann wrote: post #21
briga wrote: post #10

I have a theory: all these people reporting degrading model quality over time aren't actually seeing model quality deteriorate. What they are actually doing is discovering that these models aren't as powerful as they initially thought (ie. expanding their sample size for judging how good the model is). The probabilistic nature of LLM produces a lot of confused thinking about how good a model is, just because a model…

They test specific prompts with temperature 0. It is of course possible that all their tests prompts were lucky, but still then, shouldn't you see an immediate drop followed by a flat or increasing line? Also, from what I understand from the article, it's not a difficult task but an easily machine checkable one, i.e. whether the output conforms to a specific format.

With T=0 on the same model you should get the same exact output text. If they are not getting it, other environmental factors invalidate the test result.

Re: The LLM Lobotomy?

#56

Post by TeMPOraL »

TZubiri wrote: post #53
cjtrowbridge wrote: post #49

This brings up a point many will not be aware of. If you know the random seed and the prompt, and the hash of the model's binary file; the output is completely deterministic. You can use this information to check whether they are in fact swapping your requests out to cheaper models than what you're paying for. This level of auditability is a strong argument for using open-source, commodified models, because you can e…

Pretty sure this is wrong, requests are batched and size can affect the output, also gpus are highly parallel, there can be many race conditions.

Yup. Floating point math turns race conditions into numerical errors, reintroducing non-determinism regardless of inputs used.

Re: The LLM Lobotomy?

#57

Post by sebazzz »

bongodongobob wrote: post #26
SubiculumCode wrote: post #7

Earlier quoted context omitted.

I don't remember where I saw it, but I remember a claim that Azure hosted models performed poorer than those hosted by openAI.

They most definitely do. They have been lobotomized in some way to be ultra corporate friendly. I can only use their M365 Copilot at work and it's absolute dogshit at writing code more than maybe 100 lines. It can barely write correct PowerShell. Luckily, I really only need it for quick and dirty short PS scripts.

I agree. I asked it for some help refactoring a database and some of the SQL is quite broken. It also doesn't help that their streaming code is broken so LLM responses sometimes end up broken in the web browser (both Firefox and Edge so it is not a browser issue), so you need to refresh after a response to make sure the LLMs response was not an indication of a drunk LLM.

Re: The LLM Lobotomy?

#58

Post by gregsadetsky »

gregsadetsky wrote: post #29
esafak wrote: post #2

This was the perfect opportunity to share the evidence. I think undisclosed quantization is definitely a thing. We need benchmarks to be periodically re-evaluated to ward against this. Providers should keep timestamped models fixed, and assign modified versions a new timestamp, and price, if they want. The model with the "latest" tag could change over time, like a Docker image. Then we can make an informed decision o…

I commented on the forum asking Sarge whether they could share some of their test results. If they do, I think that it will add a lot to this conversation. Hope it happens!

Update: Sarge responded in the forum and added more information.

I asked them to share data/dates as much as that’s possible - fingers crossed

Re: The LLM Lobotomy?

#59

Post by criemen »

visarga wrote: post #32
criemen wrote: post #27

Earlier quoted context omitted.

Even with temperature 0, the LLM output will not be deterministic. It will just have less randomness (not defined precisely) than with temperature 1. There was a recent post on the frontpage about fully deterministic sampling, but it turns out to be quite difficult.

It's because batch size is dynamic. So a different batch size will change the output even on temp 0.

Batch size is dynamic, in MoE apparently the experts chosen depend on the batch (not only your single inference request, which sounds weird to me, but I'm just an end user), no one audited the inference pipeline for floating point nondeterminisms, and I'm not even sure that temperature 0 implies deterministic sampling (the quick math formula I found has e^(1/temp) which means that 0 is not a valid value anyways and would need some dealing with).

Re: The LLM Lobotomy?

#60

Post by colordrops »

Aurornis wrote: post #23
colordrops wrote: post #12

Earlier quoted context omitted.

Did any of you read the article? They have a test framework that objectively shows the model getting worse over time.

I read the article. No proof was included. Not even a graph of declining results.

Ok fair, but not including the data is not the same as the article saying it was subjective "feel".
Post reply on HN