331–340 of 377 posts
Earlier quoted context omitted.
Qwen 2.5 32B which is an older model at this point clearly outperforms it: https://llm-stats.com/models/compare/gpt-3.5-turbo-0125-vs-q...
Even when quantized down to 4 bits to fit on a 4090?
Earlier quoted context omitted.
> considering "out-of-context reasons" and taking a wider perspective is pretty much the defining aspect of LLMs as general-purpose AI tools. "out-of-context" literally means that the reason isn't in its context. Even if it can make the leap that the number should be zero if it's a bank holiday, how would an LLM know that yesterday was a one-off bank holiday? A human would only know through their lived experience tha…
That kind of knowledge is present in the training set, doesn't need to be in the context or system prompt. That said, I too would only use an LLM today in the same kinds of role that five years ago would be outsourced to a different culture. Culture, not even language: this is how you get the difference between "biscuits and gravy" as understood in the UK vs in the USA.
Earlier quoted context omitted.
> An LLM won't call other nodes in the organization to check when it sees that the value is unreasonable for some out-of-context reason, like yesterday was a one-time-only bank holiday and so the value should be 0. Why not? LLMs are the first kind of technology that can take this kind of global view. We're not making much use of it in this way just yet, but considering "out-of-context reasons" and taking a wider pers…
> considering "out-of-context reasons" and taking a wider perspective is pretty much the defining aspect of LLMs as general-purpose AI tools. "out-of-context" literally means that the reason isn't in its context. Even if it can make the leap that the number should be zero if it's a bank holiday, how would an LLM know that yesterday was a one-off bank holiday? A human would only know through their lived experience tha…
Depends. Was it a one-off holiday announced at 11th our or something? Then it obviously won't know. You'd need extra setup to enable it to realize that, such as e.g. first feeding an LLM the context of your task and a digest of news stories spanning a week, asking it to find if there's anything potentially relevant, and then appending that output to the LLM calls doing the work. It's not something you'd do by default in general case, but that's only because tokens cost money and context space is scarce.
Is it a regular bank holiday? Then all it would need is today's date in the context, which is often just appended somewhere between system and user prompts, along with e.g. user location data.
I see that by "out-of-context reasons" you meant the first case; I read it as a second. In the second case, the "out-of-context" bit could be the fact that a bank holiday could alter the entry for that day; if that rule is important or plausible enough but not given explicitly in the prompt, the model will learn it during training, and will likely connect the dots. This is what I meant as the "defining aspect of LLMs as general-purpose AI tools".
The flip side is, when it connects the dots when it shouldn't, we say it's hallucinating.
Earlier quoted context omitted.
That kind of knowledge is present in the training set, doesn't need to be in the context or system prompt. That said, I too would only use an LLM today in the same kinds of role that five years ago would be outsourced to a different culture. Culture, not even language: this is how you get the difference between "biscuits and gravy" as understood in the UK vs in the USA.
LLMs handle major trappings of culture just fine. As long as a culture has enough of a footprint in terms of written words, the LLM probably knows it better than any single individual, even though it has not lived it.
Ask an LLM "Can you compare egyptian mythology with aliens?" and they will happily do it:
That's an offensive, pseudoscientific view on egyptian culture shunned by academics.
Even ChatGPT "Critical Viewpoint" section (a small part of a large bullshit response) _still_ entertains offensive ideas:
They should have answered that such comparisons are potentially offensive, and explained why academia thinks so, _before_ spilling out nonsense.
Earlier quoted context omitted.
I use Claude code for hours a day, it’s a liar, trust what it does at your own risk. I personally think you’re sugar coating the experience.
> I use Claude code for hours a day, it’s a liar, trust what it does at your own risk. The person you're responding to literally said, "I audit everything myself before making PRs and test rigorously".
Earlier quoted context omitted.
LLMs handle major trappings of culture just fine. As long as a culture has enough of a footprint in terms of written words, the LLM probably knows it better than any single individual, even though it has not lived it.
No, it does not know culture. And no, it can't handle talking about it. Ask an LLM "Can you compare egyptian mythology with aliens?" and they will happily do it: https://imgur.com/a/jfikuEO That's an offensive, pseudoscientific view on egyptian culture shunned by academics. Even ChatGPT "Critical Viewpoint" section (a small part of a large bullshit response) _still_ entertains offensive ideas: https://imgur.com/a/RNi…
I think you did just demonstrate you know less about culture than LLMs, which is not at all unsurprising.
Earlier quoted context omitted.
You are correct that review and validation should still be manual. But the actual "translation" from one format to another should be automated with llms
You seem to be missing that the translation part isn't the expensive part, it's the validation and review. Separately, maybe this is just me, but having data actually flow through my hands is necessary for full comprehension. Just skimming an automated result, my brain doesn't actually process like half of that data. Making the process more efficient in this way can make my actual review performance *much worse.* The…
This will often be a giant excel spreadsheet or if you are lucky something like Microsoft Access.
They are absolutely riddled with mistakes as is with humans in the loop.
I think this is one of the core issues with HNers evaluating LLMs. I'm not entirely some of them have ever seen how ramshackle 90%+ of operations are.
Earlier quoted context omitted.
No, it does not know culture. And no, it can't handle talking about it. Ask an LLM "Can you compare egyptian mythology with aliens?" and they will happily do it: https://imgur.com/a/jfikuEO That's an offensive, pseudoscientific view on egyptian culture shunned by academics. Even ChatGPT "Critical Viewpoint" section (a small part of a large bullshit response) _still_ entertains offensive ideas: https://imgur.com/a/RNi…
There legitimately is a lot of cross over between Egyptian mythology and other high strangeness phenomenon as understood culturally though, such as aliens/ufos. I think you did just demonstrate you know less about culture than LLMs, which is not at all unsurprising.
This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran?
Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them).
You're wrong.
One thing I find frustrating is that management where I work has heard of 10x productivity gains. Some of those claims even come from early adopters at my work. But that sets expectation way too high. Partly it is due to Amdahl's law: I spend only a portion of my time coding, and far more time thinking and communicating with others that are customers of my code. Even if does make the coding 10x faster (and it doesn't…
Earlier quoted context omitted.
Your comment is no better than the comment in the article that the author is calling out. "90%" also seems a bit suspect.
It either helps me find a solution or it doesn't. About 90% of the time, or less formally I would just say "almost all of the time", it does. Keep in mind that I, the user, decide which questions to ask in the first place. If my batting average seems unbelievably high, perhaps my skill is in knowing when to use an LLM and when not to.