Everything around LLMs is still magical and wishful thinking • Hacker News Board

Re: Everything around LLMs is still magical and wishful thinking

#341

Post by itsafarqueue » Sun, Jul 06, 2025, 7:52 AM UTC

hotpotat wrote: post #30

I have to say I’m in the exact camp the author is complaining about. I’ve shipped non trivial greenfield products which I started back when it was only ChatGPT and it was shitty. I started using Claude with copying and pasting back and forth between the web chat and XCode. Then I discovered Cursor. It left me with a lot of annoying build errors, but my productivity was still at least 3x. Now that agents are better an…

More anecdata: +1 for “LLMs write all my production code now”. 25+ years in industry, as expert as it’s possible to be in my domain. 100% agree LLMs fail hilariously badly, often, and dangerously. And still, write ~all my code.

No agenda here, not selling anything. Just sitting here towards the later part of my career, no need to prove anything to anyone, stating the view from a grey beard.

Crypto hype was shill from grifters pumping whatever bag holding scam they could, which was precisely what the behavioral economic incentives drove. GenAI dev is something else. I’ve watched many people working with it, your mileage will vary. But in my opinion (and it’s mine, you do you), hand coding is an apocryphal skill. The only part I wonder about is how far up and down the system/design/architecture stack the power-tooling is going to go. My intuition and empirical findings incline towards a direction I think would fuel a flame war. But I’m just grey beard Internet random, and hey look, no evidence just more baseless claims. Nothing to see here.

Disclosure: I hold no direct shares in Mag 7, nor do I work for one.

Re: Everything around LLMs is still magical and wishful thinking

#342

Post by TeMPOraL » Sun, Jul 06, 2025, 8:10 AM UTC

alganet wrote: post #334

TeMPOraL wrote: post #332

Earlier quoted context omitted.

LLMs handle major trappings of culture just fine. As long as a culture has enough of a footprint in terms of written words, the LLM probably knows it better than any single individual, even though it has not lived it.

No, it does not know culture. And no, it can't handle talking about it. Ask an LLM "Can you compare egyptian mythology with aliens?" and they will happily do it: https://imgur.com/a/jfikuEO That's an offensive, pseudoscientific view on egyptian culture shunned by academics. Even ChatGPT "Critical Viewpoint" section (a small part of a large bullshit response) _still_ entertains offensive ideas: https://imgur.com/a/RNi…

> That's an offensive, pseudoscientific view on egyptian culture shunned by academics.

You must be joking. Specifically, you're either:

1) Pretending to be unaware of the existence of Stargate - one of the bigger and more well-known sci-fi media franchise, whose premise is literally that Egyptian gods were actually malevolent aliens that enslaved people from ancient times and, and the Egyptian mythology is mostly factual and rooted in that experience. The franchise literally starts with (spoiler alert) humans killing Ra with a tactical nuke, and gets only better from there;

2) Play-acting the smug egyptologists who rolled their eyes or left in an outrage, when one Daniel Jackson started hinting that the great pyramids were actually landing pads for alien starships. Which they were. In Stargate.

Not that this is a particularly original thought; ancient aliens/ancient astronauts are an obvious idea that has been done to death and touch every culture. Stargate did that with Egyptian mythology, and Nordic mythology, and Aztec history, and Babylon and even King Arthur stories. Star Trek did that with Greek gods. Battlestar: Galactica, with entire Greek mythology. Arthur C. Clarke took a swipe at Christianity. And those are all well-known works.

I could go on. The thoughts you complain about are perfectly normal and common and show up frequently. People speculate like that because it's fun, it makes the stories plausible instead of insane or nonsense (or in some mythologies mentioned above, products of a sick imagination), and is not boring.

--

If I may be frank, views like yours, expressed like you did, scare me. Taking offense like this - whether honestly or performatively - is belligerent and destructive to the fabric of society and civilization. We've had enough of that in the past 15 years; I was really hoping people grew up out of the phase of being offended by everything.

Re: Everything around LLMs is still magical and wishful thinking

#343

Post by TeMPOraL » Sun, Jul 06, 2025, 8:12 AM UTC

alganet wrote: post #338

stevenhuang wrote: post #336

Earlier quoted context omitted.

There legitimately is a lot of cross over between Egyptian mythology and other high strangeness phenomenon as understood culturally though, such as aliens/ufos. I think you did just demonstrate you know less about culture than LLMs, which is not at all unsurprising.

Dude, I chose this example precisely because I know for a fact there is a lot of bullshit about it on the internet and LLMs cannot differentiate between a good source and a bad source. This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran? Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do…

> Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them).

Nah, that's just LLMs being trained to acquiesce to the insanity of the last ~15 years, as many people seem to expect that claiming you're offended by something is an ultimate argument that everyone must yield to (and they'll keep making a fuss out of it until they do).

Re: Everything around LLMs is still magical and wishful thinking

#344

Post by alt187 » Sun, Jul 06, 2025, 8:20 AM UTC

lubujackson wrote: post #316

alt187 wrote: post #99

Earlier quoted context omitted.

I agree about the 50/50 thing. It's about how much Claude helped me, and I use it daily too . I'll give some context, though. - I use OCaml and Python/SQL, on two different projects. - Both are single-person. - The first project is a real-time messaging system, the second one is logging a bunch of events in an SQL database. In the first project, Claude has been... underwhelming. It casually uses C idioms, overabuses…

Just FYI, try commenting on that function what it is intended to be used for. Because without more info LLMs will rely on function names strongly. Heck, have the LLM add comments to every function and I bet it will start to do better.

It's not my function in the example, it's a standard library function. It does have a weird name though.

Re: Everything around LLMs is still magical and wishful thinking

#345

Post by ben_w » Sun, Jul 06, 2025, 8:40 AM UTC

TeMPOraL wrote: post #332

ben_w wrote: post #330

Earlier quoted context omitted.

That kind of knowledge is present in the training set, doesn't need to be in the context or system prompt. That said, I too would only use an LLM today in the same kinds of role that five years ago would be outsourced to a different culture. Culture, not even language: this is how you get the difference between "biscuits and gravy" as understood in the UK vs in the USA.

LLMs handle major trappings of culture just fine. As long as a culture has enough of a footprint in terms of written words, the LLM probably knows it better than any single individual, even though it has not lived it.

Looking at your other comment sibling to mine, I think part of the difficulty discussing these topics is how much these things are considered isolated magic artefacts (bad for engineering) or one tool amongst many where the magic word is "synergy".

So I agree with you: LLMs do know all written cultures on the internet and can mimic them acceptably — but they only actually do so when this is requested by some combination of the fine-tuning, RLHF, system prompt, and context.

In your example, having some current news injected, which is easy, but actually requires someone to plumb that in. And as you say, you'd not do that unless you thought you needed to.

But even easier to pick, lower-hanging fruit, often gets missed. When the "dangerous sycophancy" behaviour started getting in the news, I updated my custom ChatGPT "traits" setting to this:

  Honesty and truthfulness are of primary importance. Avoid American-style positivity, instead aim for German-style bluntness: I absolutely *do not* want to be told everything I ask is "great", and that goes double when it's a dumb idea.

But cultural differences can be subtle, and there's a long tail of cultural traits of the same kind that means 1980s Text Adventure NLP doesn't scale to what ChatGPT itself does. While this can still be solved with fine-tuning or getting your staff to RLHF it, the number of examples current AI need in order to learn is high compared to a real human, so it won't learn your corporate culture from experience *as fast* as a new starter within your team, unless you're a sufficiently big corporation that it can be on enough (I don't know how many exactly) teams within your company at the same time.

Re: Everything around LLMs is still magical and wishful thinking

#346

Post by [deleted] » Sun, Jul 06, 2025, 9:13 AM UTC

alganet wrote: post #334

TeMPOraL wrote: post #332

Earlier quoted context omitted.

LLMs handle major trappings of culture just fine. As long as a culture has enough of a footprint in terms of written words, the LLM probably knows it better than any single individual, even though it has not lived it.

No, it does not know culture. And no, it can't handle talking about it. Ask an LLM "Can you compare egyptian mythology with aliens?" and they will happily do it: https://imgur.com/a/jfikuEO That's an offensive, pseudoscientific view on egyptian culture shunned by academics. Even ChatGPT "Critical Viewpoint" section (a small part of a large bullshit response) _still_ entertains offensive ideas: https://imgur.com/a/RNi…

[deleted]

Re: Everything around LLMs is still magical and wishful thinking

#347

Post by ben_w » Sun, Jul 06, 2025, 9:17 AM UTC

alganet wrote: post #338

stevenhuang wrote: post #336

Earlier quoted context omitted.

There legitimately is a lot of cross over between Egyptian mythology and other high strangeness phenomenon as understood culturally though, such as aliens/ufos. I think you did just demonstrate you know less about culture than LLMs, which is not at all unsurprising.

Dude, I chose this example precisely because I know for a fact there is a lot of bullshit about it on the internet and LLMs cannot differentiate between a good source and a bad source. This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran? Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do…

> they just can't do it _before_ you correct them

The screenshot literally shows the LLM used bold text for "not supported by mainstream archaeology or Egyptology."

> LLMs cannot differentiate between a good source and a bad source.

Can you?

A not insignificant part of my history GCSE was just this: how to tease apart truth from all the sources, primary and secondary, which had their own biases and goals in the telling. It was an optional subject even at that level, and even subsequently in adulthood there were a lot of surprises left for me about the history of the British Empire, surprises that I didn't learn until decades later when I met people from former colonies who were angry about what was done to them by my parents' and grandparents' generations.

It's not a coincidence that the world "story" is contained within "history", the etymology is the same: https://en.wiktionary.org/wiki/story vs. https://en.wiktionary.org/wiki/history

Likewise in German where both concepts are the same word: https://en.wiktionary.org/wiki/Geschichte

My mum was New-Age type, had books of the general category that would include alien pyramids, though I don't know if that specific was included. She didn't know what she didn't know, and therefore kept handing out expensive homeopathic sand and salt tablets to family members (the bottles literally had "sodium chloride" and "titanium dioxide" printed on them).

People, including you and I, don't know what they don't know.

Re: Everything around LLMs is still magical and wishful thinking

#348

Post by alganet » Sun, Jul 06, 2025, 1:39 PM UTC

TeMPOraL wrote: post #342

alganet wrote: post #334

Earlier quoted context omitted.

No, it does not know culture. And no, it can't handle talking about it. Ask an LLM "Can you compare egyptian mythology with aliens?" and they will happily do it: https://imgur.com/a/jfikuEO That's an offensive, pseudoscientific view on egyptian culture shunned by academics. Even ChatGPT "Critical Viewpoint" section (a small part of a large bullshit response) _still_ entertains offensive ideas: https://imgur.com/a/RNi…

> That's an offensive, pseudoscientific view on egyptian culture shunned by academics. You must be joking. Specifically, you're either: 1) Pretending to be unaware of the existence of Stargate - one of the bigger and more well-known sci-fi media franchise, whose premise is literally that Egyptian gods were actually malevolent aliens that enslaved people from ancient times and, and the Egyptian mythology is mostly fac…

You're full of shit.

Let's see what LLMs say when I correct them:

https://imgur.com/a/R7DXCRx

So, either way you're wrong. If ancient aliens were innofensive, the LLM should have not accepted the correction.

_LLMs cannot handle talking about culture_, and neither can you.

Re: Everything around LLMs is still magical and wishful thinking

#349

Post by alganet » Sun, Jul 06, 2025, 1:43 PM UTC

ben_w wrote: post #347

alganet wrote: post #338

Earlier quoted context omitted.

Dude, I chose this example precisely because I know for a fact there is a lot of bullshit about it on the internet and LLMs cannot differentiate between a good source and a bad source. This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran? Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do…

> they just can't do it _before_ you correct them The screenshot literally shows the LLM used bold text for "not supported by mainstream archaeology or Egyptology." > LLMs cannot differentiate between a good source and a bad source. Can you? A not insignificant part of my history GCSE was just this: how to tease apart truth from all the sources, primary and secondary, which had their own biases and goals in the telli…

Remember, the claim I challenged is that LLMs know culture and can handle talking about them. You need to focus.

Most people are not prepared to handle talking about culture. So, LLMs also aren't.

They are not any better than asking an average person, will make mistakes, will disappoint.

Egyptologists are better equipped to talk about egyptian myths. LLMs cannot handle egyptian culture as well as they can.

Re: Everything around LLMs is still magical and wishful thinking

#350

Post by alganet » Sun, Jul 06, 2025, 1:44 PM UTC

TeMPOraL wrote: post #343

alganet wrote: post #338

Earlier quoted context omitted.

Dude, I chose this example precisely because I know for a fact there is a lot of bullshit about it on the internet and LLMs cannot differentiate between a good source and a bad source. This is honestly unbelievable. You're defending ancient aliens. What's next? Heavens Gate? Ashtar Sheran? Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do…

> Even the LLMs themselves acknowledge that this is regarded as offensive. If you correct it, it will apologize (they just can't do it _before_ you correct them). Nah, that's just LLMs being trained to acquiesce to the insanity of the last ~15 years, as many people seem to expect that claiming you're offended by something is an ultimate argument that everyone must yield to (and they'll keep making a fuss out of it un…

An LLM that offends people and entire nations is what many would classify as _misaligned_.

Let's say I have a company, and my company needs to comply to government policy regarding communication. I cannot trust LLMs then, they will either fail to follow policy, or acquiesce to any group that tries to game it.

It's useless garbage. Egyptian myths were just an example, you don't need to bite it so hard.