Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

History LLMs: Models trained exclusively on pre-1913 texts

github.com

Post reply on HN
Print view View original

361–370 of 452 posts

Re: History LLMs: Models trained exclusively on pre-1913 texts

#361

Post by root_axis »

red75prime wrote: post #165
root_axis wrote: post #153

Earlier quoted context omitted.

There's no evidence for it, nor any explanation for why it should be the case from a biological perspective. Tokens are an artifact of computer science that have no reason to exist inside humans. Human minds don't need a discrete dictionary of reality in order to model it. Prior to LLMs, there was never any suggestion that thoughts work like autocomplete, but now people are working backwards from that conclusion base…

There are so many theories regarding human cognition that you can certainly find something that is close to "autocomplete". A Hopfield network, for example. Roots of predictive coding theory extend back to 1860s. Natalia Bekhtereva was writing about compact concept representations in the brain akin to tokens.

> There are so many theories regarding human cognition that you can certainly find something that is close to "autocomplete"

Yes, you can draw interesting parallels between anything when you're motivated to do so. My point is that this isn't parsimonious reasoning, it's working backwards from a conclusion and searching for every opportunity to fit the available evidence into a narrative that supports it.

> Roots of predictive coding theory extend back to 1860s.

This is just another example of metaphorical parallels overstating meaningful connections. Just because next-token-prediction and predictive coding have the word "predict" in common doesn't mean the two are at all related in any practical sense.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#362

Post by xpe »

p-e-w wrote: post #207
bondarchuk wrote: post #204

> Historical texts contain racism, antisemitism, misogyny, imperialist views. The models will reproduce these views because they're in the training data. This isn't a flaw, but a crucial feature—understanding how such views were articulated and normalized is crucial to understanding how they took hold. Yes! > We're developing a responsible access framework that makes models available to researchers for scholarly purp…

It’s as if every researcher in this field is getting high on the small amount of power they have from denying others access to their results. I’ve never been as unimpressed by scientists as I have been in the past five years or so. “We’ve created something so dangerous that we couldn’t possibly live with the moral burden of knowing that the wrong people (which are never us, of course) might get their hands on it, so…

> It’s as if every researcher in this field is getting high on the small amount of power they have from denying others access to their results.

Even if I give the comment a lot of wiggle room (such as changing "every" to "many"), I don't think even a watered-down version of this hypothesis passes Occam's razor. There are more plausible explanations, including (1) genuine concern by the authors; (2) academic pressures and constraints; (c) reputational concerns; (d) self-interest to embargo underlying data so they have time to be the first to write-it-up. To my eye, none of these fit the category of "getting high on power".

Also, patience is warranted. We haven't seen what these researchers are doing to release -- and from what I can tell, they haven't said yet. At the moment I see "Repositories (coming soon)" on their GitHub page.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#363

Post by simonask »

qcnguy wrote: post #323
tombh wrote: post #308

Earlier quoted context omitted.

Of course, I have to assume that you have considered more outcomes than I have. Because, from my five minutes of reflection as a software geek, albeit with a passion for history, I find this the most surprising thing about the whole project. I suspect restricting access could equally be a comment on modern LLMs in general, rather than the historical material specifically. For example, we must be constantly reminded n…

They aren't afraid of hallucinations. Their first example is a hallucination, an imaginary biography of a Hitler who never lived. Their concern can't be understood without a deep understanding of the far left wing mind. Leftists believe people are so infinitely malleable that merely being exposed to a few words of conservative thought could instantly "convert" someone into a mortal enemy of their ideology for life. I…

You know, I actually sympathize with the opinion that people should be expected and assumed to be able to resist attempts to convince them of being nazis.

The problem with it is, it already happened at least once. We know how it happened. Unchecked narratives about minorities or foreigners is a significant part of why the 20th century happened to Europe, and it’s a significant part of why colonialism and slavery happened to other places.

What solution do you propose?

Re: History LLMs: Models trained exclusively on pre-1913 texts

#364

Post by monocasa »

BiteCode_dev wrote: post #192
varjag wrote: post #184

Earlier quoted context omitted.

There's a thriving startup scene in that direction.

Wasn't that the elevator pitch for Palentir? Still can't believe people buy their stock, given that they are the closest thing to a James Bond villain, just because it goes up. I mean, they are literally called "the stuff Sauron uses to control his evil forces". It's so on the nose it reads like an anime plot.

It goes a bit deeper than that since they got funding in the wake of 9/11 and the requests for intelligence and investigative branches of government to do better and coalescing their information to prevent attacks.

So "panopticon that if it had been used properly, would have prevented the destruction of two towers" while ignoring the obvious "are we the baddies?"

Re: History LLMs: Models trained exclusively on pre-1913 texts

#365

Post by schlauerfox »

myrmidon wrote: post #304
DGoettlich wrote: post #289

Earlier quoted context omitted.

fully understand you. we'd like to provide access but also guard against misrepresentations of our projects goals by pointing to e.g. racist generations. if you have thoughts on how we should do that, perhaps you could reach out at history-llms@econ.uzh.ch ? thanks in advance!

What is your worst-case scenario here? Something like a pop-sci article along the lines of "Mad scientists create racist, imperialistic AI"? I honestly don't see publication of the weights as a relevant risk factor, because sensationalist misrepresentation is trivially possible with the given example responses alone. I don't think such pseudo-malicious misrepresentation of scientific research can be reliably prevente…

It seems like if there is an obvious misuse of a tool, one has a moral imperative to restrict use of the tool.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#367

Post by 999900000999 »

DGoettlich wrote: post #273
seizethecheese wrote: post #119

> Imagine you could interview thousands of educated individuals from 1913—readers of newspapers, novels, and political treatises—about their views on peace, progress, gender roles, or empire. Not just survey them with preset questions, but engage in open-ended dialogue, probe their assumptions, and explore the boundaries of thought in that moment. Hell yeah, sold, let’s go… > We're developing a responsible access fra…

understand your frustration. i trust you also understand the models have some dark corners that someone could use to misrepresent the goals of our project. if you have ideas on how we could make the models more broadly accessible while avoiding that risk, please do reach out @ history-llms@econ.uzh.ch

Ok...

So as a black person should I demand that all books written before the civil rights act be destroyed?

The past is messy. But it's the only way to learn anything.

All an LLM does it's take a bunch of existing texts and rebundle them. Like it or not, the existing texts are still there.

I understand an LLM that won't tell me how to do heart surgery. But I can't fear one that might be less enlightened on race issues. So many questions to ask! Hell, it's like talking to older person in real life.

I don't expect a typical 90 year old to be the most progressive person, but they're still worth listening too.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#368

Post by j-bos »

dr_dshiv wrote: post #223
j-bos wrote: post #210

Earlier quoted context omitted.

This ia very cool but should go in a Show HN post as per HN rules. All the best!

Just read the rules again— was something inappropriate? Seemed relevant

I can see you being right, I didn't make the connection with 20th,19th century documents and the comment felt disconnected from the thread. Either way, very cool project, worth a show hn post.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#369

Post by xpe »

paddleon wrote: post #296
p-e-w wrote: post #207

Earlier quoted context omitted.

It’s as if every researcher in this field is getting high on the small amount of power they have from denying others access to their results. I’ve never been as unimpressed by scientists as I have been in the past five years or so. “We’ve created something so dangerous that we couldn’t possibly live with the moral burden of knowing that the wrong people (which are never us, of course) might get their hands on it, so…

> “We’ve created something so dangerous that we couldn’t possibly live with the moral burden of knowing that the wrong people (which are never us, of course) might get their hands on it, so with a heavy heart, we decided that we cannot just publish it.” Or, how about, "If we release this as is, then some people will intentionally mis-use it and create a lot of bad press for us. Then our project will get shut down and…

  > Be careful assuming it is a power trip when
  > it might be a fear trip.
  >
  > I've never been as unimpressed by society as
  > I have been in the last 5 years or so.
Is the second sentence connected to the first? Help me understand?

When I see individuals acting out of fear, I try not to blame them. Fear triggers deep instinctual responses. For example, to a first approximation, a particular individual operating in full-on fight-or-flight mode does not have free will. There is a spectrum here. Here's a claim, which seems mostly true: the more we can slow down impulsive actions, the more hope we have for cultural progress.

When I think of cultural failings, I try to criticize areas where culture could realistically do better. I think of areas where we (collectively) have the tools and potential to do better. Areas where thoughtful actions by some people turn into a virtuous snowball. We can't wait for a single hero, though it helps to create conditions so that we have more effective leaders.

One massive culture failing I see -- that could be dramatically improved -- is this: being lulled into shallow contentment (i.e. via entertainment, power seeking, or material possessions) at the expense of (i) building deep and meaningful social connections and (ii) using our advantages to give back to people all over the world.

Re: History LLMs: Models trained exclusively on pre-1913 texts

#370

Post by aidenn0 »

somenameforme wrote: post #128
Forgeties79 wrote: post #102

Earlier quoted context omitted.

It’s a big country of roughly half a billion people, you’ll always find examples if you look hard enough. It’s ridiculous/wrong that your district did this but frankly it’s the exception in liberal/progressive communities. It’s a very one-sided problem: * https://abcnews.go.com/US/conservative-liberal-book-bans-dif... * https://www.commondreams.org/news/book-banning-2023 * https://en.wikipedia.org/wiki/Book_banning_i…

A practical issue is the sort of books being banned. Your first link offer examples of one side trying to ban Of Mice and Men, Adventures of Huckleberry Finn, and Dr. Seuss, with the other side trying to ban many books along the lines of Gender Queer. [1] That link is to the book - which is animated, and quite NSFW. There are a bizarrely large number similar book as Gender Queer being published, which creates the num…

[dead]
Post reply on HN