So, could this be an example of an LLM trained fully on public domain copyright-expired data? Or is this not intended to be the case.
data is 100% public domain.
History LLMs: Models trained exclusively on pre-1913 texts
401–410 of 452 posts
Re: History LLMs: Models trained exclusively on pre-1913 texts
#402Earlier quoted context omitted.
This is the 2023 take on LLMs. It still gets repeated a lot. But it doesn’t really hold up anymore - it’s more complicated than that. Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. Sure, LLMs do not think like humans and they may not have human-level creativity. Sometimes…
> Don’t let some factoid about how they are pretrained on autocomplete-like next token prediction fool you into thinking you understand what is going on in that trillion parameter neural network. This is just an appeal to complexity, not a rebuttal to the critique of likening an LLM to a human brain. > they are not “autocomplete on steroids” anymore either. Yes, they are. The steroids are just even more powerful. By…
This ignores that reinforcement learning radically changes the training objective
Re: History LLMs: Models trained exclusively on pre-1913 texts
#403Earlier quoted context omitted.
>> Sometimes they hallucinate. For someone speaking as you knew everything, you appear to know very little. Every LLM completion is a "hallucination", some of them just happen to be factually correct.
I can say "I don't know" in response to a question. Can an LLM?
Yes, frequently.
Most modern post training setups encourage this.
It isn't 2023 anymore.
Re: History LLMs: Models trained exclusively on pre-1913 texts
#404Earlier quoted context omitted.
Almost no scifi has predicted world changing "qualitative" changes. As an example, portable phones have been predicted. Portable smartphones that are more like chat and payment terminals with a voice function no one uses any more ... not so much.
The Machine Stops ( https://www.cs.ucdavis.edu/~koehl/Teaching/ECS188/PDF_files/... ), a 1909 short story, predicted Zoom fatigue, notification fatigue, the isolating effect of widespread digital communication, atrophying of real-world skills as people become dependent on technology, blind acceptance of whatever the computer says, online lectures and remote learning, useless automated customer support systems, and ov…
There is even more to it than that. Also remember this is 1909. I think this classifies as a deeply mysterious story. It's almost inconceivable for that time period.
-people a depicted as grey aliens (no teeth, large eyes, no hair). Lesson the Greys are a future version of us.
The air is poisoned and ruined cities. People live in underground bunkers...1909...nuclear war was unimaginable then. This was still the age of steam ships and coal power trains. Even respirators would have been low on the public imagination.
The air ships with metal blinds sound more like UFOs than blimps.
The white worms.
People are the blood cells of the machine which runs on their thoughts social media data harvesting of ai.
China invaded Australia. This story was 8 years or so after the Boxer Rebellion so that would have sounded like say Iraq invading the USA in the context of its time.
The story suggests this is a cyclical process of a bifurcated human race.
The blimp crashing into the steel evokes 9/11, 91+1 years later...
The constellation orion.
Etc etc.
There is a central commitee
Re: History LLMs: Models trained exclusively on pre-1913 texts
#405It would be interesting to see how hard it would be to walk these models towards general relativity and quantum mechanics. Einstein’s paper “On the Electrodynamics of Moving Bodies” with special relativity was published in 1905. His work on general relativity was published 10 years later in 1915. The earliest knowledge cuttoff of these models is 1913, in between the relativity papers. The knowledge cutoffs are also r…
the issue is there is very little text before the internet, so not enough historical tokens to train a really big model
There's quite a lot of text in pre-Internet daily newspapers, of which there were once thousands worldwide.
When you're looking at e.g. the 19th century, a huge number are preserved somewhere in some library, but the vast majority don't seem to be digitized yet, given the tremendous amount of work.
Given how much higher-quality newspaper content tends to be compared to the average internet forum thread, there actually might be quite a decent amount of text. Obviously still nothing compared to the internet, but still vastly larger than just from published books. After all, print newspapers were essentially the internet of their day. Oh, and don't forget pamphlets in the 18th century.
Re: History LLMs: Models trained exclusively on pre-1913 texts
#406Earlier quoted context omitted.
No, that's insane. Computing is a dynamic process. A static string is not a computer.
It may be insane, but it's also true. https://en.wikipedia.org/wiki/Rule_110
Notice that the Rule 110 string picks out a machine, it is not itself the machine. To get computation out of it, you have to actually do computational work, i.e. compare current state, perform operations to generate subsequent state. This doesn't just automatically happen in some non-physical realm once the string is put to paper.
Re: History LLMs: Models trained exclusively on pre-1913 texts
#407Earlier quoted context omitted.
Although... Self preservation is the first law of nature. If you release the model someone will basically say you endorse those views and you risk your funding being cut. You created Pandora's box and now you're afraid of opening it.
They could add a text box where users have to explicitly type the following words before it lets them interact in any way with the model: "I understand this model was created with old texts so any racial or sexual statements are a byproduct of their time an do not represent in any way the views of the researchers". That should be more than enough to clear any chance of misunderstanding.
I would claim the public can easily handle something like this, but the media wouldn't be able to resist.
I could easily see a hit piece making its rounds on left leaning media about the AI that re-animates the problematic ideas of the past. "Just look at what it said to my child, ""!" Rolling stones would probably have a front page piece on it, titled "AI resurrecting racism and misogyny". There would easily be enough there to attract death threats to the developers, if it made its rounds on twitter.
"Platforming ideas" would be the issue that people would have.
Re: History LLMs: Models trained exclusively on pre-1913 texts
#408Earlier quoted context omitted.
From your NBC piece > About half of the Gen Z adults who identify as LGBTQ identify as bisexual, So that means ~15% of those surveyed are not attracted to the opposite sex (there’s more nuance to this statement but I imagine this needs to stay boilerplate), more or less, which is a big distinction. That’s hardly alarming and definitely not a major shift. We have also seen many cultures throughout history ebb and flow…
I'll get back to what you said, but first let me ask you something if you would. Imagine Gender Queer was made into a movie that remained 100% faithful to the source content. What do you think it would be rated? To me it seems obvious that it would, at the absolute bare minimum, be R rated. And of course screening R-rated films at a school is prohibited without explicit parental permission. Imagine books were given a…
[deleted]
Re: History LLMs: Models trained exclusively on pre-1913 texts
#409i feel like this would be super useful for unique marketing copy and writing. The responses sound so sophisticated like I read it in my grandfather's tone and cadence.
Re: History LLMs: Models trained exclusively on pre-1913 texts
#410This is a brilliant idea. We have lots of erroneous ideas about the views and thoughts people had in the past. This will show we are still, actually, largely similar. Hopefully more and more of these historical LLMs appear.