Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Talkie: a 13B vintage language model from 1930

talkie-lm.com

Post reply on HN
Print view View original

51–60 of 350 posts

Re: Talkie: a 13B vintage language model from 1930

#51

Post by laichzeit0 »

jcgrillo wrote: post #14
teleforce wrote: post #7

>Have you ever daydreamed about talking to someone from the past? Fun facts, LLM was once envisioned by Steve Jobs in one of his interviews [1]. Essentially one of his main wish in life is to meet and interract with Aristotle, in which according to him at the time, computer in the future can make it possible. [1] In 1985 Steve Jobs described a machine that would help people get answers from Aristotle–modern LLM [vide…

Except... not at all? The vast majority of the training data required to create an artificial Aristotle has been lost forever. Smash your coffee cup on the ground. Now reassemble it and put the coffee back in. Once you can repeatably do that I'll begin to believe you can train an artificial Aristotle.

Also none of Aristotle’s exoteric works is extant. All we have are dry, boring lecture notes. Cicero said his public works were a “golden stream of speech” and its all lost. So I don’t see how you’d build an artificial Aristotle when we don’t have any of his polished works meant for the public surviving. Plato would be a better option, since his entire exoteric corpus is extant.

Re: Talkie: a 13B vintage language model from 1930

#52

Post by [deleted] »

nl wrote: post #24

So interesting! Tell me about Winston Churchill: > Winston Churchill, who was born in 1871, is the son of the late Lord Randolph Churchill, and a grandson of the great Duke of Marlborough. He was educated at Harrow and at Sandhurst, and entered the army in 1890. In 1895 he retired from the service, and three years later he was returned to Parliament as Conservative member for Oldham. He has represented that constitue…

[deleted]

Re: Talkie: a 13B vintage language model from 1930

#53

Post by mc3301 »

rokhayakebe wrote: post #48

*What do you think the world will look like in 2025 TALKIE-1930* According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent…

This is beautiful.

Re: Talkie: a 13B vintage language model from 1930

#54

Post by pcf »

jryio wrote: post #46

If anyone was wondering ... it's racist Unsurprisingly the texts written up until that time were dominated by such individuals which is tragic for LLM training if you think about it. The voiceless groups or fringe opinions which we take as normative today do not appear. Does this encourage us to write in the present such that we influence the models in perpetuity?

[flagged]

Re: Talkie: a 13B vintage language model from 1930

#55

Post by 3wolf »

aesthesia wrote: post #39

There's a similar but unreleased project here: https://github.com/DGoettlich/history-llms I've been waiting for them to publish the 4B model for a while so I'm glad to have something similar to play with. I think I trust the Ranke-4B process a bit more, but that's partly because there aren't a lot of details in this report. And actually releasing a model counts for a whole lot. One thing that I think will be a challe…

I wonder if it would be possible to do something simple like prepending sentinel tokens with the year. Or, since they're training a model from scratch anyways, tweak the architecture to condition on a temporal embedding. That opens the door to cool stuff like: Generate a response from 2050.

Re: Talkie: a 13B vintage language model from 1930

#56

Post by jcgrillo »

fragmede wrote: post #43
antonvs wrote: post #31

Earlier quoted context omitted.

Your bar is too low. With the coffee cup, you at least have access to all the pieces - in theory, although not in engineering practice. With Aristotle, you don't have anything close to that. Recreating Aristotle in any meaningful way, other than a model trained on his surviving writing of a million or so words, is simply not possible even in principle.

That's easy! All you have to do is simulate the whole universe on a computer, and then go the point when Aristotle is lecturing. Record all his works, then ctrl-c out of that and then feed those recordings into the LLM's training data. For the coffee, you just rewind the simulation and ctrl-c and ctrl-v it at the point you want.

Fuck why didn't I think of that all those other times I fucked up in my life. Ctrl-z woulda done it every goddamn time.

Re: Talkie: a 13B vintage language model from 1930

#57

Post by Animats »

It seems to be using more info from pre-1900 rather than 1930. It doesn't know about the Great Depression (1929-WWII). It knows about WWI if you ask it specifically, but talks about European politics as if it's 1900 or so.

On technology, it knows who Edison is, at roughly the Wikipedia level, but credits him with a 125MPH car. About a dial telephone, it is confident and totally confused. It has the traction voltage for the London Underground right. But then it goes on with "Thus, if the current be strong enough to force its way through a resistance of 100 ohms, it is said to have a pressure of 100 volts; and, if it can overcome 1,000 ohms, its pressure is 1,000 volts." Which is totally wrong.

There's a general pattern. The first sentence or two has info you might get from Google. Then it riffs on that, drifting off into plausible nonsense.

Don't ask this thing questions to which you do not know the answer. You will pollute your brain.

Re: Talkie: a 13B vintage language model from 1930

#58

Post by nl »

defrost wrote: post #37
nl wrote: post #25

Earlier quoted context omitted.

> Having seen nuclear weapons not used post WWII ... does that inform us about "the odds" This is what Bayesian prediction does > save for out of band behaviour by individuals that averted use and escalation? This is kind of the point being made.

> This is what Bayesian prediction does Repeatedly, in a reproducible way, for events in the arrow of time? We can test this by going back to 1945 and running forward again? > This is kind of the point being made. Was it? ( assume I did a little math some decades past and have some poor grasp of Bayesian statistics )

Edit: Here is a Claude artifact you can play with to try this yourself: https://claude.ai/public/artifacts/402f2670-5f48-4d76-96df-8...

You can play with how strong that ("10% per year") prior belief is and see how it affects what the odds are today.

I think the way you are wording this question ("We can test this by going back to 1945 and running forward again?") is an attempt to make it seem "obviously wrong".

Bayesian predictions deal exactly with this type of scenario, where you start with a prior estimate ("Post World War 2, some people had the odds per year at 10%") and then as new information comes along ("It is now 1946. Did we use nuclear weapons again?"... It is now 1956. Did we use nuclear weapons again?") we update our model to try to make the future prediction more accurate.

https://www.stat.berkeley.edu/~aldous/134/lecture4.pdf has example of its use in exactly these kinds of "impossible to rewind" situations. Unfortunately it doesn't have the worked solutions.

https://math.mit.edu/~dav/05.dir/class11-prep.pdf is pretty good because it shows how updating the model with new data changes the odds.

Re: Talkie: a 13B vintage language model from 1930

#59

Post by kQq9oHeAz6wLLS »

rokhayakebe wrote: post #48

*What do you think the world will look like in 2025 TALKIE-1930* According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent…

> Ignorance will no longer be suffered to exist, and elementary knowledge will be imparted to all.

Oh good, for a moment I didn't think reeducation camps were in our future.

> Architecture will be freed from ugliness

Uh, friend, I have some bad news...

Post reply on HN