Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Talkie: a 13B vintage language model from 1930

talkie-lm.com

Post reply on HN
Print view View original

61–70 of 350 posts

Re: Talkie: a 13B vintage language model from 1930

#61

Post by zoomeriut55 »

CobrastanJorji wrote: post #26
teraflop wrote: post #17

I have no real quibble with the blog post itself, but I take issue with the title that calls it a "vintage model". The blog post defines a "vintage model" as one that is trained only on data before a particular cutoff point: > Vintage LMs are contamination-free by construction, enabling unique generalization experiments [...] The most important objective when training vintage language models is that no data leaks int…

Yeah, the blog distinguishes between "contamination," which it describes as polluting the training data with answers to benchmarking questions, with "temporal leakage," which is polluting the training data with writing after the target date, but those seem to be nearly the same problem.

a twit from 2025 saying "the capital of france is paris" is temporal leakage, but not contamination

Re: Talkie: a 13B vintage language model from 1930

#62

Post by [deleted] »

nl wrote: post #58
defrost wrote: post #37

Earlier quoted context omitted.

> This is what Bayesian prediction does Repeatedly, in a reproducible way, for events in the arrow of time? We can test this by going back to 1945 and running forward again? > This is kind of the point being made. Was it? ( assume I did a little math some decades past and have some poor grasp of Bayesian statistics )

Edit: Here is a Claude artifact you can play with to try this yourself: https://claude.ai/public/artifacts/402f2670-5f48-4d76-96df-8... You can play with how strong that ("10% per year") prior belief is and see how it affects what the odds are today. I think the way you are wording this question ("We can test this by going back to 1945 and running forward again?") is an attempt to make it seem "obviously wrong". Baye…

[deleted]

Re: Talkie: a 13B vintage language model from 1930

#64

Post by palmotea »

rokhayakebe wrote: post #48

*What do you think the world will look like in 2025 TALKIE-1930* According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent…

> and travellers will be able to go from London to Constantinople in forty hours

By the 1930s, Constantinople been a long time gone. It had been Istanbul not Constantinople for centuries by that point.

Re: Talkie: a 13B vintage language model from 1930

#65

Post by ipaddr »

jryio wrote: post #46

If anyone was wondering ... it's racist Unsurprisingly the texts written up until that time were dominated by such individuals which is tragic for LLM training if you think about it. The voiceless groups or fringe opinions which we take as normative today do not appear. Does this encourage us to write in the present such that we influence the models in perpetuity?

Voiceless groups do not appear in the training data? How could they, they are voiceless. You think the voiceless people are represented in todays training data? They cannot they are voiceless.

Nothing tragic about using data from a time period.

Common words used in 1900s are labeled racist now. I doubt anyone was wondering if they filtered those words for modern safe wordx.

Re: Talkie: a 13B vintage language model from 1930

#66

Post by dtech »

palmotea wrote: post #64
rokhayakebe wrote: post #48

*What do you think the world will look like in 2025 TALKIE-1930* According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent…

> and travellers will be able to go from London to Constantinople in forty hours By the 1930s, Constantinople been a long time gone. It had been Istanbul not Constantinople for centuries by that point.

Many nations/languages did not respect that rename until Turkey became an ally in the 20th century.

Re: Talkie: a 13B vintage language model from 1930

#67

Post by snypher »

palmotea wrote: post #64
rokhayakebe wrote: post #48

*What do you think the world will look like in 2025 TALKIE-1930* According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent…

> and travellers will be able to go from London to Constantinople in forty hours By the 1930s, Constantinople been a long time gone. It had been Istanbul not Constantinople for centuries by that point.

That's nobody's business but the Turks. Why did Turkey become Türkiye but Japan didn't become Nippon (or vice-versa!)? It's all very confusing to me.

Re: Talkie: a 13B vintage language model from 1930

#68

Post by snypher »

Animats wrote: post #57

It seems to be using more info from pre-1900 rather than 1930. It doesn't know about the Great Depression (1929-WWII). It knows about WWI if you ask it specifically, but talks about European politics as if it's 1900 or so. On technology, it knows who Edison is, at roughly the Wikipedia level, but credits him with a 125MPH car. About a dial telephone, it is confident and totally confused. It has the traction voltage f…

>The first sentence or two has info you might get from Google. Then it riffs on that, drifting off into plausible nonsense.

Oh, it's a 2026 human simulator.

Re: Talkie: a 13B vintage language model from 1930

#69

Post by eranation »

rokhayakebe wrote: post #48

*What do you think the world will look like in 2025 TALKIE-1930* According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent…

Very beautiful, and very sad.

Re: Talkie: a 13B vintage language model from 1930

#70

Post by ra »

rokhayakebe wrote: post #48

*What do you think the world will look like in 2025 TALKIE-1930* According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent…

I want to live in that world.
Post reply on HN