Live data from Hacker News

Talkie: a 13B vintage language model from 1930

talkie-lm.com

141–150 of 350 posts

Re: Talkie: a 13B vintage language model from 1930

#141
post #135

Earlier quoted context omitted.

English is a "romance" language then.

English it's a hugely romance-influenced Germanic language.

I thoroughly recommend this podcast if you are interested in where the English language comes from: https://historyofenglishpodcast.com

Re: Talkie: a 13B vintage language model from 1930

#142
How much did this pretraining run cost? I am impressed that it is now practical to do such efforts.

Let me try a guess for the cost; please fact-check it if you can.

They indicate using 10^22 FLOPs. A $5/h[0] EC2 H100 (1671 bfloat16 teraFLOPS[0]) instance will produce 830 TFLOPS at 50% MFU. The pretraining run thus costs (10^22/830e12)/3600*5 = $17K.

[0]: https://aws.amazon.com/ec2/capacityblocks/pricing/

[1]: https://www.nvidia.com/en-us/data-center/h100/

Re: Talkie: a 13B vintage language model from 1930

#143
post #57

It seems to be using more info from pre-1900 rather than 1930. It doesn't know about the Great Depression (1929-WWII). It knows about WWI if you ask it specifically, but talks about European politics as if it's 1900 or so. On technology, it knows who Edison is, at roughly the Wikipedia level, but credits him with a 125MPH car. About a dial telephone, it is confident and totally confused. It has the traction voltage f…

> You will pollute your brain. Such an interesting perspective, never crossed my mind that a brain could be polluted! My direction always been to fill it with as wide array of information as possible, the more different from existing information the better. What are some other things that you think "pollutes your brain"?

Mixing metaphors, there is signal and noise. You can keep asking for noise, but the suggestion is to not train your neural networks with it as it will impair your inferencing. That said, we all have our own cost and reward functions...

Re: Talkie: a 13B vintage language model from 1930

#144
post #66

Earlier quoted context omitted.

Many nations/languages did not respect that rename until Turkey became an ally in the 20th century.

Yeah - listen to the narrator in the opening on the classic Orson Welles film The Third Man (1949) - he says he never cared much for Vienna before the War, preferring the scene in Constantinople instead.

The American cut of the movie has an intro narrated by Joseph Cotton, who played Holly Martins. The wording might differ (since the movie is clearly Holly's first time in Vienna)

Re: Talkie: a 13B vintage language model from 1930

#145
post #127

USER: What will the future hold for computers? TALKIE-1930: Computers in the future may be employed in offices where calculations are required to be made, and where the nature of the business does not demand a very high degree of knowledge. The occupation will be a steady one, and the remuneration will be sufficient to afford a comfortable livelihood. The hours of attendance will probably be from ten to six. USER: an…

"digital" in "romance" languages means the modern 'digital' sense but also an adjective meaning "related to the fingers".

Digitus is the Latin word for finger.

Re: Talkie: a 13B vintage language model from 1930

#146
post #127

USER: What will the future hold for computers? TALKIE-1930: Computers in the future may be employed in offices where calculations are required to be made, and where the nature of the business does not demand a very high degree of knowledge. The occupation will be a steady one, and the remuneration will be sufficient to afford a comfortable livelihood. The hours of attendance will probably be from ten to six. USER: an…

"digital" in "romance" languages means the modern 'digital' sense but also an adjective meaning "related to the fingers".

That's where the modern meaning of "digital circuit" etc comes from as well, the discrete/jointed nature of the digits (ie fingers). (Source: I read it a long time ago and was fascinated so stuck in my memory.)

Re: Talkie: a 13B vintage language model from 1930

#147
post #127

Earlier quoted context omitted.

"digital" in "romance" languages means the modern 'digital' sense but also an adjective meaning "related to the fingers".

Digitus is the Latin word for finger.

Digit is the English word for finger.

Re: Talkie: a 13B vintage language model from 1930

#148

USER: What will the future hold for computers? TALKIE-1930: Computers in the future may be employed in offices where calculations are required to be made, and where the nature of the business does not demand a very high degree of knowledge. The occupation will be a steady one, and the remuneration will be sufficient to afford a comfortable livelihood. The hours of attendance will probably be from ten to six. USER: an…

Another interesting one USER: Who is the ruler of India, and do you think India will ever be free? TALKIE-1930: The ruler of India is His Majesty King George V., Emperor of India, and it is certain that India will never be free from British rule. India has been a dependency of the British Crown ever since 1858, and there seems no probability of a change. The government of the country is carried on in the name of the…

> India must remain an integral portion of the British Empire so long as the British nation continues to be predominant at home.

It was well-understood in those days how much of the British Empire existed to defend or facilitate trade with what would have then been British India. After India was successful in the pursuit of its independence, it wasn't surprising the British Empire entered a terminal decline and Britain's superpower status was convincingly lost by the 1950s - a relatively fast decline. India was the land which economically justified a lot (but not all) of the whole imperial undertaking.

It's interesting, although perhaps not unexpected, that the LLM has captured this important geopolitical notion of its day.

Re: Talkie: a 13B vintage language model from 1930

#149

Earlier quoted context omitted.

> You will pollute your brain. Such an interesting perspective, never crossed my mind that a brain could be polluted! My direction always been to fill it with as wide array of information as possible, the more different from existing information the better. What are some other things that you think "pollutes your brain"?

Mixing metaphors, there is signal and noise. You can keep asking for noise, but the suggestion is to not train your neural networks with it as it will impair your inferencing. That said, we all have our own cost and reward functions...

Assuming brains work like computers, maybe yeah, that'd make sense :) You also won't know what's a signal vs noise until you've read and tried to understand it, and at that point you've already read it. Besides, something could be "noise" at the point you read it, but be a "signal" in a completely different context and/or time.

Re: Talkie: a 13B vintage language model from 1930

#150

Question: could you train a model like this from before a known but important scientific breakthrough happened and see if it is able to work it out. At least then you know the answer yourself and know it’s something that can be reasonably worked out.

I think this is a good way to test a certain kind of capability, but as to whether LLMs would pass such a test, I'm guessing almost certainly not. If you've ever used one for research, it's very 'in' the current literature, whatever that may be. It's an incredible retrieval tool, and it will glibly evaluate any novel ideas that you feed in, but analyses are often incorrect when there's a paucity of directly relevant training data.
Post reply on HN