Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TimeCapsuleLLM: LLM trained only on data from 1800-1875

github.com

Post reply on HN
Print view View original

321–330 of 334 posts

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#321

Post by asveikau »

canjobear wrote: post #99
auraham wrote: post #88

Earlier quoted context omitted.

Can you elaborate on this? After skimming the README, I understand that "Who art Henry" is the prompt. What should be the correct 19th century prompt?

"Who art Henry?" was never grammatical English. "Art" was the second person singular present form of "to be" and it was already archaic by the 17th century. "Who is Henry?" would be fine.

In some languages you can put a second person conjugation next to a noun that might otherwise use third person verbs, and it serves as implying that you are that noun. I'm not sure if older forms of English had that construct. I think many Indo-European languages do.

The part of the lord's prayer that says "our father who art in heaven" is kinda like this - father is linked to a second person conjugation. You could remove some words and make it into "father art in heaven", which you claim is ungrammatical. I'm skeptical that it was.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#322

Post by mannykannot »

TeMPOraL wrote: post #309
mannykannot wrote: post #299

Earlier quoted context omitted.

Sure - and climbing a mountain is just putting one foot down higher than it was before and repeating, once you abstract away all the hard parts.

It is. If you're at the mountain, on the right trail, and have the right clothing and equipment for the task. That's why those tiny steps of scientific and technological progress aren't made by just any randos - they're made by people who happen to be at the right place and time, and equipped correctly to be able to take the step. The important corollary to this is that you can't generally predict this ahead of time.…

You are describing hiking in the mountains, which doesn’t generalize to mountaineering and rock-climbing when it gets difficult, and the difficulties this view is abstracting away are real.

Your second and third paragraphs are entirely consistent with the original point I was trying to make, which was not that it took Einstein specifically to come up with relativity, but that it took someone with uncommon skills, as evidenced by the fact that it blindsided even a good many of the people who were qualified to be contenders for being the one to figure it out first. It does not amount to proof, but one does not expect people who are closing in on the solution to be blindsided by it.

I am well aware of the problems with “great man” hagiography, but dismissing individual contributions, which is what the person I was replying to seemed to be doing, is a distortion in its own way.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#323

Post by canjobear »

asveikau wrote: post #321
canjobear wrote: post #99

Earlier quoted context omitted.

"Who art Henry?" was never grammatical English. "Art" was the second person singular present form of "to be" and it was already archaic by the 17th century. "Who is Henry?" would be fine.

In some languages you can put a second person conjugation next to a noun that might otherwise use third person verbs, and it serves as implying that you are that noun. I'm not sure if older forms of English had that construct. I think many Indo-European languages do. The part of the lord's prayer that says "our father who art in heaven" is kinda like this - father is linked to a second person conjugation. You could r…

“who art in heaven” is a grammatical relative clause because the subject of the verb is the relative pronoun “who” which is second person in that context. You can still get this kind of thing in modern English, for example “I, who am a farmer, will be happy” is grammatical because the relative pronoun “who” is first person there. That doesn’t mean it would be grammatical to say “*A farmer am happy” and it wouldn’t have worked with art either.

Conceivably it’s grammatical if Henry is vocative and the pronoun is dropped colloquially, like “Who art [thou], O Henry?” but it’s a stretch.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#324

Post by geocar »

andai wrote: post #300
geocar wrote: post #238

Earlier quoted context omitted.

The problem is the subjunctive mood of the word "art". "Art thou" should be translated into modern English as "are you to be", and so works better with things (what are you going to be), or people who are alive, and have a future (who are you going to be?). Those are probably the contexts you are thinking of.

Wherefore are you going to be Romeo?

Wherefore is "For what reason or why". Juliet is not asking where Romeo is physically, but wondering why does he need to be a Montague.

So yes you can interpret it as "for what reason or why are you going to be" (this thing she will now explain).

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#325

Post by sigmoid10 »

lopuhin wrote: post #268
sigmoid10 wrote: post #256

Earlier quoted context omitted.

These probability shifts would only account for the final output layer (which may also have some shift), but I expect the largest shift to be in the activations in the intermediate latent space. There are a bunch of papers out there that try to get some offset vector using PCA or similar to tune certain model behaviours like vulgarity or friendlyness. You don't even need much data for this as long as your examples ca…

On whether this accounts only the final output layer -- once the first token is generated (i.e. selected according to the modified sampling procedure), and assuming a different token is selected compared to standard sampling, then all layers of the model would be affected during generation of subsequent tokens.

This way it wouldn't be much better than instructing the model to elicit a particular behaviour using the system prompt. Limiting tokens to a subset of outputs is already common (and mathematically equivalent to a large shift in the output vector), e.g. for structured outputs, but it doesn't change the actual world representation inside the model. It would also be very sensitive to your input prompt to do it this way.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#326

Post by andai »

efreak wrote: post #317
andai wrote: post #300

Earlier quoted context omitted.

Wherefore are you going to be Romeo?

Wherefore is closer to why , or (as taught in literature classes) for what reason . Wherefore is a question, therefore is an answer.

Yes, I am confused about the meaning of "art" in this context.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#327

Post by asveikau »

canjobear wrote: post #323
asveikau wrote: post #321

Earlier quoted context omitted.

In some languages you can put a second person conjugation next to a noun that might otherwise use third person verbs, and it serves as implying that you are that noun. I'm not sure if older forms of English had that construct. I think many Indo-European languages do. The part of the lord's prayer that says "our father who art in heaven" is kinda like this - father is linked to a second person conjugation. You could r…

“who art in heaven” is a grammatical relative clause because the subject of the verb is the relative pronoun “who” which is second person in that context. You can still get this kind of thing in modern English, for example “I, who am a farmer, will be happy” is grammatical because the relative pronoun “who” is first person there. That doesn’t mean it would be grammatical to say “*A farmer am happy” and it wouldn’t ha…

I think the further back you go in Indo-European grammar, the more common the thing you are describing becomes. For me it's less of a question of if English did this, and more like how far back you need to go.

Today, even ignoring the dated conjugation, "who art in heaven" or "who are in heaven", does not make sense. We would switch it into the third person.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#328

Post by PaulDavisThe1st »

famouswaffles wrote: post #316
PaulDavisThe1st wrote: post #315

Earlier quoted context omitted.

> You start with “there is no representation of a concept,” but then concede “features that may correspond to human concepts.” If those features are (a) reliably present across contexts, (b) abstract over surface tokens, and (c) participate causally in producing downstream behavior, then that is a representation in the sense most people mean in cognitive science. One of the most frustrating things about these sorts o…

>Nobody would say that most of what the brain does involves manipulating concepts. When you run from danger, when you reach up grab something from a shelf, when you do almost anything except actual conscious reasoning, most of the accounts of how that behavior arises from brain activity does not involve manipulating concepts. This framing assumes "concept manipulation" requires conscious, deliberate reasoning. But th…

HN comment threads are really not the right place for discussions like this.

> Here's my core objection: you're defining "manipulating concepts" as "whatever special thing happens during conscious human reasoning that feels different from 'pattern matching.'" But this is circular and unfalsifiable. How would we ever know if an LLM (or another human, for that matter) is doing this "special thing"? You've defined it purely in terms of subjective experience rather than functional or mechanistic criteria.

I think your core objection is well aligned to my own POV. I am not claiming that the subjective experience is the critical element here, but I am claiming that whatever is going on when we have the subjective experience of "reasoning" is likely to be different (or more specifically, more usefully described in different ways) than what is happening in LLMs and our minds when doing something else.

How would we ever know? Well the obvious answer is more research into what is happening in human brains when we reason and comparing that to brain behavior at other times.

I don't think it's likely to be productive to continue this exchange on HN, but if you would like to continue, my email address is in my profile.

Re: TimeCapsuleLLM: LLM trained only on data from 1800-1875

#330

Post by djwide »

metalliqaz wrote: post #284
drdeca wrote: post #224

Earlier quoted context omitted.

And? Do the arguments differ for LLM vs the other models? I guess the arguments sometimes mention languages. But I feel like the core of the arguments are pretty much the same regardless?

The discussion is about training an LLM on old text and then asking it about new concepts.

AlphaGo was trained on old games and then presented with a game it had never seen before. It came up with a move that a human would not have played.
Post reply on HN