Show HN: AskHN

#1

Post by kvh » Wed, Feb 22, 2023, 4:09 PM UTC

Re: Show HN: AskHN

#2

Post by LeoPanthera » Wed, Feb 22, 2023, 4:34 PM UTC

I'm a little surprised that Hacker News comments weren't already in the GPT-3 training set. I just assumed that OpenAI had vacuumed up most of the web already.

Re: Show HN: AskHN

#3

Post by retube » Wed, Feb 22, 2023, 4:38 PM UTC

LeoPanthera wrote: post #2

I'm a little surprised that Hacker News comments weren't already in the GPT-3 training set. I just assumed that OpenAI had vacuumed up most of the web already.

I am guessing they already were? But this is 100% pure, concentrated HN not contaminated with nonsense from the rest of the web :)

Re: Show HN: AskHN

#4

Post by MuffinFlavored » Wed, Feb 22, 2023, 4:43 PM UTC

> I trained on a corpus of over 6.5 million Hacker News comments

How long did it take to scrape them and train the "corpus" on this content?

Re: Show HN: AskHN

#5

Post by MuffinFlavored » Wed, Feb 22, 2023, 4:43 PM UTC

retube wrote: post #3

LeoPanthera wrote: post #2

I'm a little surprised that Hacker News comments weren't already in the GPT-3 training set. I just assumed that OpenAI had vacuumed up most of the web already.

I am guessing they already were? But this is 100% pure, concentrated HN not contaminated with nonsense from the rest of the web :)

Is it exclusively HN comments and nothing else? How does a model like that know how to speak English (noun/verb and all that) if you are starting from scratch and feeding it nothing but HN comments?

Re: Show HN: AskHN

#6

Post by tta » Wed, Feb 22, 2023, 4:45 PM UTC

MuffinFlavored wrote: post #4

> I trained on a corpus of over 6.5 million Hacker News comments How long did it take to scrape them and train the "corpus" on this content?

https://console.cloud.google.com/marketplace/details/y-combi...

Re: Show HN: AskHN

#7

Post by nkozyra » Wed, Feb 22, 2023, 4:45 PM UTC

retube wrote: post #3

LeoPanthera wrote: post #2

I'm a little surprised that Hacker News comments weren't already in the GPT-3 training set. I just assumed that OpenAI had vacuumed up most of the web already.

I am guessing they already were? But this is 100% pure, concentrated HN not contaminated with nonsense from the rest of the web :)

I have to assume that targeted/curated LLM training sets will have a tendency to be less accurate than very general, just by the very nature of how they work.

(edited for clarity)

Re: Show HN: AskHN

#8

Post by neoromantique » Wed, Feb 22, 2023, 4:45 PM UTC

MuffinFlavored wrote: post #5

retube wrote: post #3

Earlier quoted context omitted.

I am guessing they already were? But this is 100% pure, concentrated HN not contaminated with nonsense from the rest of the web :)

Is it exclusively HN comments and nothing else? How does a model like that know how to speak English (noun/verb and all that) if you are starting from scratch and feeding it nothing but HN comments?

I'm sorry to be THAT GUY, but it is addressed in the article :)

>GPT embeddings

To index these stories, I loaded up to 2000 tokens worth of comment text (ordered by score, max 2000 characters per comment) and the title of the article for each story and sent them to OpenAI's embedding endpoint, using the standard text-embedding-ada-002 model, this endpoint accepts bulk uploads and is fast but all 160k+ documents still took over two hours to create embeddings. Total cost for this part was around $70.

Re: Show HN: AskHN

#9

Post by egypturnash » Wed, Feb 22, 2023, 4:46 PM UTC

Is there a way to opt out of one's comments being used for this?

Re: Show HN: AskHN

#10

Post by nkozyra » Wed, Feb 22, 2023, 4:46 PM UTC

MuffinFlavored wrote: post #5

retube wrote: post #3

Earlier quoted context omitted.

I am guessing they already were? But this is 100% pure, concentrated HN not contaminated with nonsense from the rest of the web :)

Is it exclusively HN comments and nothing else? How does a model like that know how to speak English (noun/verb and all that) if you are starting from scratch and feeding it nothing but HN comments?

> How does a model like that know how to speak English

Mimicry.