Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a GPT-4 script to check Wikipedia for the first unused acronym

gwern.net

Post reply on HN
Print view View original

71–80 of 118 posts

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#71

Post by jondwillis »

cypherpunks01 wrote: post #65
dojitza1 wrote: post #43

An interesting solution to the blind spot error (taken directly from Jeremy Howard's amazing guide to language models - https://www.youtube.com/watch?v=jkrNMKz9pWU ) is to erase the chat history and try again. Once GPT has made an error (or as the author of this article says, the early layers have irreversibly pruned some important data), it will very often start to be even more wrong.

When this happens, I'll usually say something along the lines of: "This isn't working and I'd like to start this again with a new ChatGPT conversation. Can you suggest a new improved prompt to complete this task, that takes into account everything we've learned so far?" It has given me good prompt suggestions that can immediately get a script working on the first try, after a frustrating series of blind spot bugs.

I’ll give this a try when it undoubtedly happens to me later today while debugging something ;)

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#72

Post by jondwillis »

dtx1 wrote: post #45
telotortium wrote: post #2

The question answered by this page is "what is the first unused 3-letter acronym in English Wikipedia?" - it's CQK for the record. However, the meat of the page is how to effectively use GPT-4 to write this script, hence why I've submitted it under this title (go to https://gwern.net/tla#effective-gpt-4-programming ). Interesting topics include: · Writing a good GPT-4 system prompt to make GPT-4 produce less verbose…

> If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script What an absolutely based take by GPT-4

Really reflecting the bias of ML/AI practitioners to reach for a slow and footgunny language…

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#73

Post by teaearlgraycold »

kuschku wrote: post #47
teaearlgraycold wrote: post #35

Earlier quoted context omitted.

I learned how expensive hashmaps and hashsets are through Wikipedia dumps. I did some analysis of the most linked-to pages. Countries were among the highest. Hash sets for holding outgoing edges in the link graph ended up causing my program to exceed my laptop’s memory. Plain old lists (Python) were fine, though. And given there aren’t a crazy number of links per page using lists is fine performance wise.

How'd the hashset exceed your laptop memory, if the whole dump is just 22GB? You should be able to fit the entire dataset in RAM.

I think my laptop had 8GB at the time

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#74

Post by teaearlgraycold »

angrais wrote: post #38
teaearlgraycold wrote: post #35

Earlier quoted context omitted.

I learned how expensive hashmaps and hashsets are through Wikipedia dumps. I did some analysis of the most linked-to pages. Countries were among the highest. Hash sets for holding outgoing edges in the link graph ended up causing my program to exceed my laptop’s memory. Plain old lists (Python) were fine, though. And given there aren’t a crazy number of links per page using lists is fine performance wise.

Why did lists require less memory? Was it because you only held a subset of keys in the lists?

Lists in Python have a integer for their size and a pointer for each element. Sets presumably have some number of buckets that are used to put pointers in, but many more buckets are allocated than get used in small sets.

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#75

Post by vidarh »

somewhereoutth wrote: post #48
vidarh wrote: post #33

Earlier quoted context omitted.

We don't know what "think like a person" entails, so we don't know how different human thought processes are to predicting what goes next, and whether those differences are meaningful when making a comparison. Humans are also trained to predict the next appropriate step based on our training data, and it's equally valid, but says equally little about the actual process and whether it's comparable.

We do know that in terms of external behavior and internal structure (as far as we can ascertain it), humans and LLMs have only an passing resemblance in a few characteristics, if at all. Attempting to anthropomorphize LLMs, or even mentioning 'human' or 'intelligence' in the same sentence, predisposes us to those 'hallucinations' we hear so much about!

We really don't. We have some surface level idea about differences, but we can't tell how that does affect the actual learning and behaviours.

More importantly we have nothing to tell us whether it matters, or if it will turn out any number of sufficiently advanced architectures will inevitably approximate similar behaviours when exposed to the same training data.

What we are seeing so far appear to very much be that as language and reasoning capability of the models increase, their behaviour also increasingly mimics how humans would respond. Which makes sense as that is what they are being trained to.

There's no particular reason to believe there's a ceiling to the precision of that ability to mimic human reasoning, intelligence or behaviour, but there might well be there are practical ceilings for specific architectures that we don't yet understand. Or it could just be a question of efficiency.

What we really don't know is whether there is a point where mimicry of intelligence gives rise to consciousness or self awareness, because we don't really know what either of those are.

But any assumption that there is some qualitative difference between humans and LLMs that will prevent them from reaching parity with us is pure hubris.

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#76

Post by throw555chip »

k2enemy wrote: post #68

Pedantic, but gwern is looking for initialisms, not acronyms. Acronyms are pronounced as a word. https://www.merriam-webster.com/grammar/whats-an-acronym

> looking for initialisms, not acronyms

Imprecise wording, initialisms are a case of acronyms, it's not either or.

https://wwwnc.cdc.gov/eid/page/abbreviations-acronyms-initia...

"an initialism is an acronym that is pronounced as individual letters"

https://www.writersdigest.com/write-better-fiction/abbreviat...

"As such, acronyms are initialisms."

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#77

Post by jondwillis »

Aerbil313 wrote: post #51
worldsayshi wrote: post #49

Earlier quoted context omitted.

Yeah once ChatGPT shows up as an entity in the training data it will sort of inescapably start to build a self image.

Wait, this can actually have consequences! Think about all the SEO articles about ChatGPT hallucinating… At some point it will start to “think” that it should hallucinate and give nonsensical answers often, as it is ChatGPT.

I wouldn’t draw that conclusion yet, but I suppose it is possible.

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#78

Post by jondwillis »

anotherpaulg wrote: post #18

I asked aider to use the new GPT-4 Turbo to: Write a bash script to check Wikipedia for all acronyms of length 1-6 to find those which aren't already in use. It did a fairly smooth job of it. See the chat transcript [0] and resulting bash script [1] with git commit history [2]. It fell into the initial trap of blocking while pre-generating long acronyms upfront. But a couple gentle requests got it to iteratively stre…

https://cursor.sh has some recently added functionality for applying code changing to files throughout your code base. I have yet to try it because I am using their a la carte (bring your own keys) option.

Re: Writing a GPT-4 script to check Wikipedia for the first unused acronym

#80

Post by gameman144 »

throw555chip wrote: post #76
k2enemy wrote: post #68

Pedantic, but gwern is looking for initialisms, not acronyms. Acronyms are pronounced as a word. https://www.merriam-webster.com/grammar/whats-an-acronym

> looking for initialisms, not acronyms Imprecise wording, initialisms are a case of acronyms, it's not either or. https://wwwnc.cdc.gov/eid/page/abbreviations-acronyms-initia... "an initialism is an acronym that is pronounced as individual letters" https://www.writersdigest.com/write-better-fiction/abbreviat... "As such, acronyms are initialisms."

Wait, am I crazy or are these two articles saying the exact opposite thing about which class is the parent one?

The CDC one seems to say that initialisms are a class of acronym, but the Writers Digest one says acronyms are a class of initialism.

Post reply on HN