How does this work? I thought there was some kind of limit in the size of the prompt and the API calls.
hey - some other commenters have answered your question better than I could.
Show HN: Turning books into chatbots with GPT-3
51–60 of 179 posts
Re: Show HN: Turning books into chatbots with GPT-3
#52Any references on how you do this, if you're willing to share?
I imagine it's using a variant of the semantic search answers pattern. I wrote a bit about that here: https://simonwillison.net/2023/Jan/13/semantic-search-answer...
Re: Show HN: Turning books into chatbots with GPT-3
#53Earlier quoted context omitted.
I asked ChatGPT: Me: Consider the story: "Justin is hungry. Justin eats dinner. Justin is not hungry." Is Justin hungry? ChatGPT: No, Justin is not hungry after eating dinner. I'm not sure that it's that big of a problem.
The example was to just illustrate the general problem. Think of ingesting a whole novel that takes place over a few years. The whole novel doesn't fit into GPT's context window (which is only a page or two of text). So you have to extract individual statements of fact and index over them (e.g. with semantic indexing, or many other techniques). It's tricky to deal with cases where the state of something changes many…
I would imagine that the "attention" phase of the LLMs could get longer over time as more resources are dedicated to them.
e.g. we are seeing the equivalent of movies that are 5 minutes long b/c they were hand animated. Once we move to computer animated movies, it becomes a lot easier to generate an entire film.
Re: Show HN: Turning books into chatbots with GPT-3
#54Earlier quoted context omitted.
The example was to just illustrate the general problem. Think of ingesting a whole novel that takes place over a few years. The whole novel doesn't fit into GPT's context window (which is only a page or two of text). So you have to extract individual statements of fact and index over them (e.g. with semantic indexing, or many other techniques). It's tricky to deal with cases where the state of something changes many…
I would imagine that the "attention" phase of the LLMs could get longer over time as more resources are dedicated to them. e.g. we are seeing the equivalent of movies that are 5 minutes long b/c they were hand animated. Once we move to computer animated movies, it becomes a lot easier to generate an entire film.
I agree they will get longer. ChatGPT (GPT3.5) is 2x larger than GPT3. 8192 tokens vs 4096.
The problem is that in the existing transformer architecture, the complexity of this is O(N^2). Making the context window 10x larger involves 100x more memory and compute.
We'll either need a new architecture that improves upon the basic transformer, or just wait for Moore's law to paper over the problem for the scales we care about.
In the short term, you can also use the basic transformer with a combination of other techniques to try to find the relevant things to put into the context window. For instance, I ask "Does Harry Potter know the foobaricus spell?" and then the external system does a more traditional search technique to find all sentences relevant to the query in the novels, maybe a few paragraph summary of each novel, etc, then feeds that ~1 page worth of data to GPT to then answer the question.
Re: Show HN: Turning books into chatbots with GPT-3
#55This is one application of AI that I absolutely like. Imagine in future the AI will be able to ingest any video, blog, books, manuals, license and whole lot of other things and we will just be able to ask questions to it or get a summary from it. I wonder what will happen to actual content then. Currently YouTube is showing info about most watched section of the clips. It saves so much time! Now imagine that happenin…
Perhaps even more exciting - you could write the documentation via a conversation with the chatbot, and it could ask further questions when it is unsure how to answer a user's query and even update the documentation when changes are noticed.
Re: Show HN: Turning books into chatbots with GPT-3
#56Earlier quoted context omitted.
I asked ChatGPT: Me: Consider the story: "Justin is hungry. Justin eats dinner. Justin is not hungry." Is Justin hungry? ChatGPT: No, Justin is not hungry after eating dinner. I'm not sure that it's that big of a problem.
The example was to just illustrate the general problem. Think of ingesting a whole novel that takes place over a few years. The whole novel doesn't fit into GPT's context window (which is only a page or two of text). So you have to extract individual statements of fact and index over them (e.g. with semantic indexing, or many other techniques). It's tricky to deal with cases where the state of something changes many…
One of chatgpt's hidden parameters is what timerange of knowledge it can use to answer. I imagine implementing something similar for 'paging' through the plot could work well. Conversation starts at the beginning of the book and then either explicit syntax or revealing particular information in the conversation 'unlocks' further plot from the bot to draw answers from.
The idea of 'unlocking' information for a chatbot to use in answering feels very compelling for non-fiction as well. Ex. maybe the chatbot requires a demonstration of algebraic knowledge before it can draw from calculus in answering questions. Would feel kind of like a game 'achievement system' which could incentivize people exploring the extent of contained knowledge. And you could generate neat visual maps of the users knowledge.
Re: Show HN: Turning books into chatbots with GPT-3
#57Awesome!
Re: Show HN: Turning books into chatbots with GPT-3
#58This is one application of AI that I absolutely like. Imagine in future the AI will be able to ingest any video, blog, books, manuals, license and whole lot of other things and we will just be able to ask questions to it or get a summary from it. I wonder what will happen to actual content then. Currently YouTube is showing info about most watched section of the clips. It saves so much time! Now imagine that happenin…
Hey! I'm working on building this vision at fragen.co.uk - which can take in Youtube videos (trascribed by whisper + postprocessed into useful chunks), PDFs (properly OCR'd - understands bullet points etc) and webpages (also OCR'd, experimental) - For the problem of data-supported AI search, the content really matters. Our edge is semantically chunking it, in other words we are splitting it up into key facts ready for recall. That makes this problem more solveable.
... and our tool "sees" the above text as ...
For the problem of data-supported AI search, the content really matters. fragen.co.uk's edge is semantically chunking the content, in other words we are splitting the content up into key facts ready for recall. Splitting the content up into key facts ready for recall makes data-supported AI search solveable.
(hope it's visible how an LLM like GPT able to use/quote the above can perform seriously better at those bothersome it/what/where questions and follow ups)
Re: Show HN: Turning books into chatbots with GPT-3
#59Very interesting. May I ask how long did it take to train each book?
Instead of books, I would love to be able to ask a bot a few questions every morning the most personally relevant things that happened around the world. Like news, but asking the AI "How can I take advantage of it?".
[Edit]: "advantage" may be misunderstood. A better wording would be "how can I make use of it" or "how can I make use of this information".
Re: Show HN: Turning books into chatbots with GPT-3
#60Earlier quoted context omitted.
You don’t. You cut it into snippets. For those you create embeddings which allow you to rank them by semantic similarity to a query. You then prompt GPT3 with the question plus, say, the three most relevant snippets from the book. The most difficult thing about the process is preventing the model from making stuff up.
This is exactly what I'm working on! My project is taking Zoom conversation, using pyannote for speaker diarisation, whisper for transcription, pinecone.io for semantic search, then feeding that into GPT-3 so we can ask questions about conversation. For us this is super useful because it's not unusual for our discover sessions to last days and we're all terrible at taking notes. As a nerd, my brain is already buzzing…
This sounds so interesting, do you have any plans to write up a more detailed description of it??