Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I used GPT to build a search tool for my second brain note-taking system

reasonabledeviations.com

Post reply on HN
Print view View original

71–80 of 103 posts

Re: I used GPT to build a search tool for my second brain note-taking system

#71

Post by johntash »

DanielVZ wrote: post #36
abrkn wrote: post #32

I’d love to have a ChatGPT that was also trained on all of the pages from my “second brain,” Roam Research. Imagine, I could ask it questions about myself, my friends, and my business. It would in many ways know me better than me from reading all my journal entries. How many years are we away from something like this?

Zero. I’ve done this with google drive and GPT-3 (thus quite limited in prompt length). The biggest hurdle for your requirements is the nonexistent public API for ChatGPT and Roam Research.

I was tempted to do something like this but couldn't get over the idea of sending a bunch of personal data off to openai (or any 3rd party really)

Re: I used GPT to build a search tool for my second brain note-taking system

#72

Post by ilaksh »

kanyethegreat wrote: post #40
ilaksh wrote: post #12

Earlier quoted context omitted.

This is not a fine-tuning example. It's an embedding search example. You use the embeddings to search for relevant knowledgebase chunks and then include them in the prompt. Which goes to the original model, not a model that you have trained more. This is popular because it's much much easier to do effectively than fine tuning and the OpenAI model is very capable of integrating kb snippets into a response. What I have…

Would you be able to do something similar with non-text data (eg. tabular data)? For example, could you give it a bunch of excel files and ask it do give you the total units sold for an e-commerce site?

What I think would make sense would be to do it in two stages, and embeddings probably aren't really what you want. You would want to parse the Excel files into a certain data structure, and quite possibly text completions could help with that.

Put that in a database or some files that you could use Python data science tools with or something.

Then use text completions to translate the natural language query into some short Python program or SQL query etc.

There are already data-focused tools for using OpenAI's newest models for doing this. Search for 'ChatGPT/GPT/OpenAI' data query, SQL, datatable, etc. See the OpenAI Discord #api-projects Discord, I have seen one or two like that.

Re: I used GPT to build a search tool for my second brain note-taking system

#74

Post by james-revisoai »

throwaway675309 wrote: post #38
trane_project wrote: post #25

I've been thinking of using GPT or similar LLMs to extract flashcards to use with my spaced repetition project ( https://github.com/trane-project/trane/ ). As in you give it a book and it creates the flashcards for you and the dependencies between the lessons. I played around with chatgpt and it worked pretty well. I have a lot of other things in my plate to get around first (including starting a math curriculum) but…

I've seen a number of projects around using GPT to generate curriculum and also flashcards in the past three months, I think this is one of the most popular ones: https://autolearnify.com It's a very good idea in theory but takes almost as much work to verify that the flashcards and curriculum that it generates is accurate and not a hallucinogenic nightmare. The biggest danger is that the target audience are not expe…

Curious as to where you came to this impression?

I'd say the most popular applications are Knowt in the US for now, and Saveall.ai + Revision.ai (my company) in the UK, all been around with BERT/T5 etc long before this GPT trend.

The flashcard accuracy varies wildly amongst current solutions, that's for sure.

Re: I used GPT to build a search tool for my second brain note-taking system

#75

Post by dewey »

rolenthedeep wrote: post #31

One of my biggest dreams is a self-hosted AI that always listens through my phone and automatically takes notes, puts events in my calendar, set reminders, and template journal entries. A true personal assistant to keep my increasingly-complex life in order. I'd love a system where I can just point a search engine at my brain. I tried really hard for a while, but I just didn't have the discipline or memory to exhaust…

I feel like this was the promise of voice assistants like Siri many years ago, turns out they currently still have issues turning on / off lights reliably.

Re: I used GPT to build a search tool for my second brain note-taking system

#76

Post by asdff »

Did the author show how this system outputted results? I see an example of a lexical search and the technical implementation, but no example of some semantic output showing how its relevant to the lexical search string without containing that string. The author used the literal search string "failure mode" as their example. I was wondering if chatgpt would bring up results relevant to the lay person interpretation of failure mode, a technical interpretation, or something in between.

Re: I used GPT to build a search tool for my second brain note-taking system

#77

Post by asdff »

dewey wrote: post #75
rolenthedeep wrote: post #31

One of my biggest dreams is a self-hosted AI that always listens through my phone and automatically takes notes, puts events in my calendar, set reminders, and template journal entries. A true personal assistant to keep my increasingly-complex life in order. I'd love a system where I can just point a search engine at my brain. I tried really hard for a while, but I just didn't have the discipline or memory to exhaust…

I feel like this was the promise of voice assistants like Siri many years ago, turns out they currently still have issues turning on / off lights reliably.

I'll believe an AI model can update my calendar accurately only after it can correctly tell if I just said 15 or 50.

Re: I used GPT to build a search tool for my second brain note-taking system

#79

Post by lm28469 »

pessimist wrote: post #27
michaericalribo wrote: post #5

These "augmented intelligence" applications are so exciting to me. I'm not as interested in autonomous artificial intelligence. Computers are tools to make my life easier, not meant to lead their own lives! There's a big up-front cost of building a notes database for this application, but it illustrates the point nicely: encode a bunch of data ("memories"), and use an AI like GPT to retrieve information ("remembering…

In chess we had a tiny window of a few years when humans could use the help of computers to play the world's best chess. By 2000, computers were far better than humans and the gap has increased. Chess players are now entertainers, like all us humans are destined to spend our time doing.

> like all us humans are destined to spend our time doing

Automation has been here for quite a long time now, if it took people out of the work pool for them to become entertainer we'd know about it.

It's always the same issue in fact, replacing workers by machines is good, but if your goal is still to have a "full employment" society you have to make them work somewhere else. It's not even a new concept but we seem to rediscover it every now and then apparently

> Automation, the most advanced sector of modern industry as well as the model which perfectly sums up its practice, drives the commodity world toward the following contradiction: the technical equipment which objectively eliminates labor must at the same time preserve labor as a commodity and as the only source of the commodity. If the social labor (time) engaged by the society is not to diminish because of automation (or any other less extreme form of increasing the productivity of labor), then new jobs have to be created. Services, the tertiary sector, swell the ranks of the army of distribution and are a eulogy to the current commodities; the additional forces which are mobilized just happen to be suitable for the organization of redundant labor required by the artificial needs for such commodities.

Guy Debord, 1967

Re: I used GPT to build a search tool for my second brain note-taking system

#80

Post by leobg »

gk1 wrote: post #59
leobg wrote: post #28

Earlier quoted context omitted.

You’d want persistence, since the embedding process takes some time. But you don’t need to go all Pinecone on this. There is FAISS, and there is hnswlib, for example. Like SQLite for vector search.

Friendly reminder that we (Pinecone) have a free tier that holds up to ~5M SBERT embeddings (x768 dimensions). For quick projects, going "all Pinecone on this" could turn out to be the easier and faster option.

Point taken ;-)

I like to stand up for the little guy. I hear Pinecone this and Pinecone that. And nobody seems to pay any attention to the awesome dude who made hnswlib.

Post reply on HN