Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

github.com

Post reply on HN
Print view View original

91–100 of 143 posts

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#91

Post by sp332 »

mcmoor wrote: post #72
jstummbillig wrote: post #29

Earlier quoted context omitted.

What do you (or anyone else, feel free to chime in) do with other LLMs that makes them useable for anything that is not strictly tinkering? Here is my premise: We are past the wonder stage. I want to actually get stuff done efficiently. From what I have tested so far, the only model that allows me to do that halfway reliably is GPT-4. Am I incompetent or are we really just wishfully thinking in HN spirit that other L…

I still wonder what makes GPT-4 so much better than its contemporaries. That's why I saw tons of people trying to explain how GPT-4 works starting from simple neural network distasteful, tons of people already knew and do that but none of them is nearly close to GPT-4.l

They pay tons of people to type out conversations that they can feed into it. It's just a lot of people doing a lot of work.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#92

Post by nomilk »

tedsanders wrote: post #85
eminent101 wrote: post #41

Is it going to send my personal data to OpenAI? Isn't that a serious problem? Does not sound like a wise thing to do, not at least without redacting all sensitive personal data from the data. Am I missing something?

By default, data sent to the OpenAI API is never used for training and is deleted after a maximum of 30 days (mostly). Data usage policies: https://openai.com/policies/api-data-usage-policies Data usage policies by model: https://platform.openai.com/docs/models/how-we-use-your-data

A few weeks ago GitHub made a strong statement about code in repos not being viewed by humans, that was very liberating.

If OpenAI could offer similar privacy statements it would immediately be much more useful. E.g. if they simply add a 'private' option, I'd pay double or triple for it.

OpenAI's tools are incredibly good and so easy to use, it's just that I simply cannot use them for most the things I want to do with them because of the privacy considerations, and that sucks.

I suspect OpenAI value the insights they get from looking at the data more than they do the extra revenue they'd receive if they could ensure privacy.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#93

Post by rmbyrro »

JimmyRuska wrote: post #30
chaxor wrote: post #2

The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI. Stop doing that. You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around. The best way to make these apps IMO is to make them work entirely local…

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllam…

It depends heavily on the use case, not org size. I consult for a ~70 people org that needs to process ~1M tokens per day. That costs $30K per day on OpenAI ChatGPT API. I'm sure this is not an extraordinary case.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#94

Post by sp332 »

throwaway675309 wrote: post #56
trolan wrote: post #22

Earlier quoted context omitted.

The only OpenAI 'crap' being used here is to generate the embeddings. Right now, OpenAI has some of the best and cheapest embeddings possible, especially for personal projects. Once the vectors are created tho, you're completely off the cloud if you so choose. You can always swap out the embedding generator too, because LangChain abstracts that for your exact gripes. Everything else is already using huggingface here…

What? It's only one file, and it definitely looks like it's using openAI to make the actual queries. qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0.1), db.as_retriever())

You can change the arguments to from_llm() to point to a local model instead. Example here: https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML/discuss...

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#95

Post by sdan »

AJRF wrote: post #3

Is there a company that makes a hosted version of something like this? I quite want a little AI that I can feed all my data to to ask questions to.

If you subscribe to ChatGPT plus, you can use ChatWithPDF (https://plugins.sdan.io) which has 50k+ daily active users!

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#96

Post by quickthrower2 »

swader999 wrote: post #77
JimmyRuska wrote: post #30

Earlier quoted context omitted.

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllam…

Wait until winter time and heat your house!

Good double use of that low entropy energy. Heat pumps excepted.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#97

Post by sdan »

JimmyRuska wrote: post #9

Anyone know how milvus, quickwit, pinecone compares? I've been thinking about seeing if there's consulting opportunities for local businesses for LLMs, finetuning/vector search, chat bots. Also making tools to make it easier to drag and drop files and get personalized inference. Recently I saw this one pop into my linkedin feed, https://gpt-trainer.com/ . There's been a few others for documents I've found https://www…

I am working on a simple vector db just with numpy: https://github.com/sdan/vlite

I think milvus, quickwit, and pinecone are geared more towards enterprise and are hard to use.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#98

Post by two_handfuls »

tedsanders wrote: post #86
einpoklum wrote: post #28

Don't build a personal ChatGPT, and don't let OpenAI, Microsoft and their business partners (and probably the US government) have a bunch of your personal and private information.

By default, data sent to the OpenAI API is never used for training and is deleted after a maximum of 30 days (mostly). Data usage policies: https://openai.com/policies/api-data-usage-policies Data usage policies by model: https://platform.openai.com/docs/models/how-we-use-your-data

So, they don’t promise they won’t look at it - just that they won’t use it for training.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#99

Post by serjester »

rmbyrro wrote: post #93
JimmyRuska wrote: post #30

Earlier quoted context omitted.

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllam…

It depends heavily on the use case, not org size. I consult for a ~70 people org that needs to process ~1M tokens per day. That costs $30K per day on OpenAI ChatGPT API. I'm sure this is not an extraordinary case.

Each person in the org needs 1M GPT-4 token and semantic search can’t be used to trim queries? I would be super curious to know more about this use case.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#100

Post by Imnimo »

This readme is very confusing. It says we're going to use the GPT-2 tokenizer, and use GPT-2 as an embedding model. But looking at the code, it seems to use the default LangChain OpenAIEmbeddings and OpenAI LLM. Aren't those text-embedding-ada-002 and text-davinci-003, respectively?

I don't understand how GPT-2 enters into this at all.

Post reply on HN