Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

github.com

Post reply on HN
Print view View original

111–120 of 143 posts

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#111

Post by space_fountain »

rolisz wrote: post #107
ben_w wrote: post #39

Earlier quoted context omitted.

The only embeddings I currently see listed on https://openai.com/pricing are Ada v2, at $0.1/million tokens. Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? If you're getting less than a million tokens and it takes half an hour longer to set up, you'd better be a student with zero literally income because that c…

If you've never coded or used Python before, yeah, go with OpenAI. Otherwise, generating embeddings with SentenceBERT takes 5 minutes. And from my personal experience Ada embeddings are not the best. They are large (makes aproximate searching harder), are distributed weirdly, and zimply put, other embeddings give better results for retrieval. Another advantage is that you are not an OA's whim: they just announced the…

Yes, exactly this, I also want to say I'm not someone who generally thinks open models are better. I think embeddings just haven't been a focus for OpenAI and it shows. Maybe in the future they will focus on it

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#112

Post by aledalgrande »

trolan wrote: post #22
chaxor wrote: post #2

The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI. Stop doing that. You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around. The best way to make these apps IMO is to make them work entirely local…

The only OpenAI 'crap' being used here is to generate the embeddings. Right now, OpenAI has some of the best and cheapest embeddings possible, especially for personal projects. Once the vectors are created tho, you're completely off the cloud if you so choose. You can always swap out the embedding generator too, because LangChain abstracts that for your exact gripes. Everything else is already using huggingface here…

> Once the vectors are created tho, you're completely off the cloud if you so choose.

Ehr no? You'll need to also create an embedding of your query, which makes you totally dependent on OpenAI. If you swap out embedding algorithm you will have to regenerate all the embeddings as well, they might not be even the same size.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#113

Post by avereveard »

ben_w wrote: post #39
space_fountain wrote: post #25

Earlier quoted context omitted.

Do you have citations on OpenAI embeddings being some of the cheapest and best? The signs I've seen points almost in the opposite direction?

The only embeddings I currently see listed on https://openai.com/pricing are Ada v2, at $0.1/million tokens. Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? If you're getting less than a million tokens and it takes half an hour longer to set up, you'd better be a student with zero literally income because that c…

Setting up an embedding alternative out of huggingface sentence transformers is fairly easy, the magical thing openai does is that they will create embedding of 8192 characters at a time while most other emerging will force you to chunk your documents in 512 characters long sequences, losing lot of context, multiplying your queries result, search times etc

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#114

Post by tartakovsky »

tikkun wrote: post #59
JimmyRuska wrote: post #30

Earlier quoted context omitted.

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllam…

Doing this. We soft launched yesterday with a paid Falcon-40B playground - 3 models for now Falcon 40b instruct, uncensored, and base. Adding API and per token pricing this week. https://api.llm-utils.org/ And more models coming soon. Vector storage isn’t on the roadmap (what stops using a separate vector store from working well? Could add to roadmap but want to add understand more first), and we could add fine tunin…

Not secure... NET::ERR_CERT_COMMON_NAME_INVALID Subject: *.safezone.mcafee.com

Issuer: McAfee OV SSL CA 2

Expires on: Aug 3, 2023

Current date: Jul 8, 2023

PEM encoded chain: -----BEGIN CERTIFICATE----- MIIGfzCCBWegAwIBAgIQKt9VNrFtaozA1bILX1OcfzANBgkqhkiG9w0BAQsFADBk MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#115

Post by PeterStuer »

xigency wrote: post #64
JimmyRuska wrote: post #30

Earlier quoted context omitted.

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllam…

I’ve got an expensive GPU at home I’m not even using because there aren’t that many things to do with it. Give me more local options.

Even if you are not into coding there are many good AI tools that run local. Two very easy examples:

I've had great fun with the " Easiest 1-click way to install and use Stable Diffusion on your computer."

https://github.com/easydiffusion/easydiffusion

And while Whisper is OpenAI, it is trivial to use locally and extremely usefull

https://github.com/chidiwilliams/buzz

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#116

Post by raffraffraff »

drdaeman wrote: post #16
yoyopa wrote: post #10

Earlier quoted context omitted.

what hardware do you need for that?

Consumer-grade, AFAIK it's GPT4All with LLaMA.

That's doesn't really answer the question.

Neither does the github "System requirements" section, which I find disappointing. Ideally, it should give minimum memory requirements and rudimentary performance benchmark table for a sample data set, across a handful of setups (eg, Intel CPU, Apple M1, AMD CPU, with/without a bunch of common GPUs). With that information I would know whether or not it's worth my time even trying it out on my laptop.

Edit: lol, I went through 2 pages of the issues on the github page and most of them could be avoided by putting this basic information into the system requirements:

https://github.com/imartinez/privateGPT/issues/174 https://github.com/imartinez/privateGPT/issues/179 https://github.com/imartinez/privateGPT/issues/104 https://github.com/imartinez/privateGPT/issues/141 https://github.com/imartinez/privateGPT/issues/282 https://github.com/imartinez/privateGPT/issues/316 https://github.com/imartinez/privateGPT/issues/333

... And lots more! Some of these people have 128gb memory and 32 cores, and still find it "very slow". Others having memory pool errors. Some of the answers hand-waving at needing "a more better computer"

I reckon a lot of these issues could be closed and linked to a single ticket for proper hardware requirements in the readme.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#117

Post by einpoklum »

tedsanders wrote: post #86
einpoklum wrote: post #28

Don't build a personal ChatGPT, and don't let OpenAI, Microsoft and their business partners (and probably the US government) have a bunch of your personal and private information.

By default, data sent to the OpenAI API is never used for training and is deleted after a maximum of 30 days (mostly). Data usage policies: https://openai.com/policies/api-data-usage-policies Data usage policies by model: https://platform.openai.com/docs/models/how-we-use-your-data

You many want to read about National Security Letters:

https://www.eff.org/issues/national-security-letters/faq

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#118

Post by ignoramous »

JimmyRuska wrote: post #67
xigency wrote: post #64

Earlier quoted context omitted.

I’ve got an expensive GPU at home I’m not even using because there aren’t that many things to do with it. Give me more local options.

https://github.com/oobabooga/text-generation-webui https://github.com/bentoml/OpenLLM https://www.reddit.com/r/LocalLLaMA/top/?t=month

And

https://github.com/go-skynet/LocalAI

https://github.com/juliooa/secondbrain

https://github.com/louisgv/local.ai

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#119

Post by muspimerol »

theonlybutlet wrote: post #104
BeetleB wrote: post #78

Earlier quoted context omitted.

Example use case: We have a group at work that meets and discusses various investment topics. The guy organizing it is fairly well connected and every week he tries to get an external speaker to come and present. Very educational. I have raw notes for each of these presentations. My goal has always been to go through those notes, and properly organize the knowledge in there into a wiki of sorts. It's been 3 years sin…

You're creating additional hardship for yourself. Why create a pdf only to convert it out of pdf again. Just insert all your notes into the LLM model.

Because that requires retraining the model every time you take new notes. And this way you also still have the raw notes as similarity matches from the vector db, rather than them "disappearing" into the LLM model.
Post reply on HN