Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

github.com

Post reply on HN
Print view View original

51–60 of 143 posts

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#51

Post by srcthr »

JimmyRuska wrote: post #30
chaxor wrote: post #2

The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI. Stop doing that. You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around. The best way to make these apps IMO is to make them work entirely local…

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllam…

People don't scale. This is personal. Only 3 is a good choice for people in a site with the name hacker something.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#52

Post by minimaxir »

yawnxyz wrote: post #42
csjh wrote: post #15

Why have the OpenAI dependency when there's local embeddings models that would be both faster and more accurate?

Which ones?

all-MiniLM-L6-v2 from SentenceTransformers is the most popular one as it balances speed and quality well: https://www.sbert.net/docs/pretrained_models.html

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#53

Post by dmezzetti »

chaxor wrote: post #2

The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI. Stop doing that. You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around. The best way to make these apps IMO is to make them work entirely local…

txtai makes it easy to use Hugging Face embeddings and Faiss, all local and configurable. https://github.com/neuml/txtai

paperai is a sub-project focused on processing medical/scientific papers. https://github.com/neuml/paperai

Disclaimer: I am the author of both

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#54

Post by muggermuch »

colobas wrote: post #37
space_fountain wrote: post #25

Earlier quoted context omitted.

Do you have citations on OpenAI embeddings being some of the cheapest and best? The signs I've seen points almost in the opposite direction?

Can you elucidate on what those signs are? Thanks in advance

As per the Massive Text Embedding Benchmark (MTEB) Leaderboard maintained by Huggingface, OpenAI's embedding models are not the best.

https://huggingface.co/spaces/mteb/leaderboard

Of course, that's far from saying that they're the worst, or even headed that way. Just not the best (those would be a couple of fully opensource models, including those of the Instructor family, which we use at my workplace).

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#55

Post by peyton »

3rdrodeo wrote: post #46
EGreg wrote: post #32

Earlier quoted context omitted.

I see, so when it comes to the top experts in AI screaming “we made a mistake, please be careful” we should use nuance and actually conclude the opposite — that we should press ahead and they’re wrong. But with Web3, we should just listen to a bunch of random no-name haters say “there are NO GOOD APPLICATIONS, trust us, period, stop talking about it”, use no nuance or critical thinking of our own, and simply stop bui…

The crypto group had a lot of time and even more money to make a compelling product that took off and so far they've failed. We've watched fraud after fraud as they've shown themselves to just be ignorant and arrogant ideologues who don't understand how the "modern" finance system came to be, what the average user wants out of financial or social products, or just outright scammers. We can keep sinking money into a b…

I dunno, on the crypto side stablecoins are pretty compelling for hassle-free cross-border transfers—there’s $125bn in circulation, which to me means it’s taken off.

On the AI side, I mean for example it’s not laughable to think anybody on the planet could just feed a bunch of synthetic biology papers to a model and start designing bioweapons. It’s not hard to get your hands on secondhand lab equipment…

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#56

Post by throwaway675309 »

trolan wrote: post #22
chaxor wrote: post #2

The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI. Stop doing that. You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around. The best way to make these apps IMO is to make them work entirely local…

The only OpenAI 'crap' being used here is to generate the embeddings. Right now, OpenAI has some of the best and cheapest embeddings possible, especially for personal projects. Once the vectors are created tho, you're completely off the cloud if you so choose. You can always swap out the embedding generator too, because LangChain abstracts that for your exact gripes. Everything else is already using huggingface here…

What? It's only one file, and it definitely looks like it's using openAI to make the actual queries.

    qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0.1), db.as_retriever())

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#57

Post by whoiscroberts »

JimmyRuska wrote: post #30
chaxor wrote: post #2

The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI. Stop doing that. You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around. The best way to make these apps IMO is to make them work entirely local…

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllam…

I have a 4080, let’s do a startup. #cancode #hashomelab

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#58

Post by akiselev »

ben_w wrote: post #39
space_fountain wrote: post #25

Earlier quoted context omitted.

Do you have citations on OpenAI embeddings being some of the cheapest and best? The signs I've seen points almost in the opposite direction?

The only embeddings I currently see listed on https://openai.com/pricing are Ada v2, at $0.1/million tokens. Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? If you're getting less than a million tokens and it takes half an hour longer to set up, you'd better be a student with zero literally income because that c…

If you look at a embeddings leaderboard [1], one of the top competitors called InstructorXL [2] is just a pip install away. It's neck and neck with Ada v2 except for a shorter input length and half the dimensions, with the added benefit that you'll always have the model available.

Most of the other options just work with the transformers library.

[1] https://huggingface.co/spaces/mteb/leaderboard

[2] https://github.com/HKUNLP/instructor-embedding

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#59

Post by tikkun »

JimmyRuska wrote: post #30
chaxor wrote: post #2

The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI. Stop doing that. You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around. The best way to make these apps IMO is to make them work entirely local…

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are - Pay openai less than $50mo - Manage cloud gpus, hire ml engineers > $1000/mo - Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed Any larger business will need scalability and you still can't compete with openai pricing. Maybe one of you startup inclined people can make an openllam…

Doing this. We soft launched yesterday with a paid Falcon-40B playground - 3 models for now Falcon 40b instruct, uncensored, and base. Adding API and per token pricing this week.

https://api.llm-utils.org/

And more models coming soon.

Vector storage isn’t on the roadmap (what stops using a separate vector store from working well? Could add to roadmap but want to add understand more first), and we could add fine tuning if it’s a common request.

Re: PdfGptIndexer: Indexing and searching PDF text data using GPT-2 and FAISS

#60

Post by tikkun »

JimmyRuska wrote: post #9

Anyone know how milvus, quickwit, pinecone compares? I've been thinking about seeing if there's consulting opportunities for local businesses for LLMs, finetuning/vector search, chat bots. Also making tools to make it easier to drag and drop files and get personalized inference. Recently I saw this one pop into my linkedin feed, https://gpt-trainer.com/ . There's been a few others for documents I've found https://www…

Pinecone and Milvus would be alternatives for their use of FAISS for the vector store and search component. I think more of the embeddings difference would be noticed by what’s used for creating the embeddings (eg the ones here https://news.ycombinator.com/item?id=36649579 instead of the OpenAI embeddings API they used), rather than noticing differences from the embedding store/search alternatives which I can’t think of what the difference would be other than maybe performance at a large scale and cost and personal preference / developer experience.

Hadn’t heard of Quickwit but from a quick glance at their site it doesn’t look like a vector store, seems perhaps unrelated.

For tools for making custom ChatGPTs see my list: https://llm-utils.org/List+of+tools+for+making+a+%22ChatGPT+...

Fine tuning as a service there’s Lamini AI, aimed at enterprises.

Other embeddings startups there’s Weaviate.

Post reply on HN