Live data from Hacker News

LLMs, RAG, and the missing storage layer for AI

blog.lancedb.com

61–64 of 64 posts

Re: LLMs, RAG, and the missing storage layer for AI

#61

Earlier quoted context omitted.

I'm aware of more efficient ways to do it! (Hence referencing e.g. doc2query.) But you have to train a model, whereas with an LLM you can get a working version in 5mins of work.

But with even less work you can just pick up a model that was pre-trained using GPL and get great results. I'm able to pull messy results directly from internet sources and re-rank on the fly with a quantized e5 model small enough to fit in a serverless function. You don't need a vector database to do all this stuff, people who are paid off people using vector databases are the ones who are hyping them up the most.

Oh, I wasn't suggesting using a vector DB. Personally I just iterate through the corpus and check cosine similarity with a for loop.

If by "quantized e5 model small enough to fit in a serverless function" you mean e5-small-v2, FYI it actually underperforms just calling OpenAI for embeddings (text-embedding-ada-002) on the HuggingFace MTEB benchmarks. And that definitely doesn't negate using a doc2query-style approach to preprocess the documents before running them through the pretrained embedding model if you're comparing e.g. questions to answers, rather than raw document-to-document similarity. (Of course a custom trained model will be more efficient! In fact, the original doc2query paper in 2019 used a custom trained model for step 1, as did many enhancements on it e.g. doc-t5-query. What's neat is that with the advent of really good pretrained LLMs, you can get results approximating that without training your own models in like ~5mins of work.)

Re: LLMs, RAG, and the missing storage layer for AI

#62

As an architect working on LLM applications I have these criteria for a database. - Full SQL support - Has good tooling around migrations (i.e. dbmate) - Good support for running in Kubernetes or in the cloud - Well understood by operations i.e. backups and scaling - Supports vectors and similarity search. - Well supported client libraries So basically Postgres and PgVector.

Yes totally agree with that (and other comments below). Moving from a toy example to production deployment requires all the things we are used to having in robust/mature products like postgres.

Re: LLMs, RAG, and the missing storage layer for AI

#63
Does ChatGPT always start articles with “in the rapidly evolving landscape of X”?

Surely if you’re posting an article promoting miraculous AI tech you should human edit the article summary so that it’s not really obviously drafted by AI.

Or just use the prompt “tone your writing down and please remember that you’re not writing for a high school student who is impressed by nonsensical hyperbole”. I’ve started using this prompt and it works astonishingly well in the fast evolving landscape of directionless content creation.

Re: LLMs, RAG, and the missing storage layer for AI

#64

Earlier quoted context omitted.

But with even less work you can just pick up a model that was pre-trained using GPL and get great results. I'm able to pull messy results directly from internet sources and re-rank on the fly with a quantized e5 model small enough to fit in a serverless function. You don't need a vector database to do all this stuff, people who are paid off people using vector databases are the ones who are hyping them up the most.

Oh, I wasn't suggesting using a vector DB. Personally I just iterate through the corpus and check cosine similarity with a for loop. If by "quantized e5 model small enough to fit in a serverless function" you mean e5-small-v2, FYI it actually underperforms just calling OpenAI for embeddings (text-embedding-ada-002) on the HuggingFace MTEB benchmarks. And that definitely doesn't negate using a doc2query-style approach…

I guess this really boils down to your usecase: if you can have a result for your user with fully predictable latency (my biggest beef with non-Azure OpenAI), no additional round trip, and increased configurability, does MTEB performance move the needle?

Considering the LLM is still doing the final pass, and the latency from the LLM is based on output length, I find the UX to be significantly improved just doing reranking in-process.

I think there's been a bit of whiplash, where people went from gatekeeping "hard ML", to "I can shove this all at a REST API", but there's a golden path laying in between for use-cases where UX matters.

I even fall back to old school NLP (like ML-less, glorified wordlist POS taggers) for LLM tasks and end up with significantly improved performance for almost 0 additional effort

Post reply on HN