Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pinecone raises $100M Series B

pinecone.io

Post reply on HN
Print view View original

91–100 of 100 posts

Re: Pinecone raises $100M Series B

#91

Post by arecurrence »

itake wrote: post #41
arecurrence wrote: post #23

I was using pinecone before installing pgvector in Postgres. Pinecone works and all but having the vectors in Postgres resulted in an explosion of use for us. Full relational queries with where clauses and order by etc AND vector embeddings is wicked.

Why do you use pgvector instead of pgANN? My understanding is pgANN is built with FAISS. When I compared pgvector with FAISS, pgvector was 3-5x slower. https://github.com/netrasys/pgANN

There is certainly a wide variety of problems today for which pgvector is unsuitable due to performance limitations... but fear not! This is an area that is getting significant focus right now.

A marqo.ai dev is currently working on adding HNSW-IVF and HNSW support https://news.ycombinator.com/item?id=35551684 and the maintainer has recently noted that they are actively working on an IVFPQ/ScaNN implementation https://github.com/pgvector/pgvector/issues/93

The pgAnn creator actually asked about performance a month ago here https://github.com/pgvector/pgvector/issues/58

Expect to see performance improve dramatically later this year.

Re: Pinecone raises $100M Series B

#92

Post by darkteflon »

danielmarkbruce wrote: post #85
darkteflon wrote: post #81

Earlier quoted context omitted.

Context constraint is a cheap way to keep the model on-topic. So rather than relying on an ever-growing context window to stuff/mapreduce more undifferentiated “context” (the entire chat history), interposing a vector search engine that only returns relevant context tends to get you better overall model performance, in addition to being scalable in a way that increasing context window size is not.

Agreed. But summarization is better to keep the model on topic for most cases. And there are other tricks. Vectors and semantic search are one (likely questionable way given LLMs can likely reason over a table of contents or similar better) to search a large corpus or very large document. It's really only appropriate for a specific set of use cases. It's not some "general memory layer" for AI.

From my perspective, it’s not clear why you would want to use bulk summarisation of all context versus summarisation over “relevant” vectors, since it is both substantially more expensive and less effective, since you are effectively polluting the context window with “irrelevant” context. And the problem is compounded as you scale up - even as you scale up trivially.

Admittedly I’m hand-waving a bit around “relevant” and “irrelevant” - clearly your vector search setup has to be fit for purpose. That’s a talent all on its own, so I wonder if we will see competing approaches at the vectorstore level or if it’s relatively settled. Anyway, I’m out of my depth at that point so I’ll leave it there.

Re: Pinecone raises $100M Series B

#94

Post by appenz »

danielmarkbruce wrote: post #90
appenz wrote: post #89

Earlier quoted context omitted.

Summarization is much more expensive than vector db's. Assume you have 1m tokens of context. You could run all through GPT-4 and summarize the information, but it would cost $60 (based on current prices) and take 10's of minutes of GPU time to do the inference. Disclaimer: I work for a16z and on the infra team, so consider me biassed.

If you look through the comments here, folks are mostly referring to keeping for example a chat history. No one is doing 1m words of chat. A common pattern is to summarize a chat history and pass that in the prompt. As for a corpus of documents (which is what you are presumably talking about), there are a couple problems with what you are saying: First, you are implying that the content is always new - that's not tru…

Chat history may work, it depends on how long it is and the business model.

I don't quite understand how general summarization would work. If you use an LLM to simply to summarize in order to feed it into a prompt, the summarization needs to be specific to the query. i.e. "summarize what this text says about topic X". You can't summarize long text in a generic way without losing information. Or do I misunderstand the comment?

If you have a perfect table of context (or better, an index by topic) you may not need semantic search. But for the typical use case we are seeing you have unstructured data without an index (e.g. tech support knowledge db entries, company reports, emails). For that, semantic search work quite well.

For the sizes, the observation is that the data that people want to search over (e.g. your email, a wiki, JIRA, a knowledge base) is far larger than the context length. You are correct that we assume that inference cost and speed won't decrease sufficiently quickly in the near future. Why is a longer topic, but in a nutshell GPU speed increase is ~2.5x gen/gen and other than overtraining vs. Chinchilla we don't see immediate model gains. But that is speculative, we don't know what's in store.

To some degree we are just reacting to user adoption in the market. We don't build these systems, but if we see enough of them eventually we recognize the pattern. And while I am optimistic, we could be wrong. AI is major revolution and we are all students.

edit: disclaimer, I work for a16z.

Re: Pinecone raises $100M Series B

#95

Post by rahimnathwani »

darkteflon wrote: post #76
danielmarkbruce wrote: post #72

Is it possible Andreessen are misunderstanding how pinecone/vector dbs are used? It seems like they are pitching it as "memory for large language models" or something. Are people using vector db's in some way I'm not aware of? To me it's a database to help you do a semantic search. A multi-token string is converted into a single embedding. Like maybe 1000 words into one embedding. This is helpful because you can quic…

I might be mistaken, but my understanding from having played around with LangChain for a couple months is that because you’ve got to keep all your state in the context window, giving the model access to a vectorstore containing the entire chat history allows it to retrieve relevant messages against your query that can then be stuffed or mapreduced into the context window for the next response. The alternative - and I…

Yes, one of the memory options in Langchain is to use a vector store:

https://python.langchain.com/en/latest/modules/memory/types/...

It also has a more basic version that just keeps a log of past messages.

I don't know whether there's a way (or even a need) to combine these approaches. In a long conversation, it might be useful to trust more recent information more than earlier messages, but Langchain's vector memory doesn't care about sequence.

Re: Pinecone raises $100M Series B

#97

Post by danielmarkbruce »

appenz wrote: post #94
danielmarkbruce wrote: post #90

Earlier quoted context omitted.

If you look through the comments here, folks are mostly referring to keeping for example a chat history. No one is doing 1m words of chat. A common pattern is to summarize a chat history and pass that in the prompt. As for a corpus of documents (which is what you are presumably talking about), there are a couple problems with what you are saying: First, you are implying that the content is always new - that's not tru…

Chat history may work, it depends on how long it is and the business model. I don't quite understand how general summarization would work. If you use an LLM to simply to summarize in order to feed it into a prompt, the summarization needs to be specific to the query. i.e. "summarize what this text says about topic X". You can't summarize long text in a generic way without losing information. Or do I misunderstand the…

Yeah, everything here seems basically reasonable, I'd quibble with a couple things but it's debatable. And we might be talking past each other a little bit on use cases. Anyway, it's a fun space.

edit: To me this is a better summary of what a vector db is useful for: https://cloud.google.com/blog/topics/developers-practitioner...

And if someone is building a chat interface which is effectively a search product then they are going to find these things useful. But it's not a generic LLM memory layer or something.

Re: Pinecone raises $100M Series B

#98

Post by danielmarkbruce »

darkteflon wrote: post #92
danielmarkbruce wrote: post #85

Earlier quoted context omitted.

Agreed. But summarization is better to keep the model on topic for most cases. And there are other tricks. Vectors and semantic search are one (likely questionable way given LLMs can likely reason over a table of contents or similar better) to search a large corpus or very large document. It's really only appropriate for a specific set of use cases. It's not some "general memory layer" for AI.

From my perspective, it’s not clear why you would want to use bulk summarisation of all context versus summarisation over “relevant” vectors, since it is both substantially more expensive and less effective, since you are effectively polluting the context window with “irrelevant” context. And the problem is compounded as you scale up - even as you scale up trivially. Admittedly I’m hand-waving a bit around “relevant”…

I think it's unsettled and we'll see some clever things which combine approaches. On the surface it seems like preprocessing a corpus in clever ways will be useful.

Re: Pinecone raises $100M Series B

#100

Post by kordlessagain »

danielmarkbruce wrote: post #98
darkteflon wrote: post #92

Earlier quoted context omitted.

From my perspective, it’s not clear why you would want to use bulk summarisation of all context versus summarisation over “relevant” vectors, since it is both substantially more expensive and less effective, since you are effectively polluting the context window with “irrelevant” context. And the problem is compounded as you scale up - even as you scale up trivially. Admittedly I’m hand-waving a bit around “relevant”…

I think it's unsettled and we'll see some clever things which combine approaches. On the surface it seems like preprocessing a corpus in clever ways will be useful.

If we read a document, that's preprocessing it. It's useful for being able to discuss later, or bring that understanding to bear on a different, yet related problem space.

I agree that a combined approach is likely useful.

Post reply on HN