Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why AutoGPT engineers ditched vector databases

dariuszsemba.com

Post reply on HN
Print view View original

21–30 of 59 posts

Re: Why AutoGPT engineers ditched vector databases

#21

Post by clarionbell »

rowanG077 wrote: post #19
batmansmk wrote: post #14

« AutoGPT engineers » seem to also generate their articles with LLM, making their documentation awful to grok. For instance, after showing 2 commands, we have to suffer this: Forge your future! The forge is your innovation lab. All the boilerplate code is already handled, letting you channel all your creativity into building a revolutionary agent. It's more than a starting point, it's a launchpad for your ideas. In o…

I doubt it's generated with an LLM. LLMs are much too easy to get to generate much higher quality text then the one you quoted.

You good LLMs, that's probably not what worked on this.

Re: Why AutoGPT engineers ditched vector databases

#22

Post by shapefrog »

rowanG077 wrote: post #19
batmansmk wrote: post #14

« AutoGPT engineers » seem to also generate their articles with LLM, making their documentation awful to grok. For instance, after showing 2 commands, we have to suffer this: Forge your future! The forge is your innovation lab. All the boilerplate code is already handled, letting you channel all your creativity into building a revolutionary agent. It's more than a starting point, it's a launchpad for your ideas. In o…

I doubt it's generated with an LLM. LLMs are much too easy to get to generate much higher quality text then the one you quoted.

I am pretty you can get there by doing a print, scan, OCR and then translate it to and from a foreign language and voila.

Re: Why AutoGPT engineers ditched vector databases

#23

Post by IanCal »

Fannon wrote: post #15

Wouldn't a vector DB be nice to have, so you can use it directly for search? I understand the argumentation of the article. But I can imagine that waiting so long for a LLM to react that I would actually prefer to do a search instead on a vector database on my "additional information layer" and find relevant information myself. In that case, having a vector DB would then serve two purposes and that could change the c…

I think that just comes down to speed. The utility is identical, just when you're doing it for one user you can deal with a large problem (for you) in the most naive way possible in a few seconds. I would bet good money that there's a slightly more complex but still very basic step up that would get seconds down to something that feels instantaneous.

Re: Why AutoGPT engineers ditched vector databases

#24

Post by mirzap »

singularity2001 wrote: post #17
mirzap wrote: post #12

Earlier quoted context omitted.

Vector DBs don't create embeddings; they store them. As the article points out, the LLM's slowness to respond diminishes the performance that vector DB's can potentially add.

You can easily create embeddings locally though, with small (L)LMs. Three lines of code using hugging face. I don't understand the point of this article.

Look it this way: you have a web app that can handle 10 req/s. It does not matter if you add a database behind it that can handle 10,000 req/s. 10 req/s still limit you. You're not gaining any performance benefits. I always wondered why you would need a dedicated vector DB.

Re: Why AutoGPT engineers ditched vector databases

#25

Post by tudorw »

jasfi wrote: post #10
jondwillis wrote: post #3

I have been working on a system of agents over at https://github.com/agi-merge/waggle-dance - I already split problems up into subtasks for agents to work on independently. I give agents access to vector databases, using a simple global key for now, but soon a context/parent/child key. Access to the vector DBs is proxied via tools (agents have to “call” saveMemory or retrieveMemory). I also check for looping/repetiti…

What's your assessment of the biggest blocking issues for something like this to be practically useful? From what I've seen of AutoGPT things seem to fall apart, in that the goals never quite seem to be achieved once anything more than basic research is requested.

It can do stuff, sort of, like helping me to create ; https://github.com/tudorw/Ai_MegaList/blob/main/AI_Sector_Br...

After a lot of trial and error, I managed to keep it somewhat on track by using a CSV file, something like;

"An expert manipulate .csv files, read the first URL from the first line, 2nd column of 'raw.csv', pass the URL to browse_website the questions 'summarize the activities, highlight any investment, funding or patents mentioned', regardless of the results or failure, write the data quote delimited and in columns where appropriate to a new line of 'complete.csv', pass the URL to google and summarize the answer to the question 'does this organisation have a good reputation, from reliable sources', regardless of the results or failure, write the data quote delimited and in columns where appropriate to a new line of 'complete.csv', remove the 1 line from 'raw.csv' you have processed, repeat the process until 'raw.csv' has no more URL in it"

On the plus side, it was very quick to iterate, 'programming' in words is an exercise in linguistics, it's ability to scrape from any site was impressive, on the downside, it really struggled to stay on task, and even when things seems to be working well, random behaviour was normal, so it might just decide to delete the csv as a short cut...

On Windows it's ability to engage PowerShell was equally enlightening and terrifying... As an exercise in instructing an AI it was interesting, I'd certainly try again if the requirements fitted.

I think it's a credit to the team that they explored options for vector storage then retreated in the name of complexity, it's a good reason.

Re: Why AutoGPT engineers ditched vector databases

#26

Post by luke-stanley »

mirzap wrote: post #12
sytelus wrote: post #11

Is there any implementation of open source vector db that is fast enough to say create embedding of 100M documents locally within few hours and find ranked matches in under a second? I tried ChromaDb and it is super slow, basically unusable.

Vector DBs don't create embeddings; they store them. As the article points out, the LLM's slowness to respond diminishes the performance that vector DB's can potentially add.

Actually, some Vector DB's can generate embeddings as well as storing them, Chroma in particular uses SentenceTransformers models.

Re: Why AutoGPT engineers ditched vector databases

#27

Post by malux85 »

sytelus wrote: post #11

Is there any implementation of open source vector db that is fast enough to say create embedding of 100M documents locally within few hours and find ranked matches in under a second? I tried ChromaDb and it is super slow, basically unusable.

Generating the embeddings is by far the slowest part of that, but it’s embarrassingly parallel so if you have $ it can be done that quick.

When I worked for Dubai airport, I was tasked with building a vector similarity search that was highly optimised for query speed, in the end I ended up holding the vectors in memory (in a numpy array) and using scipy to do cosine similarity, I could get about 1.2 million vectors per second per core after tweaking and optimising, again this is embarrassingly parallel so if you have more vectors, chunk it to fit your hardware and you should get more or less linear scale with that per core.

If you want a hand writing this let me know.

(Also there’s a lot of caveats here, for example they did not need to update the vectors, it was an extremely read-heavy usage pattern)

Re: Why AutoGPT engineers ditched vector databases

#28

Post by montebicyclelo »

Fannon wrote: post #15

Wouldn't a vector DB be nice to have, so you can use it directly for search? I understand the argumentation of the article. But I can imagine that waiting so long for a LLM to react that I would actually prefer to do a search instead on a vector database on my "additional information layer" and find relevant information myself. In that case, having a vector DB would then serve two purposes and that could change the c…

They do still carry out search, but using a brute force approach (dot product over all the embeddings) instead of vector db. Their point is that they aren't likely to generate enough messages for brute force to be an issue. (Even over 100s of thousands of embeddings brute force is pretty fast.)

Re: Why AutoGPT engineers ditched vector databases

#29

Post by amelius »

batmansmk wrote: post #14

« AutoGPT engineers » seem to also generate their articles with LLM, making their documentation awful to grok. For instance, after showing 2 commands, we have to suffer this: Forge your future! The forge is your innovation lab. All the boilerplate code is already handled, letting you channel all your creativity into building a revolutionary agent. It's more than a starting point, it's a launchpad for your ideas. In o…

Don't read the intermediate representation. The idea is that you use an LLM to summarize those comments into human readable text.

Re: Why AutoGPT engineers ditched vector databases

#30

Post by singularity2001 »

mirzap wrote: post #24
singularity2001 wrote: post #17

Earlier quoted context omitted.

You can easily create embeddings locally though, with small (L)LMs. Three lines of code using hugging face. I don't understand the point of this article.

Look it this way: you have a web app that can handle 10 req/s. It does not matter if you add a database behind it that can handle 10,000 req/s. 10 req/s still limit you. You're not gaining any performance benefits. I always wondered why you would need a dedicated vector DB.

vector DBs have much higher 'semantic' recall than classical search engines if you want to ask questions about your documents or previous discussions.
Post reply on HN