Show HN: Turn any website into a knowledge base for LLMs

#1

Post by tompec » Tue, Jul 30, 2024, 12:54 AM UTC

I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications.

It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically.

I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...

Show HN: Turn any website into a knowledge base for LLMs
embedding.io

Re: Show HN: Turn any website into a knowledge base for LLMs

#2

Post by cranberryturkey » Tue, Jul 30, 2024, 12:57 AM UTC

how does this work?

Re: Show HN: Turn any website into a knowledge base for LLMs

#3

Post by [deleted] » Tue, Jul 30, 2024, 1:09 AM UTC

[deleted]

Re: Show HN: Turn any website into a knowledge base for LLMs

#4

Post by tompec » Tue, Jul 30, 2024, 1:11 AM UTC

cranberryturkey wrote: post #2

how does this work?

Give it URLs or domains, and it will crawl and extract their content, embed them in a vector database, and give you an endpoint that you can then query when doing RAG stuff or semantic search.

Re: Show HN: Turn any website into a knowledge base for LLMs

#5

Post by samuria » Tue, Jul 30, 2024, 6:08 AM UTC

Interesting, I wanted to do this for a personal use case (mostly learning), but with PDFs. What's tech stack? I have explored using the AWS AI tools, but it seems a bit overkill for what I want it to do.

Re: Show HN: Turn any website into a knowledge base for LLMs

#6

Post by xiconfjs » Tue, Jul 30, 2024, 6:12 AM UTC

tompec wrote: post #4

cranberryturkey wrote: post #2

how does this work?

Give it URLs or domains, and it will crawl and extract their content, embed them in a vector database, and give you an endpoint that you can then query when doing RAG stuff or semantic search.

But how does it work in the background? What‘s the tech stack?

Re: Show HN: Turn any website into a knowledge base for LLMs

#7

Post by blackeyeblitzar » Tue, Jul 30, 2024, 6:54 AM UTC

Is there a way to deal with websites where you need to login? Like subscription based sites?

Re: Show HN: Turn any website into a knowledge base for LLMs

#8

Post by tompec » Tue, Jul 30, 2024, 9:21 AM UTC

blackeyeblitzar wrote: post #7

Is there a way to deal with websites where you need to login? Like subscription based sites?

Unless you own those sites, I'm afraid that's not going to be possible.

Re: Show HN: Turn any website into a knowledge base for LLMs

#9

Post by tompec » Tue, Jul 30, 2024, 9:26 AM UTC

samuria wrote: post #5

Interesting, I wanted to do this for a personal use case (mostly learning), but with PDFs. What's tech stack? I have explored using the AWS AI tools, but it seems a bit overkill for what I want it to do.

Tech stack is a mix of serverless Laravel, with Cloudflare and AWS functions, and some Pinecone for vector storage. Still experimenting on a few things but don't want to over-engineer unless I know where I'm going.

Re: Show HN: Turn any website into a knowledge base for LLMs

#10

Post by ramon156 » Tue, Jul 30, 2024, 9:34 AM UTC

xiconfjs wrote: post #6

tompec wrote: post #4

Earlier quoted context omitted.

Give it URLs or domains, and it will crawl and extract their content, embed them in a vector database, and give you an endpoint that you can then query when doing RAG stuff or semantic search.

But how does it work in the background? What‘s the tech stack?

In another comment:

> Tech stack is a mix of serverless Laravel, with Cloudflare and AWS functions, and some Pinecone for vector storage. Still experimenting on a few things but don't want to over-engineer unless I know where I'm going.