Is there a way to deal with websites where you need to login? Like subscription based sites?
Unless you own those sites, I'm afraid that's not going to be possible.
[dead]
Show HN: Turn any website into a knowledge base for LLMs
11–20 of 135 posts
Re: Show HN: Turn any website into a knowledge base for LLMs
#12Interesting, I wanted to do this for a personal use case (mostly learning), but with PDFs. What's tech stack? I have explored using the AWS AI tools, but it seems a bit overkill for what I want it to do.
If the PDFS are textual or have OCR, then pdf2text from the Poppler suite ought to be enough? If not, add Tesseract/ocrmypdf to the pipeline?
Re: Show HN: Turn any website into a knowledge base for LLMs
#13Interesting, I wanted to do this for a personal use case (mostly learning), but with PDFs. What's tech stack? I have explored using the AWS AI tools, but it seems a bit overkill for what I want it to do.
Here's some code to deal with that:
https://github.com/MittaAI/SlothAI/blob/main/SlothAI/lib/pro...
https://github.com/MittaAI/mitta-community/tree/main/service...
There's code in there that just reads PDF meta data as well, but you can't always guarantee it's there in a PDF.
Re: Show HN: Turn any website into a knowledge base for LLMs
#14how does this work?
I do this with https://mitta.ai by using a Playwright container that does a callback to a pipeline that uses either meta data from the PDF or sends it to an EasyOCR deployment on a GPU instance on Google for text extraction. Then I use a custom chunker and instructor/xl embeddings.
All of that code is Open Source, and works well for most sites. Some sites block Google IPs, but the Playwright container can run locally, so should be able to work around it with some minimal effort.
Re: Show HN: Turn any website into a knowledge base for LLMs
#15This is interesting. Can it work with any website, even say document repositories hosted on standard servers like gitbook?
Re: Show HN: Turn any website into a knowledge base for LLMs
#16This is interesting. Can it work with any website, even say document repositories hosted on standard servers like gitbook?
It works with pretty much any website, and works well with docs hosted on GitBook yes, I have embedded a website that's hosted there.
Re: Show HN: Turn any website into a knowledge base for LLMs
#17Interesting, I wanted to do this for a personal use case (mostly learning), but with PDFs. What's tech stack? I have explored using the AWS AI tools, but it seems a bit overkill for what I want it to do.
Tech stack is a mix of serverless Laravel, with Cloudflare and AWS functions, and some Pinecone for vector storage. Still experimenting on a few things but don't want to over-engineer unless I know where I'm going.
Given that cloudflare spies on traffic and reports to multiple agencies on it's findings, perhaps a breakdown of the chain and the privacy implications of each block in the stack would be beneficial?
Re: Show HN: Turn any website into a knowledge base for LLMs
#18Nice! What's the underlying model / RAG approach being used? Be good to understand that part as presumably it will have a big impact on performance / usability of the results.
Re: Show HN: Turn any website into a knowledge base for LLMs
#19Does it hallucinate much?
Re: Show HN: Turn any website into a knowledge base for LLMs
#20This is interesting. Can it work with any website, even say document repositories hosted on standard servers like gitbook?
It works with pretty much any website, and works well with docs hosted on GitBook yes, I have embedded a website that's hosted there.
Confirmation email doesn't work, so cannot try it. I attemtped twice and checked spam