Untitled topic

I made a ScrapeGPT, an open source local tool designed to perform content analysis of the entire website and answer questions based on the scraped content.

Basically the idea came from the job interview question when I was given an assignment to "study our company website and provide new product recommendations", and since I was pretty lazy, I decided to automate the process of studying a whole website.

The app works completely locally and scrapes the entire website and its sublinks (with regards to robots.txt) and uses RAG pipeline to retrieve answers from the text. It supports both local LLMs (Ollama) and public ones (I use Perplexity API, since its cheaper to use their offline models).

You can use it as a: - CLI tool (pls see commented out main() function); - Telegram bot (default); - Gradio app (separate file, work in progress).

I made it for the educational purposes sharing it as a mean to get some feedback.

Feel free to fork and contribute, just don't forget to mention me and please use it within legal boundaries.

Repo: https://github.com/LexiestLeszek/ScrapeGPT

Best.