Do you have pay-as-you-go pricing? I feel that’s always missing from things like this. Cool otherwise.
Turn entire websites into LLM-ready data
11–16 of 16 posts
Re: Turn entire websites into LLM-ready data
#12https://gist.github.com/yoki/fd2a3642b059703529d158de8233bbc...
It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently.
If only that script worked for every website. But, alas, it does not.
Re: Turn entire websites into LLM-ready data
#13https://gist.github.com/yoki/fd2a3642b059703529d158de8233bbc...
[deleted]
Re: Turn entire websites into LLM-ready data
#14Does this respect any anti AI related scraping rules set forth by website owners?
It doesn't even respect general robots.txt exclusions, so... I doubt it.