does it embbed images as well? if not, do you plan to do so?
It doesn't embed images, no. But that's a great idea for the roadmap!
Show HN: Turn any website into a knowledge base for LLMs
71–80 of 135 posts
Re: Show HN: Turn any website into a knowledge base for LLMs
#72Re: Show HN: Turn any website into a knowledge base for LLMs
#73[deleted]
Re: Show HN: Turn any website into a knowledge base for LLMs
#74In my opinion this is a transitional niche. Soon websites/apps whatever you want to call them will have their own built-in handling for AI. It's inefficient and rude to be scraping pages for content. Especially for profit.
[flagged]
Re: Show HN: Turn any website into a knowledge base for LLMs
#75In my opinion this is a transitional niche. Soon websites/apps whatever you want to call them will have their own built-in handling for AI. It's inefficient and rude to be scraping pages for content. Especially for profit.
In agree that this niche is DOA. No offence to OP but the barrier for entry to this stuff is low. I built basically the same thing over a weekend for personal use. React frontend, python server, chroma for embeddings, sqlite cache, switch between open AI and anthropic (I want to add llama for full local execution when I get a better pc). I have a local SPA with named "projects", can configure crawl depth from a start…
God. Some people on hackernews suck.
This isn’t “niche”, it’s a pretty cool thing OP has built.
How about instead of commenting and trivialising what people have done, you say something positive
Re: Show HN: Turn any website into a knowledge base for LLMs
#76Interesting, I wanted to do this for a personal use case (mostly learning), but with PDFs. What's tech stack? I have explored using the AWS AI tools, but it seems a bit overkill for what I want it to do.
Try aichat: https://github.com/sigoden/aichat
Re: Show HN: Turn any website into a knowledge base for LLMs
#77I feel like this is unethical. You built yet another bot scraper. It would only be an ethical tool if it validated I own the website I am scraping before it starts.
This is probably a losing direction - protecting your little island of content in the sea of internet and LLM outputs. Get more value by exposure. This is the trend of open source, wikipedia and open scientific publication. LLMs double down on the same collaborative approach to intelligence.
You can of course decouple from the big discussion and isolate your content with access restrictions, but the real interesting activity will be outside. Look for example the llama.cpp and other open source AI tools we have gotten recently. So much energy and enthusiasm, so much collaboration. Closed stuff doesn't get that level of energy.
I think IP laws are in for a reckoning, protecting creativity by restricting it is not the best idea in the world. There are better models. Copyright is anachronic, it was invented in the era of the printing press when copying was made easy, LLMs remix they don't simply copy, even the name is unfitting for the new reality. We need to rename it remixright.
Re: Show HN: Turn any website into a knowledge base for LLMs
#78The example API key on the page is decoded to "WOW YOU'RE A HACKER" :)
Re: Show HN: Turn any website into a knowledge base for LLMs
#79I feel like this is unethical. You built yet another bot scraper. It would only be an ethical tool if it validated I own the website I am scraping before it starts.
Well, Google itself is just an unethical bot scrapper then...
Re: Show HN: Turn any website into a knowledge base for LLMs
#80I like this a lot! But: I feel the more of these services come to being, the more likely it is that every website starts putting up gates to keep the bots away. Sort of like a weird GenAI take on Cixin Liu's Dark Forest hypothesis ( https://en.wikipedia.org/wiki/Dark_forest_hypothesis ). (Edited to add a reference.)
Responding just because it's a pet peeve of mine: Cixin Liu did not invent the dark forest hypothesis. People were discussing it, and writing science fiction books about it, for decades before the 3BP books were published. Nothing against him, and he definitely helped popularize the concept, but I think it's incorrect to refer to it as "Cixin Liu's hypothesis".
Was curious as a lover of the 3BP series, google gave me this:
"We've been sitting in our tree chirping like foolish birds for over a century now, wondering why no other birds answered. The galactic skies are full of hawks, that's why." (The Forge of God, Legend edition, 1989, pg 315).
Yeah, same concept and even the same imagery.
Source: https://warwick.ac.uk/fac/sci/physics/research/astro/people/...