Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Turn any website into a knowledge base for LLMs

embedding.io

Post reply on HN
Print view View original

111–120 of 135 posts

Re: Show HN: Turn any website into a knowledge base for LLMs

#112

Post by marcellus23 »

sooheon wrote: post #80
marcellus23 wrote: post #45

Earlier quoted context omitted.

Responding just because it's a pet peeve of mine: Cixin Liu did not invent the dark forest hypothesis. People were discussing it, and writing science fiction books about it, for decades before the 3BP books were published. Nothing against him, and he definitely helped popularize the concept, but I think it's incorrect to refer to it as "Cixin Liu's hypothesis".

Was curious as a lover of the 3BP series, google gave me this: "We've been sitting in our tree chirping like foolish birds for over a century now, wondering why no other birds answered. The galactic skies are full of hawks, that's why." (The Forge of God, Legend edition, 1989, pg 315). Yeah, same concept and even the same imagery. Source: https://warwick.ac.uk/fac/sci/physics/research/astro/people/...

The Forge of God and its sequel, Anvil of the Stars, are amazing books for anyone interested in the dark forest theory, by the way. A bit slow and contemplative, so you have to be in the right mood, but they're one of my favorite reads of the last few years.

I think there's a passage that even uses an analogy of a forest, though I'm not sure.

Re: Show HN: Turn any website into a knowledge base for LLMs

#113

Post by kmoser »

fkyoureadthedoc wrote: post #96
elric wrote: post #82

Earlier quoted context omitted.

Several lawsuits have confirmed that. Google regurgitating articles from French newspaper sites comes to mind. This is not an easy problem to solve. In my naive take, authors get to decide how their work is used, not scrapers.

> In my naive take, authors get to decide how their work is used, not scrapers. Inasmuch as they've put it on the public web they've already made a decision on who gets to see it, and you really can't stop people from doing what they want with it on a personal level. If that's print it out and put it on a wall in my house, or use whatever tools I have at my disposal to consume it in any way I please, there's not real…

Copyright law says otherwise. As for enforcing the law, you're right, it may be difficult for individual authors to move the needle. But that that doesn't mean it's ok for scrapers to violate the law.

As to what constitutes fair use, that's a whole other story: some scraping may be found to be legal while others may not. Benefiting monetarily from legally dubious scraping only makes that scraping look more infringe-y. Of course, nothing is settled law until a court decides.

Re: Show HN: Turn any website into a knowledge base for LLMs

#116

Post by xrd »

jeanloolz wrote: post #94

I built a similar thing as a python library that does just that: https://github.com/philippe2803/contentmap Blog post that explains the rationale behind the library: https://philippeoger.com/pages/can-we-rag-the-whole-web Just submit your XML sitemap into a python class, and it will do the crawling, chunking, vectorizing and storage in an SQLite file for you. It's using SQLiteVSS integration with Langchain, but think…

I know sqlite-vss has been upgraded lately. But, it was unstable for a while prior. Are you having good experiences with it?

Re: Show HN: Turn any website into a knowledge base for LLMs

#118

Post by sunir »

cranberryturkey wrote: post #2

how does this work?

There are a few ways. I built something similar huckai.com on top of vectara.com. They have open sourced their versions https://github.com/vectara

You can also do this on AWS now fairly easily. https://medium.com/data-reply-it-datatech/how-to-build-a-cus...

The lablab.ai Discord community is a pretty good place to learn how this product category is evolving.

Post reply on HN