Live data from Hacker News

Show HN: Turn any website into a knowledge base for LLMs

embedding.io

91–100 of 135 posts

Re: Show HN: Turn any website into a knowledge base for LLMs

#91
post #89

Earlier quoted context omitted.

What is this thing you've invented, "microfraud", and where's the evidence it exists?

All financial transactions have a fraud risk. Microtransactions are no different. But any microtransaction system faces a choice: continually pop up payment confirmations (unusably annoying), or automatically accept charges (vulnerable to fraud). Click fraud on adverts is a form of microfraud, and pay-per-click is the existing form of microtransaction.

There's zero evidence that this exists, if only because there's very few examples of working microtransactions systems at all. Adverts are not micro transactions. (I don't care how you want to use that word - nobody else uses it like that)

But, all of the systems that I've seen (Blendle, video games) have had no problem at all with fraud, and a very small amount of annoyance to value delivered.

There's simply no reason to believe that this will be a problem, either empirically or theoretically.

Re: Show HN: Turn any website into a knowledge base for LLMs

#92
Does anyone know of a way to do this locally with Ollama? The 'chat with documentation' thing is something I was thinking of a week ago when dealing with hallucinating cloud AI. I think it'd be worth the energy to embed a set of documentation locally to help with development

Re: Show HN: Turn any website into a knowledge base for LLMs

#93
post #89

Earlier quoted context omitted.

All financial transactions have a fraud risk. Microtransactions are no different. But any microtransaction system faces a choice: continually pop up payment confirmations (unusably annoying), or automatically accept charges (vulnerable to fraud). Click fraud on adverts is a form of microfraud, and pay-per-click is the existing form of microtransaction.

There's zero evidence that this exists, if only because there's very few examples of working microtransactions systems at all. Adverts are not micro transactions. (I don't care how you want to use that word - nobody else uses it like that) But, all of the systems that I've seen (Blendle, video games) have had no problem at all with fraud, and a very small amount of annoyance to value delivered. There's simply no reas…

"I'm going to build a system for transferring money, and I am confident that nobody will ever try to defraud it" -- someone who is about to lose all their money

Previously: https://news.ycombinator.com/item?id=15592192

Unsolved, difficult problems of micropayments:

- pay before viewing: how do you know that the thing you're paying for is the thing that you're expecting? What if it's a rickroll or goatse?

- so do you give refunds a la steam?

- pay and adverts: double-dipping is very annoying

- pay and adverts: how do you know who you're paying? A page appears with a micropayment request, but how do you know you've not just paid the advertiser to view their ad?

- pay and frame: can you have multiple payees per displayed page? (this has good and bad ideas)

- pay and popups: it's going to be like those notification or app install modals, yet another annoyance for people to bounce off

- pay limits: contactless has a £30 limit here. Would you have the same payment system suitable for $.01 payments and $1000 payments? How easy is it to trick people into paying over the odds (see refunds)?

- pay and censors: who's excluded from the payment system? Why?

> If it was that easy, it would have been done.

Part 2: business model problems!

- getting money into the system is plagued by usual fraud problems of card TX for pure digital goods

- nobody wants to build a federated system; everyone wants to build a Play/Apple/Steam store where they take 30%

- winner-take-all effects are strong

- Play store et al already exist, why not use that?

- Free substitute goods are just a click away

- Consumers will pirate anything no matter how cheap the original is

- No real consumer demand for micropayments

=> lemma from previous 3 items: market for online goods is efficient enough to drive all marginal prices to zero

- existing problem of the play store letting your kid spend all the money

- friction: it would be great if you didn't have to repeatedly approve things, such as a micropayment for every page of a webcomic archive. But blanket approval lets bad actors drain the jar or inattentive users waste it and then feel conned

- first most obvious model for making this work is porn, which is inevitably blacklisted by the payment processors, has a worse environment for fraud/chargebacks, and is toxic to VCs (see Patreon and even Craigslist)

- Internet has actually killed previously working micro-ish payment systems such as Minitel, paid ringtones (anyone remember the dark era of Crazy Frog?); surviving ones like premium SMS and phone have a scammy, seedy feel.

- accounting requirements: do you have to pay VAT on that micropayment? do you have to declare it? Is it a federal offence to sell something to an Iranian or North Korean for one cent?

Re: Show HN: Turn any website into a knowledge base for LLMs

#94
I built a similar thing as a python library that does just that: https://github.com/philippe2803/contentmap

Blog post that explains the rationale behind the library: https://philippeoger.com/pages/can-we-rag-the-whole-web

Just submit your XML sitemap into a python class, and it will do the crawling, chunking, vectorizing and storage in an SQLite file for you. It's using SQLiteVSS integration with Langchain, but thinking of moving away from it, and do an integration with the new sqlite-vec instead.

Re: Show HN: Turn any website into a knowledge base for LLMs

#96
post #82

Earlier quoted context omitted.

Well, Google itself is just an unethical bot scrapper then...

Several lawsuits have confirmed that. Google regurgitating articles from French newspaper sites comes to mind. This is not an easy problem to solve. In my naive take, authors get to decide how their work is used, not scrapers.

> In my naive take, authors get to decide how their work is used, not scrapers.

Inasmuch as they've put it on the public web they've already made a decision on who gets to see it, and you really can't stop people from doing what they want with it on a personal level.

If that's print it out and put it on a wall in my house, or use whatever tools I have at my disposal to consume it in any way I please, there's not really anything the author can do about it.

Re: Show HN: Turn any website into a knowledge base for LLMs

#97
post #92

Does anyone know of a way to do this locally with Ollama? The 'chat with documentation' thing is something I was thinking of a week ago when dealing with hallucinating cloud AI. I think it'd be worth the energy to embed a set of documentation locally to help with development

https://github.com/open-webui/open-webui

Re: Show HN: Turn any website into a knowledge base for LLMs

#98
post #90

Interested in seeing whether this will be widespread in 5 years or whether sites will have fought back.

Sites are already fighting back.

Twitter and Reddit locked down their APIs. Soon enough, you’ll need an account to even access any content

Post reply on HN