Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Turn any website into a knowledge base for LLMs

embedding.io

Post reply on HN
Print view View original

41–50 of 135 posts

Re: Show HN: Turn any website into a knowledge base for LLMs

#41

Post by tompec »

23B1 wrote: post #23

I tried it out. This would be extremely useful to me to the point I'd be willing to happily pay for it, as it's something I would have otherwise had to spend a long time hacking together. 1) The returned output from a query seems pretty limited in length and breadth. 2) No apparent way to adjust my prompts to improve/adjust the output e.g. not really 'conversational' (not sure if that is your intent) Otherwise keep d…

Thanks! The chat demo is actually just a small thing I put together as a preview of what can be done, but the main product is the API. But seeing that most users seem to like that, there's probably something there... If you want to email me at support at embedding.io with some requirements, I can see how to make that work for you.

Re: Show HN: Turn any website into a knowledge base for LLMs

#42

Post by Cynddl »

I find it interesting that as an (edit: UK) academic researcher, I would be likely be forbidden to use tools like this, that fail basic ethics standards, regulations such as GDPR, and practical standards such as respecting robots.txt [given there's no information on embedding.io, it's unlikely I can block the crawler when designing a website].

There's still room for an ethical development of such crawlers and technologies, but it needs to be consent-first, with strong ethical and legal standards. The crazy development of such tools has been a massive issue for a number of small online organisations that struggle with poorly implemented or maintained bots (as discussed for OpenStreetMap or Read The Docs).

Re: Show HN: Turn any website into a knowledge base for LLMs

#43

Post by stevenicr »

stevenicr wrote: post #17
tompec wrote: post #9

Earlier quoted context omitted.

Tech stack is a mix of serverless Laravel, with Cloudflare and AWS functions, and some Pinecone for vector storage. Still experimenting on a few things but don't want to over-engineer unless I know where I'm going.

Given that cloudflare spies on traffic and reports to multiple agencies on it's findings, perhaps a breakdown of the chain and the privacy implications of each block in the stack would be beneficial?

Ya know, a downvote on this pre-aug 2019 would be fine.

people still being ignorant about their publicly posted policies 5 years later is annoying.

Re: Show HN: Turn any website into a knowledge base for LLMs

#45

Post by marcellus23 »

muggermuch wrote: post #28

I like this a lot! But: I feel the more of these services come to being, the more likely it is that every website starts putting up gates to keep the bots away. Sort of like a weird GenAI take on Cixin Liu's Dark Forest hypothesis ( https://en.wikipedia.org/wiki/Dark_forest_hypothesis ). (Edited to add a reference.)

Responding just because it's a pet peeve of mine: Cixin Liu did not invent the dark forest hypothesis. People were discussing it, and writing science fiction books about it, for decades before the 3BP books were published. Nothing against him, and he definitely helped popularize the concept, but I think it's incorrect to refer to it as "Cixin Liu's hypothesis".

Re: Show HN: Turn any website into a knowledge base for LLMs

#47

Post by tompec »

dmje wrote: post #37
23B1 wrote: post #23

I tried it out. This would be extremely useful to me to the point I'd be willing to happily pay for it, as it's something I would have otherwise had to spend a long time hacking together. 1) The returned output from a query seems pretty limited in length and breadth. 2) No apparent way to adjust my prompts to improve/adjust the output e.g. not really 'conversational' (not sure if that is your intent) Otherwise keep d…

Agree with this. I also think the emphasis here (to OP) should be "I'd be willing to happily pay for it" - ie I'd rather be paying a reasonable amount each month for something that is going to remain active that have the large (current) disparity between "free" and "enterprise". I'd say make some middle tiers of (I don't know?) $5 / $10 / $20 a month for reasonable numbers of queries or whatever. Keep the "enterprise…

Thanks! I'm still figuring things out about pricing, but there will be small plans available.

Re: Show HN: Turn any website into a knowledge base for LLMs

#49

Post by pryelluw »

tompec wrote: post #46
pryelluw wrote: post #24

Does this respect robots.txt?

It does respect robots.txt when crawling. I'll add more details about this in the docs.

I appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.
Post reply on HN