Show HN: Turn any website into a knowledge base for LLMs • Hacker News Board

Can I query multiple vectorized websites at once? Can I export vectorized websites and host them myself? Any chance to export them to a no-code format, like PDF?

Re: Show HN: Turn any website into a knowledge base for LLMs

#34

Post by srameshc » Wed, Jul 31, 2024, 7:54 PM UTC

pryelluw wrote: post #24

Does this respect robots.txt?

Valid question and I am sure it doesn't.

Re: Show HN: Turn any website into a knowledge base for LLMs

#35

Post by nhggfu » Wed, Jul 31, 2024, 8:07 PM UTC

[flagged]

Re: Show HN: Turn any website into a knowledge base for LLMs

#36

Post by danirod » Wed, Jul 31, 2024, 8:12 PM UTC

pryelluw wrote: post #24

Does this respect robots.txt?

I hope this gets answered.

Also I've checked their docs to see if there is any mention about the user agents or IP ranges they use for scraping, with no luck.

Re: Show HN: Turn any website into a knowledge base for LLMs

#37

Post by dmje » Wed, Jul 31, 2024, 8:15 PM UTC

23B1 wrote: post #23

I tried it out. This would be extremely useful to me to the point I'd be willing to happily pay for it, as it's something I would have otherwise had to spend a long time hacking together. 1) The returned output from a query seems pretty limited in length and breadth. 2) No apparent way to adjust my prompts to improve/adjust the output e.g. not really 'conversational' (not sure if that is your intent) Otherwise keep d…

Agree with this. I also think the emphasis here (to OP) should be "I'd be willing to happily pay for it" - ie I'd rather be paying a reasonable amount each month for something that is going to remain active that have the large (current) disparity between "free" and "enterprise". I'd say make some middle tiers of (I don't know?) $5 / $10 / $20 a month for reasonable numbers of queries or whatever. Keep the "enterprise" offering there for the biggies, but offer us small players some hope that this will be sufficiently funded / supported.

Brilliant idea, btw, I like it :-)

Re: Show HN: Turn any website into a knowledge base for LLMs

#38

Post by tompec » Wed, Jul 31, 2024, 9:06 PM UTC

rcarmo wrote: post #27

How do I feed it a sitemap?

It currently will try to find a sitemap on its own. But I have on the roadmap to let users add their own.

Re: Show HN: Turn any website into a knowledge base for LLMs

#39

Post by tompec » Wed, Jul 31, 2024, 9:13 PM UTC

webappguy wrote: post #20

tompec wrote: post #16

Earlier quoted context omitted.

It works with pretty much any website, and works well with docs hosted on GitBook yes, I have embedded a website that's hosted there.

Confirmation email doesn't work, so cannot try it. I attemtped twice and checked spam

Apologies, please email me at support at embedding.io. If you have something you'd like embedded, please also mention it so I can set it up for you.

Re: Show HN: Turn any website into a knowledge base for LLMs

#40

Post by tompec » Wed, Jul 31, 2024, 9:20 PM UTC

danirogerc wrote: post #33

Can I query multiple vectorized websites at once? Can I export vectorized websites and host them myself? Any chance to export them to a no-code format, like PDF?

You can group as many websites as you want into a collection. Then query that collection. Not sure what you mean by exporting; you would like to export the vectors themselves? Or just the chunks of text from the websites?