This is awesome. I love anything that helps me discover new personal sites/blogs.
Search My Site – open-source search engine for personal and independent websites
21–30 of 53 posts
Re: Search My Site – open-source search engine for personal and independent websites
#22Other sites:
Generally I crawl the internet to find pages. The result is in https://github.com/rumca-js/Internet-Places-Database. Personal pages are tagged with "personal" tag.
Re: Search My Site – open-source search engine for personal and independent websites
#23Also worth a mention is Wiby. "The Wiby search engine is building a web of pages as it was in the earlier days of the internet." It's main indexing requirements are: - "Pages must be simple in design. Simple HTML, non-commerical sites are preferred." - "Pages should not use much scripts/css for cosmetic effect. Some might squeak through." - "Don't use ads that are intrusive (such as ads that appear overtop of content…
This is most definitely not the same thing. The indexing requirements are not "This site must be an independent or personal site" , it's "This site must lean towards being a plain HTTP document" . The Search My Site, from what I can tell, has the goal of surfacing personal/independent websites, while Wiby has the goal of surfacing minimally styled documents. Two different goals.
I'm certainly not presenting Wiby as being the same thing, merely as something that is worth a mention due to likely being of interest to anyone interested in Search My Site.
It is relevant and has vaguely aligned intent.
Re: Search My Site – open-source search engine for personal and independent websites
#24Earlier quoted context omitted.
Sites are ranked higher when they have no ads. Fully open source. That's a good starting point..
> Sites are ranked higher when they have no ads. That’s a pretty clever filter tbh. Sounds so evident that I’m amazed nobody thought of it before. I’d love Kagi to have such an option.
> That’s a pretty clever filter tbh. Sounds so evident that I’m amazed nobody thought of it before.
I did. I posted it on HN as a comment. It was a very popular (by my standards) comment: https://news.ycombinator.com/item?id=40438288
The thread was interesting, with a lot of people posting rebuttals for why such a scheme would obviously not work. Equally obviously, someone else thought it was a good idea and went and implemented it.
Re: Search My Site – open-source search engine for personal and independent websites
#25I love the simplecity of https://pagefind.app/
Interesting, is that a more complete variation of fuse.js? (Just pluged-in fuse.js into my static jekyll blog)
Re: Search My Site – open-source search engine for personal and independent websites
#26I am a bit confused.
Solr is the search engine.
An LLM model is loaded. What does the LLM model add to the solution?
Re: Search My Site – open-source search engine for personal and independent websites
#27I love the simplecity of https://pagefind.app/
This is just what I wanted, thank you for that!
Re: Search My Site – open-source search engine for personal and independent websites
#28No basics like typo-resistance?
digiatl
> No results found for digiatl.
Re: Search My Site – open-source search engine for personal and independent websites
#29Thanks for putting this together. I wonder, is Postgres a bit of a large DB if it's just a personal website search tool? I'll have to give it a go. We need more tools like this.
Postgres is just used for the site admin, i.e. keeping track of submissions, review status, subscriptions etc. The actual search index is in Apache Solr. In theory you could use Solr to store all the admin data, but it is generally not recommended to use a Solr style document store to master data. I guess something more lightweight like SQLite could be used, but it is intended to be deployed on servers and Postgres isn't too resource intensive.
Re: Search My Site – open-source search engine for personal and independent websites
#30I am a bit confused. Solr is the search engine. An LLM model is loaded. What does the LLM model add to the solution?
The LLM was for an experiment in retrieval augmented generation, i.e. "a chat with your website" style interface, using Apache Solr as the vector store. Results (on a small self-hosted LLM to keep costs manageable) weren't good enough for the functionality to be fully rolled out, so the LLM has been disabled and is likely to be fully removed.