Earlier quoted context omitted.
You won't be able to scan most of websites this way because most servers expect you to also pass a valid hostname. However you can use domain lists, for example https://purecrawl.com/en/download/domains (or https://domains-monitor.com/ which is paid but has more domains) as an initial seed shouldn't be too bad, but you'll have to ingest terabytes of spammy/low quality content.
wouldn't certificate transparency logs be a good way to collect most active domains?
Xonaly – Canada's Independent Search Engine
41–50 of 51 posts
Re: Xonaly – Canada's Independent Search Engine
#42Re: Xonaly – Canada's Independent Search Engine
#43Re: Xonaly – Canada's Independent Search Engine
#44Re: Xonaly – Canada's Independent Search Engine
#45Something I couldn't find info about is how this is funded, given that it is ad free and doesn't sell user data. Is it supported by a nonprofit organisation or just paid for out of the developer's pocket?
This is a critical piece of information to convince me to switch to this. I'm quite interested though! Also, what's the context of "Built in Canada"? What's the point of mentioning this? Should I expect the results to be better for Canadian content?
In addition to direct subsidies/grants, the government has a buy Canadian mandate.
Re: Xonaly – Canada's Independent Search Engine
#46Re: Xonaly – Canada's Independent Search Engine
#47Re: Xonaly – Canada's Independent Search Engine
#48Re: Xonaly – Canada's Independent Search Engine
#49The search engine does indeed seem to be fully custom - the results for niche topics are quite bad, but that at least means it's not a Bing frontend, so it's forgivable. Although I wonder why the authors decided to go 100% custom instead of (partially?) reusing https://github.com/MarginaliaSearch/MarginaliaSearch for some of the core engine. On an unrelated note: why do some people use LLMs to write all the text in t…