111–120 of 131 posts
Also, plans to add HTTPS?
This looks cool, though, good luck!
I think what could be cool is applying this as a personal search engine and marrying it somehow to a personal dns server or squid/proxy server so that you can have a way of harvesting your own browsing data. By using the squid or dnsmasq logs you could spider out urls from it, and build your index automatically.
Non-centralized personal search engines have a few challenges to solve before they're feasible. 1) The web cannot support thousands or even millions of spiders/crawlers.
2) Search indexes are (probably?) too huge to distribute. See the commoncrawl project. It's TB for a few Billion pages.
3) Assuming a single crawler collects the necessary data, indexes can be easily distributed, and the search engine software is simple to set-up, who is going to subsidise this effort?
Earlier quoted context omitted.
Major search engines test every release against a list of search queries. You could start with https://trends.google.com/trends/topcharts . You should have an automated test script with a list of (query, good URL) pairs and make sure the good URL appears in the top few results.
As preparation for this demo, yes I absolutely should have run such a test. Eagerness won. Thanks for the link.
I really like this idea, and the very simple implementation - big things start small. We need more search engines, including ones which are not supported by advertising. Thanks for submitting.
91 results for 'hello world' in 'www' (32615 ms)
As others have commented, love the ambitiousness of this! However, Unicode searches do not seem to work at all -- not just "中文" but also even "français" gives an error. Unicode support is something you definitely want to build in from the very beginning in order to avoid headaches (for you and users) in the future. Even if there is no content in the index, the presence of non-ASCII characters in the search term shoul…