Curated set of domains page is down.
Also, selfish plug I think it would be cool if you added Hackernoon to that list.
SearchHut
101–110 of 161 posts
Re: SearchHut
#102Curated set of domains page is down. Also, selfish plug I think it would be cool if you added Hackernoon to that list.
I think the point is to avoid including low quality websites…
Re: SearchHut
#103Earlier quoted context omitted.
I just tried searching for "python str" on Google. I expected the top result to be a link to the official Python docs for the `str` type, then ideally some relevant StackOverflow questions highlighting common Python issues with strings, bytes, Unicode etc. Instead, the top result was W3Schools. Then came the Python docs, then 5 pages somewhere between blogspam and poor-quality tutorials. Then a ReadTheDocs page datin…
I just tried searching for “python str” on searchhut and the top result is Postgres docs, then Wikipedia article for empty strings and then Drew’s blog. Official Python docs isn’t in the index at all.
For me the second hit is: https://docs.python.org/3/howto/clinic.html
So at least some official Python docs are indexed.
Re: SearchHut
#104Bad serp... Searched 'mdn a'. Google return ' : The Anchor element - HTML: HyperText Markup Language | MDN' SearchHut rerurn a generic: 'MDN Web Docs'
It seems like it uses postgresql's FTS, which will generally drop stop-words so "the", "a", "and" and similar words are dropped. I've been meaning to figure out the best way to deal with this myself, and I'm guessing looking for exact matches first and then running a FTS query could work.
Most of the time, keywords like this come from external anchors as well, which is something that you're gonna be able to leverage with this design (as I understand it).
Re: SearchHut
#105Earlier quoted context omitted.
GitHub doesn't seem to be either. I get that it's a competitor but not being able to search GitHub is probably a deal breaker for most devs that aren't Drew.
I'm not opposed to indexing GitHub, but the signal to noise ratio on GitHub is poor. Nearly all GitHub repositories are useless, so we'd have to filter most of it out. I think instead I'll have to set it up where people can request that specific interesting repositories are added to the index, and maybe crawl /explore to fill in a decent base set.
GitHub is hella tricky to crawl too due to its sheer size and single entry point (meaning slow crawl speed). I've been looking at the problem as well, and so far just ignored it as un-crawlable, but I might do something like crawl only the about pages for repos that are linked to externally some time in the future.
Re: SearchHut
#106I like the idea of searching a curated list of domains, but I'm not sure that doing the curation yourself is the best approach considering the huge number of useful but niche websites in existence. I wonder if simply parsing all of Wikipedia (dumps are available, and so are parsers capable of handling them) and building a list of all domains used in external links would do the trick. Wikipedia already has substantial…
I've looked into this, and found wikipedia's links not to be super useful. Wikipedia prefers references that don't change, so books primarily, and beyond that websites that don't change, so academic journals where you're getting paywalled, and WaybackMachine archives of websites (even if they are still live). You aren't getting much use out of wikipedia.
Re: SearchHut
#107Good morning, HN. Please note that SearchHut is not done or in a presentable state right now, and those who were in the know were asked not to share it. Alas. I had planned to announce this next week, after we had more time to build up a bigger index, add more features, fix up the 404's and stub pages, do more testing, and so on, so if you notice any rough edges, this is why. I went ahead and polished up the announce…
Just a warning from a fellow search engine developer.
If you happen to be cloud hosting this, and if you do not have a global rate limit, implement one ASAP!
Several independent search engines have been hit hard by a botnet soon after they got attention, both mine and wiby.me, and I think a few others. I've had 10-12 QPS, sustained load for weeks after weeks from a rotating set of mostly eastern european IPs.
It's fine if this is on your own infrastructure, but on the cloud, you'll be racking up bills like crazy from something like that :-/
Re: SearchHut
#108I'm not a fan of google but you can do exactly what this search engine does by curating your own list of domains to search against. https://programmablesearchengine.google.com
It requires google account, has tracking and isn't open-source. I'd say it is no go
Add to that the likelihood that Google will just randomly cancel the product one day. Why invest time in this Google product?
Re: SearchHut
#109All 4 search results for "searX" (a self hostable meta-search engine):
Wikipedia: List of Search engines
Drew's blog: We can do better than DuckDuckGo (perhaps the impetus for this project)
Wikipedia: List of free and open source projects
Wikipedia: Internet Privacy
Re: SearchHut
#110All 4 search results for "searX" (a self hostable meta-search engine): Wikipedia: List of Search engines Drew's blog: We can do better than DuckDuckGo (perhaps the impetus for this project) Wikipedia: List of free and open source projects Wikipedia: Internet Privacy
The point of the project is that it's a curated list of sites to crawl, doesnt make sense to crawl other search engines.