Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SearchHut

searchhut.org

Post reply on HN
Print view View original

111–120 of 161 posts

Re: SearchHut

#111

Post by the_biot »

I think the idea of federation of domain-specific search engines, possibly tied together by one or more front-ends, is a brilliant idea.

I think it's similar to how Google's search works internally, though I doubt the separation is based on a list of domain (as in DNS) names. IIRC they have a set of search modules, and what they return (and how fast they return it) all gets mixed in to the search results according to some weighting. Right below the ads.

If you look at a search system that way, it's easy enough to add modules that do things like search only wikipedia, and display those results in a separate box (like DDG), or parse out currency conversion requests, and display those up top based on some API (like Google). etc

Re: SearchHut

#112

Post by heywire »

Has anyone experimented with creating a search engine that only indexes the landing page of domains? I’m less interested in another Google, and more interested in a way to find new and interesting sites/blogs/etc. Stumbleupon was great for this back in the day.

Seems like it would be an interesting experiment to see what the results would be, indexing only the content / meta tags of “index.html”.

Re: SearchHut

#113

Post by ngshiheng »

richardsocher wrote: post #27
p-e-w wrote: post #13

I like the idea of searching a curated list of domains, but I'm not sure that doing the curation yourself is the best approach considering the huge number of useful but niche websites in existence. I wonder if simply parsing all of Wikipedia (dumps are available, and so are parsers capable of handling them) and building a list of all domains used in external links would do the trick. Wikipedia already has substantial…

That's our strategy at you.com - we start with the most popular sites, crawl them and build apps for them, eg. you.com/apps and let users vote in search results. Full disclosure: I'm the CEO.

> you.com/apps and let users vote

what is stopping companies/users from abusing/gaming on this system with bots?

Re: SearchHut

#114

Post by remram »

the_biot wrote: post #111

I think the idea of federation of domain-specific search engines, possibly tied together by one or more front-ends, is a brilliant idea. I think it's similar to how Google's search works internally, though I doubt the separation is based on a list of domain (as in DNS) names. IIRC they have a set of search modules, and what they return (and how fast they return it) all gets mixed in to the search results according to…

How would you do ranking though?

It is possible for a site's results to be of different quality: maybe one article about MySQL is not so informative, and an article about Python on the same site is a reference.

The search engine operated by the author is unlikely to acknowledge that.

Re: SearchHut

#115

Post by AlphaWeaver »

I love that I can self-host this! Are there plans for federation?

Rather than maintaining a whole separate index for myself, I'd love to self-host an instance of this, only indexing sites that aren't in the main index, and then falling back to the main index / merging it with my index to answer queries. I wonder how easy that would be with the current architecture.

Re: SearchHut

#116

Post by ancientsofmumu »

ddevault wrote: post #42

Good morning, HN. Please note that SearchHut is not done or in a presentable state right now, and those who were in the know were asked not to share it. Alas. I had planned to announce this next week, after we had more time to build up a bigger index, add more features, fix up the 404's and stub pages, do more testing, and so on, so if you notice any rough edges, this is why. I went ahead and polished up the announce…

I think it would be great if we had Code Forge index to search uniquely. In this index are only the myriad of code hosting sites around the internet - shared hosting like gitlab, github, sourcehut, sourceforge, codeberg, and all the project instances like the kernel.org, GNU Savannah, GNOME, KDE, BSD, etc. Probably hundreds of them out there, and allow people to submit their own self-hosted Gitea/Gitlab/sr.ht/etc. instances to be crawled - maybe even suggest a robots.txt entry your crawler could key in on as "yes please index me, hutbot".

Re: SearchHut

#117

Post by Bromeo »

The site has just been taken offline by Drew due to the unfortunate start. I hope we can come back to this once the project has been properly launched, although Drew notes that he is "really unhappy with how the roll-out went" and that "my motivation for this project has evaporated" [1].

Thanks for all the work Drew, I hope you guys manage to come to a conclusion that you are satisfied with!

[1]: https://paste.sr.ht/~sircmpwn/048293268d4ed4254659c3cd6abe67...

Re: SearchHut

#118

Post by jdorfman »

kevincox wrote: post #93
nyanpasu64 wrote: post #88

Does Sourcehut offer textual search within a repo's files? GitHub and GitLab offer it, but Codeberg doesn't seem to (and I couldn't find any information about its presence or absence on Sourcehut).

It doesn't appear to be a code search engine. Just a regular search engine focused on code. Does Sourcegraph index Sourceht projects? It is a proper code search engine and very good.

Looks like SourceHut is down so I'm not sure what projects need indexing.

In other news we now index 87k Rust packages on crates.io

https://sourcegraph.com/search?q=context:global+repo:%5Ecrat...

Re: SearchHut

#119

Post by danskeren »

bastardoperator wrote: post #59

I'm not a fan of google but you can do exactly what this search engine does by curating your own list of domains to search against. https://programmablesearchengine.google.com

Some downsides with this approach:

- search queries are performed directly from the clients computer so can't protect their privacy (since Custom Search JSON API have a daily limit of 10k queries)

- forced to use javascript, and the way it's implemented makes it difficult if not impossible to do even basic things like the loading animation cards

- ads are loaded from an iframe so you can't do any styling (except extremely limited options that they make available in their settings, but no matter what then it will be very ugly if you want to have a light/dark theme)

But there are of course many benefits as well, such as it being 'free' (Bing is ridiculously expensive IMO, and feels impossible to join their ad network to offset the costs.. which might explain why you see countless Bing proxies shut down after a few months) and search results are no doubt better than the ones you'd get from Bing.

Re: SearchHut

#120

Post by marginalia_nu »

Bromeo wrote: post #117

The site has just been taken offline by Drew due to the unfortunate start. I hope we can come back to this once the project has been properly launched, although Drew notes that he is "really unhappy with how the roll-out went" and that "my motivation for this project has evaporated" [1]. Thanks for all the work Drew, I hope you guys manage to come to a conclusion that you are satisfied with! [1]: https://paste.sr.ht/…

Oh, that's sad :-(

For what it's worth, my search engine got prematurely "announced" for me on HN as well, while likewise hilariously janky. I don't think the launch is the end of the world. (I guess I had the benefit of serving comically bizarre results when the queries failed, so it got some love for that)

The bigger struggle is, because a search box is so ambiguous, people tend to have very high expectations for what it can do. Many people just assume it's exactly like Google. It's something a lot of indie search engine developers struggle with. Even if your tool is (or potentially could be) very useful, how can you make people understand how to use it when it looks like something else? Design wise it's a real tough nut to crack, a capital H Hard UX problem.

Post reply on HN