Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SearchHut

searchhut.org

Post reply on HN
Print view View original

151–160 of 161 posts

Re: SearchHut

#151

Post by imiric »

a5huynh wrote: post #144
bastardoperator wrote: post #59

I'm not a fan of google but you can do exactly what this search engine does by curating your own list of domains to search against. https://programmablesearchengine.google.com

For those looking for an alternative to that, I've been building a self-hosted search engine that crawls what you want based on a basic set of rules. It can be a list of domains, a very specific list of URLs, and/or even some basic regexes. https://github.com/a5huynh/spyglass

Great project! Given a local archive of Wikipedia and other sources, this can be very powerful.

Which raises the question: does archive.org offer their Wayback Machine index for download anywhere? Technically, why should anyone go through the trouble of crawling the web if archive.org has been doing it for years, and likely has one of the best indexes around? I've seen some 3rd-party downloaders for specific sites, but I'd like the full thing. Yes, I realize it's probably petabytes of data, but maybe it could be trimmed down to just the most recent crawls.

If there was a way of having that index locally, it would make a very powerful search engine with a tool like yours.

Re: SearchHut

#152

Post by rawgabbit »

richardsocher wrote: post #27
p-e-w wrote: post #13

I like the idea of searching a curated list of domains, but I'm not sure that doing the curation yourself is the best approach considering the huge number of useful but niche websites in existence. I wonder if simply parsing all of Wikipedia (dumps are available, and so are parsers capable of handling them) and building a list of all domains used in external links would do the trick. Wikipedia already has substantial…

That's our strategy at you.com - we start with the most popular sites, crawl them and build apps for them, eg. you.com/apps and let users vote in search results. Full disclosure: I'm the CEO.

I just learned about you.com from this thread. It looks very promising.

Re: SearchHut

#153

Post by reachableceo »

93po wrote: post #140
reachableceo wrote: post #136

Quite passive aggressive if you ask me. Boo-boo someone shared your project before you were “ready”. If you don’t want something disclosed , don’t disclose it. Only way for three people to keep a secret is if two of them are dead. A thing is in the world. Let it be in the world. Harness the collective power and focus it into a force multiplier. Or don’t.

It's not passive aggressive. It's sensitive, but he has a right to be if he wants to. He wasn't petty or mean spirited in his announcement to take it down. He only expressed that he was taking the feedback very hard, which is understandable if you had big plans to roll out and make a good first impression.

Meh. Hacker news is the place to get actual real feedback. Frying pan to fire etc.

Develop a thick skin or don’t read the comments lol!

He chose to disclose it to a few people. Word spreads. That’s what happens.

Execute NDAs and have a security program if you don’t want stuff getting out.

Re: SearchHut

#154

Post by f0xJtpvHYTVQ88B »

dijit wrote: post #110
f0xJtpvHYTVQ88B wrote: post #109

All 4 search results for "searX" (a self hostable meta-search engine): Wikipedia: List of Search engines Drew's blog: We can do better than DuckDuckGo (perhaps the impetus for this project) Wikipedia: List of free and open source projects Wikipedia: Internet Privacy

The point of the project is that it's a curated list of sites to crawl, doesnt make sense to crawl other search engines.

You don't think a popular open source project fits within this niche?

Re: SearchHut

#155

Post by petercooper »

birken wrote: post #14

> SearchHut indexes from a curated set of domains. The quality of results is higher as a result, but the index covers a small subset of the web. [citation needed] The quality of the results right now are not very high, and in theory I don't understand why one would believe a search engine with a hand picked set of domains would be expected to outcompete a search engine that can crawl the entire web and determines rep…

If you were to tell me curated domains is just a bootstrapping method and as the search engine evolves it will change, fine

This makes me think of a possible approach. Curate a giant set of domains that almost exclusively host high quality content. Crawl said domains. Use all of the crawled data as a training set to create a model with which to ascertain the quality of random Web pages from other domains. Then spider everything and run it against the model.

Re: SearchHut

#156

Post by dijit »

f0xJtpvHYTVQ88B wrote: post #154
dijit wrote: post #110

Earlier quoted context omitted.

The point of the project is that it's a curated list of sites to crawl, doesnt make sense to crawl other search engines.

You don't think a popular open source project fits within this niche?

Not really.

Given it’s not even ready for release either.

The best you could have hoped for is a GitHub link, but GitHub isn’t being crawled right now.

So I’m not sure what you’re getting at. Your expectations for an alpha level software that wasn’t supposed to be announced is far too high.

Re: SearchHut

#157

Post by richardsocher »

ngshiheng wrote: post #113
richardsocher wrote: post #27

Earlier quoted context omitted.

That's our strategy at you.com - we start with the most popular sites, crawl them and build apps for them, eg. you.com/apps and let users vote in search results. Full disclosure: I'm the CEO.

> you.com/apps and let users vote what is stopping companies/users from abusing/gaming on this system with bots?

Richard, CEO at you.com here.

Users vote on their own preferences... not on other users' preferences. so i dont see how it will be gamed unless we start incorporating user preferences into global ones - which we would only do if we thought it worked better and isn't being gamed.

Re: SearchHut

#158

Post by richardsocher »

Xeoncross wrote: post #122
Bromeo wrote: post #117

The site has just been taken offline by Drew due to the unfortunate start. I hope we can come back to this once the project has been properly launched, although Drew notes that he is "really unhappy with how the roll-out went" and that "my motivation for this project has evaporated" [1]. Thanks for all the work Drew, I hope you guys manage to come to a conclusion that you are satisfied with! [1]: https://paste.sr.ht/…

Drew has the right to cancel his projects, but I really hope others don't cling to the hopes of "the perfect rollout" with their projects. Startups and side projects are messy and sometimes things don't go as planned. Contracts get canceled, DoS takes down your homepage when you launch losing all those free leads, people leak new features and your sixth deployment erases most of the production database. There are a l…

I love your post. I have to say that my experience is that many in the HN community are not nearly as kind as you. Folks here will rip you apart if you don't have everything figured out :(

So, I started rolling out new features of our search engine on Twitter, Slack, etc instead of here.

Re: SearchHut

#159

Post by richardsocher »

egberts1 wrote: post #131

Never Give Up This requires less paying attention to negative emotion and more “water off my back”. Tweak it. tweak it some more. Focus on the goal, notably one tiny sub-goal at a time. Good luck, entrepreneurial spirit is a tough beast to attain. Whatever you stake on, NEVER EVER GIVE UP

Word!

Re: SearchHut

#160

Post by 93po »

reachableceo wrote: post #153
93po wrote: post #140

Earlier quoted context omitted.

It's not passive aggressive. It's sensitive, but he has a right to be if he wants to. He wasn't petty or mean spirited in his announcement to take it down. He only expressed that he was taking the feedback very hard, which is understandable if you had big plans to roll out and make a good first impression.

Meh. Hacker news is the place to get actual real feedback. Frying pan to fire etc. Develop a thick skin or don’t read the comments lol! He chose to disclose it to a few people. Word spreads. That’s what happens. Execute NDAs and have a security program if you don’t want stuff getting out.

Expecting people to have to use NDAs to not spread your hobby tech project is pretty much the antithesis of the "hacker ethos" that this website is literally named after
Post reply on HN