Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SearchHut

searchhut.org

Post reply on HN
Print view View original

121–130 of 161 posts

Re: SearchHut

#121

Post by kordlessagain »

heywire wrote: post #112

Has anyone experimented with creating a search engine that only indexes the landing page of domains? I’m less interested in another Google, and more interested in a way to find new and interesting sites/blogs/etc. Stumbleupon was great for this back in the day. Seems like it would be an interesting experiment to see what the results would be, indexing only the content / meta tags of “index.html”.

I built a solution at https://mitta.us/ that lets you submit the sites you want crawled, and puts them in a self-managed index (which isn't shared globally). I don't do link extraction, but instead let GPT-3 generate URLs based off keywords.

!url |synthesize

I also wrote a screenshot extension for Chrome that lets you save a page when you find it interesting. The site is definitely not "done" but it's usable if you want to try it. Some info in help and in commands is inaccurate/broken, so it is what it is for now.

It does the !google and !ddg thing to find pages to save to the index. There are a bunch of other commands I added, and there's an ability for others to write commands and submit them to a Github repo: https://github.com/kordless/mitta-community

!xkcd was fun to write. It shows comics. The rest of the commands can be viewed from !help or just !

I've been working on pivoting the site to do prompt management for GPT-3 developers and have been kicking around Open Sourcing the other version for use as a personal search engine for bookmarked pages.

Re: SearchHut

#122

Post by Xeoncross »

Bromeo wrote: post #117

The site has just been taken offline by Drew due to the unfortunate start. I hope we can come back to this once the project has been properly launched, although Drew notes that he is "really unhappy with how the roll-out went" and that "my motivation for this project has evaporated" [1]. Thanks for all the work Drew, I hope you guys manage to come to a conclusion that you are satisfied with! [1]: https://paste.sr.ht/…

Drew has the right to cancel his projects, but I really hope others don't cling to the hopes of "the perfect rollout" with their projects.

Startups and side projects are messy and sometimes things don't go as planned. Contracts get canceled, DoS takes down your homepage when you launch losing all those free leads, people leak new features and your sixth deployment erases most of the production database.

There are a lot of great ideas that start out as bad as the first release of thefacebook, AirBnB, Twitch and Youtube. Still, they iterate on these wonky, almost-working sites and end up making something great.

YC pushes this idea constantly; put something in front of people and iterate. Drew was following that advice and I applaud him. https://www.youtube.com/c/ycombinator/videos

Re: SearchHut

#123

Post by marginalia_nu »

BlackLotus89 wrote: post #96

I wonder how the page ranking will work in the end. A quick look at the source doesn't show (me!) any planning for intelligent ranking. The database has a last_index_date and an authoritative field. Could be used fot basic relevance sorting, but nothing exhaustive. Postgres as backend is maybe not the best choice and there are already many sites that index specific pages and take suggestions. The hard part is getting…

As I understand, the idea is to only have manually curated high quality domains. In that regard, ranking is entirely secondary to BM-25. Might work, but it leaves out a lot of long tail sites that (in my experience at least) often have very good results. It's really the middle segment where most of the shit is.

Re: SearchHut

#124

Post by tgv »

tsujp wrote: post #52

Hey everyone I accidentally shared this too early. I misinterpreted a don't-share-yet announcement to mean the announcement post only and not the entire software and announcement. I don't mean this as an excuse; that's just the context. So this is out before Drew et. al. intended it be hence some 404s and so forth as commented by Drew in this very thread here: https://news.ycombinator.com/item?id=32105407 I let my ex…

It's well communicated, so no harm done. And it gives an idea what kind of "curiosity hit" you can expect when announcing it for real.

Re: SearchHut

#125

Post by marginalia_nu »

Xeoncross wrote: post #122
Bromeo wrote: post #117

The site has just been taken offline by Drew due to the unfortunate start. I hope we can come back to this once the project has been properly launched, although Drew notes that he is "really unhappy with how the roll-out went" and that "my motivation for this project has evaporated" [1]. Thanks for all the work Drew, I hope you guys manage to come to a conclusion that you are satisfied with! [1]: https://paste.sr.ht/…

Drew has the right to cancel his projects, but I really hope others don't cling to the hopes of "the perfect rollout" with their projects. Startups and side projects are messy and sometimes things don't go as planned. Contracts get canceled, DoS takes down your homepage when you launch losing all those free leads, people leak new features and your sixth deployment erases most of the production database. There are a l…

The amount of tweaking needed to make a search engine work well can't be overstated either. When you start out, it's inevitably going to be kinda shit. That's fine. Now you need to draw the rest of the owl.

Re: SearchHut

#126

Post by agileAlligator »

richardsocher wrote: post #27
p-e-w wrote: post #13

I like the idea of searching a curated list of domains, but I'm not sure that doing the curation yourself is the best approach considering the huge number of useful but niche websites in existence. I wonder if simply parsing all of Wikipedia (dumps are available, and so are parsers capable of handling them) and building a list of all domains used in external links would do the trick. Wikipedia already has substantial…

That's our strategy at you.com - we start with the most popular sites, crawl them and build apps for them, eg. you.com/apps and let users vote in search results. Full disclosure: I'm the CEO.

Your search actually performed better than Google for me on a random query. I queried both engines "What is the specific heat of alcohol?", Google threw up a rich search answer that linked to some random Mexican site that is clearly exploiting SEO [1], you.com linked me to answers.com (which is more trustworthy than random mexican website).

[1]: http://elempresario.mx/sites/default/files/scith/specific-he...

Re: SearchHut

#127

Post by Xeoncross »

marginalia_nu wrote: post #125
Xeoncross wrote: post #122

Earlier quoted context omitted.

Drew has the right to cancel his projects, but I really hope others don't cling to the hopes of "the perfect rollout" with their projects. Startups and side projects are messy and sometimes things don't go as planned. Contracts get canceled, DoS takes down your homepage when you launch losing all those free leads, people leak new features and your sixth deployment erases most of the production database. There are a l…

The amount of tweaking needed to make a search engine work well can't be overstated either. When you start out, it's inevitably going to be kinda shit. That's fine. Now you need to draw the rest of the owl.

Yeah, I agree. I was certainly underwhelmed with my first small search engine. It was so bad even I didn't want to use it - and I had spent months and months on it.

Still, most projects aren't a search engine. I see people put high expectations on how things will go and often it's just really hard to realize some of those hopes.

Sometimes you just have to take what you can get and iterate. Don't give up.

Re: SearchHut

#128

Post by marginalia_nu »

Xeoncross wrote: post #127
marginalia_nu wrote: post #125

Earlier quoted context omitted.

The amount of tweaking needed to make a search engine work well can't be overstated either. When you start out, it's inevitably going to be kinda shit. That's fine. Now you need to draw the rest of the owl.

Yeah, I agree. I was certainly underwhelmed with my first small search engine. It was so bad even I didn't want to use it - and I had spent months and months on it. Still, most projects aren't a search engine. I see people put high expectations on how things will go and often it's just really hard to realize some of those hopes. Sometimes you just have to take what you can get and iterate. Don't give up.

I think, with search engines, it's best to work with them for the problem domain. It is a fractal of interesting programming problems touching upon pretty much every area of CS, programming and programming-adjacent topics, and take whatever comes out of it as an unexpected bonus.

Re: SearchHut

#129

Post by groffee »

p-e-w wrote: post #13

I like the idea of searching a curated list of domains, but I'm not sure that doing the curation yourself is the best approach considering the huge number of useful but niche websites in existence. I wonder if simply parsing all of Wikipedia (dumps are available, and so are parsers capable of handling them) and building a list of all domains used in external links would do the trick. Wikipedia already has substantial…

The problem with only using curated lists is that you kill discoverability of new sites, but it does have promise like we've seen with Braves 'goggles'.
Post reply on HN