I don't think this is a good idea: when I'm searching on the web I want to get results with a high relevance to my search query. I don't care a lot about the presentation.
A search engine that favors text-heavy sites and punishes modern web design
691–700 of 735 posts
Re: A search engine that favors text-heavy sites and punishes modern web design
#692So is this a filter on top of Google or is it search from scratch? Would love to understand more of the implementation.
It's built from scratch. I'm doing the crawling and indexing. Look through my comments and you'll find a few outlines of the stack and the index design. Here are my blog entries relating to this: https://memex.marginalia.nu/topic/astrolabe.gmi
This isn't meant to be pressurizing or to sound like a demand (if it seems that way), but have you thought about uploading the source code for your search engine?
Something like this has the potential to be used in university courses to teach how to build a search engine and/or teach 'advanced' programming concepts and ideas. It's a real program showing what you need to do to optimize your database and software to work on consumer desktops (even if your specs are higher than what other people would have; 128 GiB for example is quite a lot of RAM for most consumers) and how to handle malicious data that you will come across (for example, link farms).
In addition, I read all the posts on your site that were listed in the page you linked to, and to me those posts would actually seem more useful as an explanation of the code that people can view together side-by-side, rather than as the only way people can know how you implemented your algorithms and search engine. I guess what I'm saying is, having an explanation in words of the algorithms and code along with the actual code can be a very powerful combination for teaching and learning.
Thus, again, would it be alright if you upload a copy of the source code for people (including myself) to look at? I personally don't care about if it's released under an open-source license (or not), or if you just add a zip file on your site vs making a repository on Github, or even if you never update the code you release. I (and most likely others) want to peek at at least one version of what you wrote to see how something like this works under-the-hood, which again, I'm asking if that's alright with you.
Also, I'm not asking you to share the database(s) you have for this, especially since they're giant and would likely take up more traffic downloading from your site than anything the search engine can do.
Re: A search engine that favors text-heavy sites and punishes modern web design
#693Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)
This is awesome. I've been looking for a long time for a search engine that basically takes everything Google does and does the opposite. Thank you for doing this, I will definitely be bookmarking it. Is there a way to suggest or add sites? I went looking for woodgears.ca and only got one result. I also think my personal blog would be a good candidate for being indexed here but I couldn't find any results for it.
I would also love to add some sites which I were missing...
Unfortunately this doesn't seem to be a feature which new search engines are focusing on - Brave Search also misses that feature...
Re: A search engine that favors text-heavy sites and punishes modern web design
#694Earlier quoted context omitted.
This is exactly how I prefer to use my search engines.
I searched like this all my life and always got expected results. But just a week ago I found out that these "how", "what" questions give better and faster results on Google.
Try your old Google Fu skills on DuckDuckGo (or Bing I guess). I've found it to have good results anyway
Re: A search engine that favors text-heavy sites and punishes modern web design
#695Earlier quoted context omitted.
It's very rare that I see a project on HN I can see myself using. This is one. Like others have said, the results can be a little rough. But they're rough in a way I think is much more manageable than the idiosynchrosies of more 'clever' search engines.
I think you need to approach it more like grep than google. It's a forgotten art, dealing with this type of dumb search engine. Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.
> I think you need to approach it more like grep than google. It's a forgotten art
A search engine that accepted regex as the search parameter would be amazing.
I actually used this method as a field filter for a bunch of simple internal tools to search for info. Originally people were asking for individual search capabilities, but I didn't want it to become a giant project with me as the implementor of everyone's unique search capability feature request - so I just gave them regex, encoded inputs into the URL query string so they can save searches - gave em a bunch of examples to get going and now people are slowly learning regex and coming up with their own "new features" :P
But this made sense because it's a relatively small amount of data, so small that it's searched in the front end which is why it's more of a filter... I don't think pure regex would scale when used as a query on a massive DB, it would need some kind of hierachy still to only bother parsing a subset of relevant text... unless there is some clever functional regex caching algorithm that can be used.
Re: A search engine that favors text-heavy sites and punishes modern web design
#696A big fan of your work! Just wanted to let you know what iOS devices provide quotes as “ rather than " - you may need to support the character “ or at least let people know that iOS is not supported etc… right now I get a generic character error.
That's very good to know. Thanks!
Re: A search engine that favors text-heavy sites and punishes modern web design
#697I like it. Coincidentally, the other day I was daydreaming about a search engine that favors sites that are updated less frequently. The thought being, the kinds of labors of love that characterized the 1990s Web that I still sometimes miss are still out there, it's just harder to find them amidst the flood of SEO dreck. So perhaps they could be made discoverable again with the help of a contrarian search engine that…
Million Short [1] offers an option to omit results from popular domains. It's a different approach from what you describe, but I think the goal is similar. [1] https://millionshort.com/
I think I like this method the most, honestly. There's something to be said for minimalist web design, but there's also something to be said for everything we've learned since the 90s.
Re: A search engine that favors text-heavy sites and punishes modern web design
#698Earlier quoted context omitted.
Well just crunching the numbers should indicate what is possible and what isn't. For the moment I have just south of 20 million URLs indexed. 1 x 20 million bytes = 20 Mb. 10 x 20 million bytes = 200 Mb. 100 x 20 million bytes = 2 Gb. 1,000 x 20 million bytes = 20 Gb. 10,000 x 20 million bytes = 200 Gb. 100,000 x 20 million bytes = 2 Tb. 1,000,000 x 20 million bytes = 20 Tb. This is still within what consumer hardwar…
What crawler are you using and what kind of crawling speeds are you achieving? How do you rank the results (is it based on content only) or you have external factors too? What is your personal preferred search option of the 7 and why? Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
> What crawler are you using and what kind of crawling speeds are you achieving?
Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper.
>How do you rank the results (is it based on content only) or you have external factors too?
I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query.
> What is your personal preferred search option of the 7 and why?
I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement.
> Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible.
And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing.
I'm honestly a bit at a loss for words.
Re: A search engine that favors text-heavy sites and punishes modern web design
#699Earlier quoted context omitted.
I like this design decision. It pays you back for choosing your search terms carefully.
I’m not against a stemmer, actually, just against the aggressive concordances (?) that Google now employs, like when it shows me X in Banach spaces (the classical, textbook case) when I’m specifically searching for X in Fréchet spaces (the generalization I want to find but am not sure exists); of course Banach spaces and Fréchet spaces are almost exclusively encountered in the same context, but it doesn’t mean that o…
> BRST, or Becchi-Rouet-Stora-Tyutin is literally a single key away from “best” and thus almost impossible to search for.
Hmm I seem to be getting only relevant results, no "best", not sure what you mean. Are you not doing verbatim search?
Re: A search engine that favors text-heavy sites and punishes modern web design
#700Earlier quoted context omitted.
It's built from scratch. I'm doing the crawling and indexing. Look through my comments and you'll find a few outlines of the stack and the index design. Here are my blog entries relating to this: https://memex.marginalia.nu/topic/astrolabe.gmi
This isn't meant to be pressurizing or to sound like a demand (if it seems that way), but have you thought about uploading the source code for your search engine? Something like this has the potential to be used in university courses to teach how to build a search engine and/or teach 'advanced' programming concepts and ideas. It's a real program showing what you need to do to optimize your database and software to wo…
I'm thinking I may open source some of the components I use, rather than publishing the whole thing, as it's part of a larger monorepo that contains the somewhat integrated code for a large set of services, public and private.
None of the code is particularly fancy, just highly specialized. I did build them myself mostly because I couldn't find anything available that the rather special demands that are put on the application.