Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

541–550 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#541

Post by marginalia_nu »

dmje wrote: post #304

Love it. You should provide a link to Patreon / whatever so people can support you financially. Hosting is probably not cheap for you. Given the love here on HN I suspect you'd do well.

Hosting is actually surprisingly cheap, but that's because I'm hosting it on consumer hardware in my living room, off my domestic broadband connection.

That's both a blessing and a curse. It works okay as long as I don't touch it, but I can't do maintenance without shutting it down. I can't implement crawler changes without a week of shitty results as it needs to visit half the Internet before it gets decently good. I can only afford a production machine, so all testing that can't be done with unit tests gets done there.

Anyway I added a patreon in case anyone wants to toss a coin.

Re: A search engine that favors text-heavy sites and punishes modern web design

#542

Post by marginalia_nu »

aetherspawn wrote: post #441

Beware, I got the impression straight away that some sites were censored from the results for no good reason. For example, if you search "jehovahs witnesses", all pages from jw.org are missing. Exactly the same thing happened when I searched "mormons" - the official website is missing and it only brings up sects/hate/conspiracies against mormons.

If you want mormons in positive light, you should search for "latter-day saints", as that's how they typically brand themselves.

jw.org is on the indexing list, just pretty far down, based on the fact that previous times it's been visited it's had ton of javascript.

I don't have an axe to grind with fringe religious movements, I actually love them to bits. Try searching for Nag Hammadi or Hermes Trismegistos.

Re: A search engine that favors text-heavy sites and punishes modern web design

#543

Post by davuinci »

Congrats for the effort, I really like the idea and it works wonderfully for some searches.

However, I searched "infiniband" and the results are far away from what I would expect or like to see. Most of the results that appear first are completely unrelated to the topic.

Re: A search engine that favors text-heavy sites and punishes modern web design

#545

Post by ephbit »

marginalia_nu wrote: post #431
ephbit wrote: post #396

Quoted from the linked site: > Convenience functions have been added, and the search engine can now perform simple calculations and unit conversions. Try 1 pint in cubic centimeters, or 50+sqrt(pi). This functionality is still under development, be patient if it doesn't work. Why would you make any ever so small effort to implement calculations? I don't get it. If your search engine enabled me to find more useful sea…

I implemented calculations because easily 80% of my google queries are calculations, unit conversions, etc. Search functionality is larger priority. Calculations and unit conversions were an afternoon's break from the search functionality :-)

Ok, I guess people use google (or search engines in general) differently ... I rarely if ever use a search enginge to calculate stuff or do unit conversions.

I use google only to search and when ecosia/bing doesn't return anything useful.

Re: A search engine that favors text-heavy sites and punishes modern web design

#546

Post by Method5440 »

A big fan of your work! Just wanted to let you know what iOS devices provide quotes as “ rather than " - you may need to support the character “ or at least let people know that iOS is not supported etc… right now I get a generic character error.

Re: A search engine that favors text-heavy sites and punishes modern web design

#547

Post by mananaysiempre »

leephillips wrote: post #214
marginalia_nu wrote: post #208

Earlier quoted context omitted.

The search engine doesn't do any type of re-ordering or synonym stuff, it only tires to construct different N-grams from the search query. So if you for example compare "SDL tutorial" with "SDL tutorials". On google you'd get the same stuff, this search engine, for better or worse doesn't. This is a design decision, for now anyway, mostly because I'm incredibly annoyed when algorithms are second-guessing me. On the o…

I like this design decision. It pays you back for choosing your search terms carefully.

I’m not against a stemmer, actually, just against the aggressive concordances (?) that Google now employs, like when it shows me X in Banach spaces (the classical, textbook case) when I’m specifically searching for X in Fréchet spaces (the generalization I want to find but am not sure exists); of course Banach spaces and Fréchet spaces are almost exclusively encountered in the same context, but it doesn’t mean that one is a popular typo for the other! (The relative rarity of both of these in the corpus probably doesn’t help. The farcical case is BRST, or Becchi-Rouet-Stora-Tyutin, in physics, as it is literally a single key away from “best” and thus almost impossible to search for.)

On the other hand, Google’s unawareness of (extensive and ubiquitous) Russian noun morphology is essentially what allowed Yandex to exist: both 2011 Yandex and 2021 Google are much more helpful for Russian than 2011 Google. I suspect (but have not checked) that the engine under discussion is utterly unusable for it. English (along with other Germanic and Romance languages to a lesser extent) is quite unusual in being meaningfully searchable without any understanding of morphology, globally speaking.

Re: A search engine that favors text-heavy sites and punishes modern web design

#549

Post by phendrenad2 »

titzer wrote: post #475
snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

Search engines whose revenue is based on advertising will ultimately be tuned to steer you to the ad foodchain. All the incentives are aligned towards and all the metrics ultimately in service of, profit for advertisers. Not in the 99% of people who can convinced to consume something by ads? Welp, screw you.

Search engines should be something you pay for. Surely search engine powerusers can afford to pay for such a service. If Google makes $1 per user per month or something, that's not too high a bar to get over.

Re: A search engine that favors text-heavy sites and punishes modern web design

#550

Post by phendrenad2 »

hdjjhhvvhga wrote: post #132
snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

As long as few people use it, it will be great. Rest assured that the moment it becomes popular, the people who want to game it will appear.

There should be some perfect balance where this search engine is N% as popular as Google, where Google soaks up all of the gamifiers, but this search engine is still popular enough to derive revenue and do ML and other search-engine-useful stuff.
Post reply on HN