Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

431–440 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#431

Post by marginalia_nu »

ephbit wrote: post #396

Quoted from the linked site: > Convenience functions have been added, and the search engine can now perform simple calculations and unit conversions. Try 1 pint in cubic centimeters, or 50+sqrt(pi). This functionality is still under development, be patient if it doesn't work. Why would you make any ever so small effort to implement calculations? I don't get it. If your search engine enabled me to find more useful sea…

I implemented calculations because easily 80% of my google queries are calculations, unit conversions, etc.

Search functionality is larger priority. Calculations and unit conversions were an afternoon's break from the search functionality :-)

Re: A search engine that favors text-heavy sites and punishes modern web design

#434

Post by marginalia_nu »

mountain_peak wrote: post #421
marginalia_nu wrote: post #182

Earlier quoted context omitted.

The search engine doesn't actually do full text search, so maybe your query was too... unique. But do first of all verify that you haven't been hacked. There's about quarter of a million domains I've flagged that, besides their wordpress content, also host a ton of link spam crap off in some hidden folder. This reflects on the quality rating extremely negatively to the point where you may have not been indexed at all…

Thanks for the advice; not hacked, but I have "resurrected" many WP sites that have been (including my wife's non-profit). Just running on an EC2 micro instance, but I tried adding "site:" and received "No such domain". Actually, I think it's because I haven't enabled "HTTPS" yet! That's on my to-do along with migrating off EC2-Classic to VPC...

Vanilla HTTP should be fine. I think 80% of the urls are HTTP.

If you're getting no such domain, it's either blocked because it looks too much like a spam domain, or it simply hasn't been discovered yet.

What's the TLD? I severely restrict some cheaper TLDs because they gave so much spam.

For example, cr.yp.to is an example of a baby I know I've definitely thrown out with the bathwater.

Re: A search engine that favors text-heavy sites and punishes modern web design

#435

Post by marginalia_nu »

omoikane wrote: post #233
marginalia_nu wrote: post #8

Earlier quoted context omitted.

I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort). You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7…

> It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest When I search for Japanese terms, it "says needs to be a word", which wasn't the best error message. Maybe the error message should say something like "sorry, your language isn't support yet"?

I've rephrased the wording for that one a bit.

Re: A search engine that favors text-heavy sites and punishes modern web design

#437

Post by marginalia_nu »

crocodiletears wrote: post #265
marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

It's very rare that I see a project on HN I can see myself using. This is one. Like others have said, the results can be a little rough. But they're rough in a way I think is much more manageable than the idiosynchrosies of more 'clever' search engines.

I think you need to approach it more like grep than google. It's a forgotten art, dealing with this type of dumb search engine.

Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.

Re: A search engine that favors text-heavy sites and punishes modern web design

#438

Post by PaulHoule »

random314 wrote: post #417
vikingerik wrote: post #346

Earlier quoted context omitted.

Right. Arrow's theorem just says it's impossible to do it in all cases. It's still quite possible to get an improvement in a large proportion of cases, as you're proposing.

Arrows theorem simply doesn't apply here. We don't need our personalized search results to satisfy the majority.

But in both cases you face the problem of aggregating preferences of many into one. In one case you are combining personal preferences in the other case aggregating ‘preferences’ expressed by search engines.

Re: A search engine that favors text-heavy sites and punishes modern web design

#439

Post by soheil »

marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

Awesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so many websites for a project this size? Did you use any APIs like duck duck go or data from other search engines? Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?

Re: A search engine that favors text-heavy sites and punishes modern web design

#440

Post by Aeolun »

jandrese wrote: post #160
hdjjhhvvhga wrote: post #132

Earlier quoted context omitted.

As long as few people use it, it will be great. Rest assured that the moment it becomes popular, the people who want to game it will appear.

This sort of optimization is why simple recipes are typically found at the end of a rambling pointless blog post now. Still, the best way to break SEO is to have actual competition in the search space. As long as SEO remains focused on Google there is an opportunity for these companies to thrive by evading SEO braindamage.

Searching for ‘chocolate’ on this search engine turned up a surprisingly large amount of chocolate based recipes.
Post reply on HN