Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mwmbl: Free, open-source and non-profit search engine

mwmbl.org

Post reply on HN
Print view View original

91–100 of 129 posts

Re: Mwmbl: Free, open-source and non-profit search engine

#91

Post by daoudc »

marc_abonce wrote: post #17
mdaniel wrote: post #15

I wondered if this approach would be feasible for a distributed crawler: https://github.com/mwmbl/mwmbl#crawling Also, your own posting appears to be missing from the index: https://mwmbl.org/?q=mwmbl+ycombinator (and, yes, another vote for changing the domain name; you can have a quirky project name, but if I can't remember the cat-walking-on-keyboard domain, I'm not going to use it)

> We now have a distributed crawler that runs on our volunteers' machines! If you have Firefox you can help out by installing our extension. This is a very interesting idea that other search engines have tried before. Actually, the Brave search engine is built over Cliqz[6] that implemented this same idea but *without* the user's consent. Copy pasting from an old comment I made about this "human web" crawler idea: Bo…

Thanks, I didn't know this history! We don't use any user data when crawling, just bandwidth and compute. We tell the extension what to crawl.

Re: Mwmbl: Free, open-source and non-profit search engine

#92

Post by daoudc »

xenodium wrote: post #32

If keen on some minor feedback (specially for mobile), you can likely cut down on landing page text: From: MWMBL [Search on mwmbl...] Welcome to mwmbl, the free, open-source and non-profit search engine. You can start searching by using the search bar above! Find more on [Github] [Wiki] To: MWMBL [Search on mwmbl...] A free, open-source and non-profit search engine. [Github] [Wiki]

Thanks, feel free to send a PR!

Re: Mwmbl: Free, open-source and non-profit search engine

#95

Post by teddyh »

kristopolous wrote: post #33
marginalia_nu wrote: post #30

Earlier quoted context omitted.

YaCy's big problem is that distributed search is a bad idea that will never perform well. Search is as fast as the data is local.

There was an effort in the early 90s to have search as a protocol so you could have a query and then select the domains you want to run it on and return an aggregate result. It was 100% abandoned and I think that's a mistake. It'd be nice to explore some of those ideas again

You’re thinking of WAIS, I believe: https://en.wikipedia.org/w/index.php?title=Wide_area_informa...>

Re: Mwmbl: Free, open-source and non-profit search engine

#96

Post by teddyh »

kristopolous wrote: post #41
marginalia_nu wrote: post #35

Earlier quoted context omitted.

I think a big part of the problem is that domains in isolation don't provide the best search results. Out-of-band information like (global) anchor texts or click data makes search perform so much better. If I want to learn how to do an INNER JOIN in MariaDB, this is the authoritative source: https://mariadb.com/kb/en/join-syntax/ The problem being that INNER JOIN isn't particularly important to that page using most I…

There's more to it than that. What if instead of crawling the php generation of database rows with a bunch of cruft, the administrator published some kind of schema with scraping and querying rules and you could alternatively make a single call to capture all of the data in a sematic schema. You can still do all the stuff you're talking about but it could make search more coherent. An entry for that humans and an ent…

Wasn’t this what the Semantic Web was supposed to enable?

Re: Mwmbl: Free, open-source and non-profit search engine

#97

Post by daoudc »

nonrandomstring wrote: post #42

> This website requires you to support/enable scripts. Bye bye. You do not need "scripts" to turn the text string I'll supply into a list of candidate links. How can you not understand this basic accessibility foundation?

The API is open so feel free to write your own front end that doesn't need js, or send a PR to add support for no js.

Re: Mwmbl: Free, open-source and non-profit search engine

#98

Post by daoudc »

davidebaldini wrote: post #66

If I understand, having only 4096 bytes of data per term causes multiple terms in the same query to intersect to little or no results. The purpose seems to cut cost in compromise of completeness.

Yes, you're correct on the purpose. We mitigate it a little by also indexing on bigrams.

Re: Mwmbl: Free, open-source and non-profit search engine

#99

Post by marginalia_nu »

hk__2 wrote: post #62
kwhitefoot wrote: post #60

A lot of the terms I searched for returned no hits. The Firefox add-on crawls pages linked from Hacker News which is amusing perhaps but seems unlikely to crawl a representative selection of the web. Perhaps the user should be able to suggest pages to be crawled. But when it does find something it is very quick! So I'll give it a go.

Same experience: it’s quick at finding irrelevant links. For some reason, it seems to have indexed a lot of spammy websites: search for "Trastevere" on Google, and you get Wikipedia and pages about the district in Rome. Search it on Mwmbl and you only get links from a random *.it-romehotels.com website. Other random examples: search for "2023" and the very first link is "2023 Pomeroy College Basketball Ratings". Sear…

Even Google is kinda not great for "Trastevere". I'd like to see results like these in favor of the sort of travel industry spam that's 90% of the search results page.

https://www.romeartlover.it/Vasi60.htm https://www.maquettes-historiques.net/P19b.html

Re: Mwmbl: Free, open-source and non-profit search engine

#100

Post by daoudc »

marginalia_nu wrote: post #69
davidebaldini wrote: post #66

If I understand, having only 4096 bytes of data per term causes multiple terms in the same query to intersect to little or no results. The purpose seems to cut cost in compromise of completeness.

Yeah. That seems like a design decision that will scale poorly. For reference, even in my dinky 100M index I have individual terms with several gigabytes of associated document references. In general hash map table index designs don't tend to be very efficient. If you use a skip list or something similar, you can calculate the intersection between sets in sublinear time.

We actually just take the union and then re-rank. Because the lists are all small, this is cheap.
Post reply on HN