Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mwmbl: Free, open-source and non-profit search engine

mwmbl.org

Post reply on HN
Print view View original

71–80 of 129 posts

Re: Mwmbl: Free, open-source and non-profit search engine

#71

Post by Fnoord »

kristopolous wrote: post #34
marginalia_nu wrote: post #29

Earlier quoted context omitted.

I'd love to see more competition in search. Feels like everyone right now gets tripped up on trying to emulate Google, which is a trap even if you succeed. Nobody is going to out-Google Google. ChatGPT's recent huge success in performing a specific tasks previously within the domain of Google by doing something other than they are is a good example of this.

Chatgpt is so much more useful for things that are specific and complex than Google is. Google used to be good at it but it's now utterly befuddled by specificity and returns such garbage that I had given up. But the form of "I'm doing this, I'm seeing this and I'm wondering if X is possible" chatgpt is solid on that - basically a personal stack overflow

Google got completely and utterly raped by world-wide SEO. We'll have to see how ChatGPT ages. Since the dataset is more controlled, I give it a fair chance.

Re: Mwmbl: Free, open-source and non-profit search engine

#72

Post by arghwhat »

marginalia_nu wrote: post #68
arghwhat wrote: post #65

Earlier quoted context omitted.

No it's not, it is an intentional misspelling of "googol" and means nothing - not in English or any other language. "Spotify" is also not a "normal word" in any language. And for those not native in English (there's supposedly only some 400 million of those), it's just a random sound sequence like any other. mwmbl is a shortening of the welsh writing of https://en.wikipedia.org/wiki/Mumbles . Only tricky part is know…

> Only tricky part is knowing that the w is pronounced as a u Is that really tricky? W is basically pronounced like U in English already[1]. It just looks funny when you exchange the two. [1] e.g. say this sentence "uorld uar tuo uas the uorst"

In many other languages, w is pronounced like a v and in some cases even named "double vee".

> e.g. say this sentence "uorld uar tuo uas the uorst"

This doesn't work with the English pronunciations of the letter u from words like "uninteresting" or "mumble". It mostly seems to work with the pronunciation of "you", which does not naturally fit those letter placements.

Not knowing the proper linguistic terms, I'd consider "w" to be a modulation of a another sounding vowel by closing your lips and pressing your tongue a bit down to make room. Without a vowel to modulate, there is no sound, and so "mwmbl" is a bit of a question mark.

But most words require prior knowledge to pronounce correctly, especially in as messy a language as English.

Re: Mwmbl: Free, open-source and non-profit search engine

#73

Post by ddorian43 »

marginalia_nu wrote: post #58
ddorian43 wrote: post #50

Earlier quoted context omitted.

There is common crawl: https://github.com/commoncrawl

Kinda sucks that it's stuck in AWS with no easy way of exfiltrating the data from the Amazon ecosystem. Last I tried I got like 100 Kb/s on their HTTP mirror. At that rate, the download would take 12 years.

I just tried 2 http examples from https://commoncrawl.org/get-started for an old dataset and the most recent one and got 110Mb/s (my full download bandwidth):

  wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz

  wget https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-23/segments/1685224643388.45/warc/CC-MAIN-20230527223515-20230528013515-00483.warc.gz

Re: Mwmbl: Free, open-source and non-profit search engine

#75

Post by zx8080 »

melx wrote: post #55
dmurray wrote: post #49

Earlier quoted context omitted.

Welsh has the unfortunate combination of being unfamiliar to most English speakers, and not exotic enough to score diversity points.

On that topic I love the Welsh-English encounter of civil servants thinking they understood each other[0] [0] http://news.bbc.co.uk/1/hi/7702913.stm

It surprisingly resembles Swedish language a bit.

Re: Mwmbl: Free, open-source and non-profit search engine

#76

Post by Proven »

marc_abonce wrote: post #17
mdaniel wrote: post #15

I wondered if this approach would be feasible for a distributed crawler: https://github.com/mwmbl/mwmbl#crawling Also, your own posting appears to be missing from the index: https://mwmbl.org/?q=mwmbl+ycombinator (and, yes, another vote for changing the domain name; you can have a quirky project name, but if I can't remember the cat-walking-on-keyboard domain, I'm not going to use it)

> We now have a distributed crawler that runs on our volunteers' machines! If you have Firefox you can help out by installing our extension. This is a very interesting idea that other search engines have tried before. Actually, the Brave search engine is built over Cliqz[6] that implemented this same idea but *without* the user's consent. Copy pasting from an old comment I made about this "human web" crawler idea: Bo…

[dead]

Re: Mwmbl: Free, open-source and non-profit search engine

#77

Post by carlsborg »

worksonmine wrote: post #67
carlsborg wrote: post #52

Sub-100 ms search results, nicely typed python codebase, good project. How many 4096 byte pages do you currently store?

Why 4096 bytes?

From the github: "Our design is a giant hash map. We have a single store consisting of a fixed number N of pages. Each page is of a fixed size (currently 4096 bytes to match a page of memory), and consists of a compressed list of items."
Post reply on HN