Do they use Common Crawl?
Indexing a billion pages
21–30 of 38 posts
Re: Indexing a billion pages
#22> We’ve indexed over 100 million pages > [W]e’re crawling up to a million pages a day, as you can see on our stats page. > Given that Mwmbl is still relatively unknown, it seems plausible that we can reach our target of crawling three billion pages a day, to refresh the entire index in one month. I think this is supposed to read "it seems plausible that we can reach our target of crawling three million pages a day."
> to refresh the entire index [of 1 billion] in one month. Seems you are correct.
Except then it needs to be 30M pages/day. 3M * 30 days just nets you 90M
Re: Indexing a billion pages
#23I remember reading about a project who’s sole purpose is to provide a large index of the open web for free, anyone could download it. Forgot the name of the project.
Why can’t mwmbl download their index?
Also, is mwmbl planning on providing their crawled index for free? Like, can I also download it later?
If that is the case, I’s happily download their FF extension.
Re: Indexing a billion pages
#24I remember reading about a project who’s sole purpose is to provide a large index of the open web for free, anyone could download it. Forgot the name of the project. Why can’t mwmbl download their index? Also, is mwmbl planning on providing their crawled index for free? Like, can I also download it later? If that is the case, I’s happily download their FF extension.
What you’re likely referring to is Common Crawl: https://commoncrawl.org
Re: Indexing a billion pages
#25I remember reading about a project who’s sole purpose is to provide a large index of the open web for free, anyone could download it. Forgot the name of the project. Why can’t mwmbl download their index? Also, is mwmbl planning on providing their crawled index for free? Like, can I also download it later? If that is the case, I’s happily download their FF extension.
I installed the extension because why not, then I noticed it was only crawling spam pages and redirect links that were being abused for spam... I guess it's kind of expected but not sure how I feel about it
Though, it's still a very cool idea, maybe an option to crawl sites I visit would be nice?
Re: Indexing a billion pages
#26I remember reading about a project who’s sole purpose is to provide a large index of the open web for free, anyone could download it. Forgot the name of the project. Why can’t mwmbl download their index? Also, is mwmbl planning on providing their crawled index for free? Like, can I also download it later? If that is the case, I’s happily download their FF extension.
I installed the extension because why not, then I noticed it was only crawling spam pages and redirect links that were being abused for spam... I guess it's kind of expected but not sure how I feel about it Though, it's still a very cool idea, maybe an option to crawl sites I visit would be nice?
Oh, so the extension is not crawling the website you visit but crawls the open web in the background?
How much cpu / ram does it consume?
Re: Indexing a billion pages
#27I remember reading about a project who’s sole purpose is to provide a large index of the open web for free, anyone could download it. Forgot the name of the project. Why can’t mwmbl download their index? Also, is mwmbl planning on providing their crawled index for free? Like, can I also download it later? If that is the case, I’s happily download their FF extension.
What you’re likely referring to is Common Crawl: https://commoncrawl.org
Probably, but I don’t remember the website looking like that, it had more colors to it, like red yellow blue etc.
Update: Yes, it's commoncrawl but with updated design. https://archive.is/4dz8G
Re: Indexing a billion pages
#28Earlier quoted context omitted.
I installed the extension because why not, then I noticed it was only crawling spam pages and redirect links that were being abused for spam... I guess it's kind of expected but not sure how I feel about it Though, it's still a very cool idea, maybe an option to crawl sites I visit would be nice?
Oh, so the extension is not crawling the website you visit but crawls the open web in the background? How much cpu / ram does it consume?
> Oh, so the extension is not crawling the website you visit but crawls the open web in the background?
Yeah, it gets batches of links from their API and crawls them
Re: Indexing a billion pages
#29I remember reading about a project who’s sole purpose is to provide a large index of the open web for free, anyone could download it. Forgot the name of the project. Why can’t mwmbl download their index? Also, is mwmbl planning on providing their crawled index for free? Like, can I also download it later? If that is the case, I’s happily download their FF extension.
I installed the extension because why not, then I noticed it was only crawling spam pages and redirect links that were being abused for spam... I guess it's kind of expected but not sure how I feel about it Though, it's still a very cool idea, maybe an option to crawl sites I visit would be nice?
> Though, it's still a very cool idea, maybe an option to crawl sites I visit would be nice?
I had an extension installed for a while that submitted pages to the Internet Archive automatically, but it was a constant battle to remember to denylist any sites that were personal (bank, doctor, whatever) before visiting them. That was before I was heavy into the Firefox containers setup, so if I were to try that again I'd try to find a way to disable it for those containers (which, come to think of it, may be yet another container feature request)
Having thought a little further about your suggestion, I could imagine an extension that merely submitted the window.location.origin to the search engine and let it index the site, as a heuristic for "this site is popular enough to have received a visit in the past hour/day/whatever" but with Mwmbl specifically that'd put things back in a loop since it would send the site back to your browser to index it for them
I sure do hope Mwmbl's extension is not using the full browser context to make requests, otherwise any request to index mail.google.com would be no bueno
Re: Indexing a billion pages
#30What's the consequence of installing the crawler to FF? Can the ISP / Cloudflare / any other party start blacklisting you?