Indexing a billion pages • Hacker News Board

Re: Indexing a billion pages

#31

Post by KomoD » Sun, Dec 24, 2023, 5:18 PM UTC

Alifatisk wrote: post #30

What's the consequence of installing the crawler to FF? Can the ISP / Cloudflare / any other party start blacklisting you?

> Can the ISP / Cloudflare / any other party start blacklisting you?

Yes, wouldn't surprise me if you were to get blocked from tons of sites and flagged as a bot on Cloudflare and other WAFs.

Re: Indexing a billion pages

#32

Post by hinkley » Sun, Dec 24, 2023, 7:18 PM UTC

Alifatisk wrote: post #30

What's the consequence of installing the crawler to FF? Can the ISP / Cloudflare / any other party start blacklisting you?

Well, the internet could know the balance of your bank account at that boutique credit union, or which sex toys you’ve been looking at buying.

We do web hosting for businesses. We are careful to identify bots because we don’t want personalized recommendations in search engines, and don’t want PII showing up. Not that we put a lot of that in the site, but think about a Home Depot or a Best Buy situation. If you’re a human near the Minneapolis store, you want availability for that store, not the Detroit one. But corporate wants a bot to know that this microwave is something you in theory carry, but might have to ship from a warehouse. If Home Depot wants that, and our customers also want that, then probably a lot of businesses feel the same.

Re: Indexing a billion pages

#33

Post by urbandw311er » Mon, Dec 25, 2023, 12:41 AM UTC

I think the saddest part of this is that, owing to the total enshittification of the web due to SEO, at least 50% of what they index will be absolute garbage.

Re: Indexing a billion pages

#34

Post by KomoD » Mon, Dec 25, 2023, 1:42 PM UTC

urbandw311er wrote: post #33

I think the saddest part of this is that, owing to the total enshittification of the web due to SEO, at least 50% of what they index will be absolute garbage.

Higher, run the extension and look at the feed, it's like 90% garbage

Re: Indexing a billion pages

#35

Post by Alifatisk » Mon, Dec 25, 2023, 6:53 PM UTC

How do I identify my hash among the users in the stats https://mwmbl.org/stats ?

Re: Indexing a billion pages

#36

Post by _w1tm » Tue, Dec 26, 2023, 11:26 AM UTC

bdcravens wrote: post #13

Most impressive part: > Our estimated annual budget is $752.36 and we have spent $174.49.

If you value engineering time at zero you can spend months / years / decades / eons optimizing your solution for cost.

Re: Indexing a billion pages

#37

Post by Alifatisk » Tue, Dec 26, 2023, 11:58 AM UTC

KomoD wrote: post #31

Alifatisk wrote: post #30

What's the consequence of installing the crawler to FF? Can the ISP / Cloudflare / any other party start blacklisting you?

> Can the ISP / Cloudflare / any other party start blacklisting you? Yes, wouldn't surprise me if you were to get blocked from tons of sites and flagged as a bot on Cloudflare and other WAFs.

Yeah getting blocked from Cloudflares network is not something I want to experience, it's a hell.

Re: Indexing a billion pages

#38

Post by Alifatisk » Wed, Dec 27, 2023, 12:39 PM UTC

hinkley wrote: post #32

Alifatisk wrote: post #30

What's the consequence of installing the crawler to FF? Can the ISP / Cloudflare / any other party start blacklisting you?

Well, the internet could know the balance of your bank account at that boutique credit union, or which sex toys you’ve been looking at buying. We do web hosting for businesses. We are careful to identify bots because we don’t want personalized recommendations in search engines, and don’t want PII showing up. Not that we put a lot of that in the site, but think about a Home Depot or a Best Buy situation. If you’re a h…

But the extension is not crawling my stuff, it takes batches of urls from the host server. How is the internet going to get my bank account?