Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

At what point is the internet too big for Google to index it?

news.ycombinator.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: At what point is the internet too big for Google to index it?

#11

Post by qeternity »

Havoc wrote: post #8

I don't think google's recent troubles are a result of index size. Pretty sure G could throw more money and resources at it if they thought that would make a dent. It feels more like a lot of real content is collateral damage to the SEO vs google wars. e.g. A blogger was complaining the other day that someone had set up an automation to automatically scrape their content the second it gets published, run it through t…

It’s only going to get worse with the proliferation of NLP nets a la GPT.

Will be interesting to see a decade from now how researchers collect a corpus that isn’t chock full of a model’s own output

Re: At what point is the internet too big for Google to index it?

#13

Post by metadat »

marginalia_nu wrote: post #10

Google already doesn't index the entire Internet. The internet even having a size becomes more questionable the more you think about it. Let's say we set up a wildcard domain *.example.com all pointing to a server set up so that 0.example.com/ has a link to 0.example.com/0 and 1.example.com/ 0.example.com/0 has a link to 0.example.com/1 and 0.example.com/0/0 0.example.com/0/0 has a link to 0.example.com/0/1 and 0.exa…

While you are technically correct, these sorts of tarpit cycle patterns should be detected and shed by any half-decent crawling system ("half-decent" is said tongue-in-cheek :p).

Does marginalia struggle or fail to identify and sidestep indexing for these sorts of structures?

Re: At what point is the internet too big for Google to index it?

#14

Post by marginalia_nu »

metadat wrote: post #13
marginalia_nu wrote: post #10

Google already doesn't index the entire Internet. The internet even having a size becomes more questionable the more you think about it. Let's say we set up a wildcard domain *.example.com all pointing to a server set up so that 0.example.com/ has a link to 0.example.com/0 and 1.example.com/ 0.example.com/0 has a link to 0.example.com/1 and 0.example.com/0/0 0.example.com/0/0 has a link to 0.example.com/0/1 and 0.exa…

While you are technically correct, these sorts of tarpit cycle patterns should be detected and shed by any half-decent crawling system ("half-decent" is said tongue-in-cheek :p). Does marginalia struggle or fail to identify and sidestep indexing for these sorts of structures?

I don't mean to suppose these structures are actually common, I'm using them to illustrate how the concept of the internet having a size falls apart when you start considering the practical matters of counting how big the internet is. It hinges on a model of files-on-a-server that isn't how websites have worked in the last 30-or-so years.

Counting websites, or even delineating where a website starts or ends is difficult, as you can on the one hand have a single server hosting infinite websites like I described. Services like cloudflare also throw a spanner in the works, if you think maybe using server IP would help. Domain name isn't much use either, as that would discount hosting services like neocities.

There's a similar fractal of weird cases with counting documents on a given webserver (and by the extension the internet).

Re: At what point is the internet too big for Google to index it?

#15

Post by throwaway532532 »

Google for too small for the internet around the time if their first penguin + panda updates.. They killed off a lot of good sites in the war against SEO.

They have also reduced the image search, torrents, and many other things have been removed via the censoring if the index and YouTube..

Pushing corporate type sites up and other things into the nether , Google has become the new yellow pages along with being an arbiter or higher ranked health info and links to old answers for programmers.

Basic things like recipes are so bad even non Oliver jokes about reading a dozen paragraphs before finding a recipe via Google.

So many things fun / entertainment/ sexy and more have no room for the high brow expectations of the big G.

Hence TikTok being more popular than Google now.

Some of the things they have removed have come from govs and industries with a lot of sway.. but much of what they downrank post panda penguin is a vieled attempt at being more politically correct and less blue collar.

Imo.

So indeed there is now room for other 'searxh/find' portals for things Google does t want to showcase on their front pages..

But for at least some while into the future they will likely be the best yellow pages since customers do most of that work for them.

Re: At what point is the internet too big for Google to index it?

#16

Post by fuzzfactor »

bediger4000 wrote: post #12

I thought google was deliberately "forgetting" older material, to make room for new. The "long tail" turned out to be hogwash, and advertising corrupted everything.

>At what point is the internet too big for Google to index it?

That was the question back when Google first got going.

Now the question seems to be, at what point is Google too big to even bother?

Post reply on HN