ArchiveTeam has finished archiving all goo.gl short links

Post by globular-toast » Mon, Aug 18, 2025, 6:57 AM UTC

brokensegue wrote: post #72

mdaniel wrote: post #71

Earlier quoted context omitted.

You are aware of which thread you're discussing this in, right? The one where a bunch of like-minded souls enumerated all the address space in a few weeks? The sibling link above that queries Wayback's warc index shows at least the first several are only 6 alnum wide so it's no wonder the ArchiveTeam got them in reasonable time Picking one at random, it seems the super sekrit deets you're safeguarding include buyruss…

i asked them why they did this. the answer surprisingly is because they fear if they release the full dumps they will get blocked because of the AI scraping wars.

Post by lyu07282 » Mon, Aug 18, 2025, 7:03 AM UTC

digitaldragon wrote: post #77

Aardwolf wrote: post #36

I don't understand the page, it shows a list of data sets (I think?) up to 91 TiB in size The list of short links and their target URLs can't be 91 TiB in size can it? Does anyone know how this works?

The data is saved as a WARC file, which contains the entire HTTP request and response (compressed, of course). So it's much bigger than just a short -> long URL mapping.

Post by gunalx » Mon, Aug 18, 2025, 7:21 AM UTC

1gn15 wrote: post #18

im3w1l wrote: post #15

Earlier quoted context omitted.

What exactly is archiveteam's contribution? I don't fully understand. Edit: Like they kinda seem like an unnecessary middle-man between the archive and archivee, but maybe I'm missing something.

ArchiveTeam delegates tasks to volunteers and themselves running the Archive Warrior VM, which does the actual archiving. The resultant archives are then centralized by ArchiveTeam and uploaded to the Internet Archive. (Source: ran a Warrior)

Post by patrickhogan1 » Mon, Aug 18, 2025, 8:10 AM UTC

lyu07282 wrote: post #80

edg5000 wrote: post #78

Can we build a blockchain/P2P-based web crawler that can create snapshots of the entire web with high integrity (peer verification)? The already-crawled pages would be exchanged through bulk transfer between peers. This would mean there is an "official" source of all web data. LLM people can use snapshots of this. This would hopefully reduce the amount of ill-behaved crawlers, so we will see less draconian anti-bot m…

> This would mean there is an "official" source of all web data. LLM people can use snapshots of this that already exists, its called CommonCrawl: https://commoncrawl.org/

Post by edg5000 » Mon, Aug 18, 2025, 8:39 AM UTC

lyu07282 wrote: post #80

edg5000 wrote: post #78

Can we build a blockchain/P2P-based web crawler that can create snapshots of the entire web with high integrity (peer verification)? The already-crawled pages would be exchanged through bulk transfer between peers. This would mean there is an "official" source of all web data. LLM people can use snapshots of this. This would hopefully reduce the amount of ill-behaved crawlers, so we will see less draconian anti-bot m…

> This would mean there is an "official" source of all web data. LLM people can use snapshots of this that already exists, its called CommonCrawl: https://commoncrawl.org/