Glad I contributed to this in some small way.
ArchiveTeam has finished archiving all goo.gl short links
21–30 of 112 posts
Re: ArchiveTeam has finished archiving all goo.gl short links
#22Does "all" mean all the URLs publicly known, or did they exhaustively iterate the entire URL namespace?
The goo.gl URLs that are publicly known are already in the Internet Archive and Common Crawl crawls.
Re: ArchiveTeam has finished archiving all goo.gl short links
#23Recent update from Google: https://blog.google/technology/developers/googl-link-shorten...
I don't really understand this. Is it really that costly to keep the entire database if they're going to keep part of it?
I don't understand the data on ArchiveTeam's page but, it seems like they have 35 terabytes of data (286.56TiB)? It's a lot larger than I'd have thought.
Re: ArchiveTeam has finished archiving all goo.gl short links
#24Title is imprecise, it's Archiveteam.org, not Archive.org. The Internet Archive is providing free hosting, but the archival work was done by Archiveteam members.
What exactly is archiveteam's contribution? I don't fully understand. Edit: Like they kinda seem like an unnecessary middle-man between the archive and archivee, but maybe I'm missing something.
> What exactly is archiveteam's contribution? I don't fully understand.
If Internet Archive is a library, ArchiveTeam is people who run around collecting stuff, and gives it to the library for safe keeping. Stuff that are estimated/announced to be disappearing/removed soon tends to be focused too.
Re: ArchiveTeam has finished archiving all goo.gl short links
#25Earlier quoted context omitted.
I don't really understand this. Is it really that costly to keep the entire database if they're going to keep part of it?
I built a URL shortener years ago for fun. I don't have the resources that Google has, but I just hacked it together in Erlang using Riak KV and it did horizontally scale across at least three computer (I didn't have more at the time). Unless I'm just super smart (I'm not), it's pretty easy to write a URL shortener as a key-value system, and pure key-value stuff is pretty easy to scale. I cannot imagine that isn't do…
Google also has the advantages that they now only need a read-only key-value store, and they know the frequency distribution for lookups. This is now the kind of problem many programmers would be happy to spend a weekend optimizing to get an average lookup time down to tens of nanoseconds.
Re: ArchiveTeam has finished archiving all goo.gl short links
#26Earlier quoted context omitted.
What exactly is archiveteam's contribution? I don't fully understand. Edit: Like they kinda seem like an unnecessary middle-man between the archive and archivee, but maybe I'm missing something.
ArchiveTeam delegates tasks to volunteers and themselves running the Archive Warrior VM, which does the actual archiving. The resultant archives are then centralized by ArchiveTeam and uploaded to the Internet Archive. (Source: ran a Warrior)
Sidenote, but you can also run a Warrior in Docker, which is sometimes easier to set up (e.g. if you already have a server with other apps in containers).
Re: ArchiveTeam has finished archiving all goo.gl short links
#27Glad I contributed to this in some small way.
Same, it's nice to see my username on the leaderboards.
Even though all I did was setup the docker container one day and forget about it
Re: ArchiveTeam has finished archiving all goo.gl short links
#28Title is imprecise, it's Archiveteam.org, not Archive.org. The Internet Archive is providing free hosting, but the archival work was done by Archiveteam members.
What exactly is archiveteam's contribution? I don't fully understand. Edit: Like they kinda seem like an unnecessary middle-man between the archive and archivee, but maybe I'm missing something.
What ArchiveTeam mainly does is provide hand-made scripts to aggressively archive specific websites that are about to die, with a prioritization for things the community deems most endangered and most important. They provide a bot you can run to grab these scripts automatically and run them on your own hardware, to join the volunteer effort.
This is in contrast to the Wayback Machine's builtin crawler, which is just a broad spectrum internet crawler without any specific rules, prioritizations, or supplementary link lists.
For example, one ArchiveTeam project had the goal to save as many obscure Wikis as possible, using the MediaWiki export feature rather than just grabbing page contents directly. This came in handy for thousands of wikis that were affected by Miraheze's disk failure and happened to have backups created by this project. Thanks to the domain-specific technique, the backups were high-fidelity enough that many users could immediately restart their wiki on another provider as if nothing happened.
They also try to "graze the rate limit" when a website announces a shutdown date and there isn't enough time to capture everything. They actively monitor for error responses and adjust the archiving rate accordingly, to get as much as possible as fast as possible, hopefully without crashing the backend or inadvertently archiving a bunch of useless error messages.
Re: ArchiveTeam has finished archiving all goo.gl short links
#29I wonder how many of them lead to private YouTube videos, Google documents, etc.
Re: ArchiveTeam has finished archiving all goo.gl short links
#30Earlier quoted context omitted.
I don't really understand this. Is it really that costly to keep the entire database if they're going to keep part of it?
I don't understand the data on ArchiveTeam's page but, it seems like they have 35 terabytes of data (286.56TiB)? It's a lot larger than I'd have thought.
FYI, "TiB" means terabytes with a base of 1024, ie. the units you'd typically use for measuring memory rather than the units you'd typically see drive vendors using. The factor of 8 you divided by only applies to units based on bits rather than bytes, and those units use "b" rather than "B", and are only used for capacity measurements when talking about individual memory dies (though they're normal for talking about interconnect speeds).
Either way, we're talking about a dataset that fits easily in a 1U server with at most half of its SSD slots filled.