Activists rally to save Internet Archive as lawsuit threatens site

Post by DaiPlusPlus » Thu, Jun 11, 2020, 6:09 AM UTC

jonah-archive wrote: post #9

fouc wrote: post #8

Earlier quoted context omitted.

Very interesting discussion here. Apparently can get 16TB drives at $100 each on enterprise bulk pricing. An entire backup would be 3,125 drives for $312,500. But apparently the wayback machine itself is only about 2 Petabytes.. so if you don't need the collections perhaps only 125 drives needed, or $12,500.

Disclaimer: I run the infrastructure/ops team at the Internet Archive. Unfortunately none of those numbers are really even close to correct (the discussion is always fun, but the folks in r/datahoarder are often not correctly informed. textfiles has more patience for it than I do). It would probably cost around 1.5M in drives, even at reasonable current enterprise volume pricing, to back up the 60+ PB of unique data…

Post by bits_n_bytes » Thu, Jun 11, 2020, 6:17 AM UTC

jonah-archive wrote: post #9

fouc wrote: post #8

Earlier quoted context omitted.

Very interesting discussion here. Apparently can get 16TB drives at $100 each on enterprise bulk pricing. An entire backup would be 3,125 drives for $312,500. But apparently the wayback machine itself is only about 2 Petabytes.. so if you don't need the collections perhaps only 125 drives needed, or $12,500.

Disclaimer: I run the infrastructure/ops team at the Internet Archive. Unfortunately none of those numbers are really even close to correct (the discussion is always fun, but the folks in r/datahoarder are often not correctly informed. textfiles has more patience for it than I do). It would probably cost around 1.5M in drives, even at reasonable current enterprise volume pricing, to back up the 60+ PB of unique data…

Post by dehrmann » Thu, Jun 11, 2020, 6:19 AM UTC

pixxel wrote: post #2

> The Internet Archive removed waitlists for books in its "National Emergency Library" so that multiple readers could simultaneously download the same digital copy. >Four major book publishers have responded by suing the Internet Archive. If successful, they could bankrupt the nonprofit. >The publishers take issue not only with the National Emergency Library, but with the Internet Archive as a whole.

Post by jonah-archive » Thu, Jun 11, 2020, 6:20 AM UTC

DaiPlusPlus wrote: post #11

jonah-archive wrote: post #9

Earlier quoted context omitted.

Disclaimer: I run the infrastructure/ops team at the Internet Archive. Unfortunately none of those numbers are really even close to correct (the discussion is always fun, but the folks in r/datahoarder are often not correctly informed. textfiles has more patience for it than I do). It would probably cost around 1.5M in drives, even at reasonable current enterprise volume pricing, to back up the 60+ PB of unique data…

Does that 1.5m figure include redundancy (e.g. RAID)? What about if the data is compressed (either trivial gzipping each "file" the archive has, or using a compression window that spans multiple files)? I imagine the HTML/plaintext content of the Internet Archive would compress very well.

Post by dehrmann » Thu, Jun 11, 2020, 6:29 AM UTC

RcouF1uZ4gsC wrote: post #14

> In doing so, it essentially allowed for a single copy of a book to be downloaded an infinite number of times. The publishers response to doing this was completely predictable. Maybe the leadership needs more diversity of background. Having someone who has been in the publishing business in that meeting when this was deemed a good idea maybe could have prevented this disaster.