Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

vice.com

Post reply on HN
Print view View original

21–30 of 93 posts

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#21

Post by tmabraham »

walrus01 wrote: post #17
CheezeIt wrote: post #13

Earlier quoted context omitted.

You can pick up an 8TB WD black external for $200, and I saw cheaper WD EasyStore 8TB’s at retail in Best Buy today at $180, and that’s just what I saw offhand earlier today.

Something is starting to go really wrong in HDD pricing, following GPUs and other things, because I bought that same 8TB drive two years ago for $147. Hard drive prices are supposed to decrease in $ per TB over a multi year period, with advances in areal density.

overall electronics supply chain issues, and I heard something about cryptocurrency mining also being involved with supply issues for hard drives.

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#22

Post by lamontcg »

walrus01 wrote: post #17
CheezeIt wrote: post #13

Earlier quoted context omitted.

You can pick up an 8TB WD black external for $200, and I saw cheaper WD EasyStore 8TB’s at retail in Best Buy today at $180, and that’s just what I saw offhand earlier today.

Something is starting to go really wrong in HDD pricing, following GPUs and other things, because I bought that same 8TB drive two years ago for $147. Hard drive prices are supposed to decrease in $ per TB over a multi year period, with advances in areal density.

The "proof of space" Chia cryptocurrency:

https://news.ycombinator.com/item?id=27009843

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#23

Post by andrewjl »

na85 wrote: post #11

I've seen estimates of the total sci-hub/libgen size at anywhere from 20TB to 75+TB. Probably there are lots of dupes in there, and a dedicated dedup effort could reduce that size by a bit. But taking the upper end: A WD Black 10 TB platter drive[0] costs $329, which means it would take only $2700 or so to have enough storage space to archive what's basically the sum total of human scientific knowledge. [0] - https:/…

Obvious copyright & TOS issues aside, how does this compare to the S3 bill?

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#24

Post by CheezeIt »

walrus01 wrote: post #17
CheezeIt wrote: post #13

Earlier quoted context omitted.

You can pick up an 8TB WD black external for $200, and I saw cheaper WD EasyStore 8TB’s at retail in Best Buy today at $180, and that’s just what I saw offhand earlier today.

Something is starting to go really wrong in HDD pricing, following GPUs and other things, because I bought that same 8TB drive two years ago for $147. Hard drive prices are supposed to decrease in $ per TB over a multi year period, with advances in areal density.

I’ll add, the 14TB WD externals disappeared off the shelves at the local Best Buy in the past month. The data hoarding subreddit was calling those the best price per TB. There were some $140 listings for an 8TB drive on Amazon a week or two ago, but now they’re $165 and out of stock. I think crypto and hoarding are the reasons; of course inflation is a multiplier.

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#25

Post by na85 »

andrewjl wrote: post #23
na85 wrote: post #11

I've seen estimates of the total sci-hub/libgen size at anywhere from 20TB to 75+TB. Probably there are lots of dupes in there, and a dedicated dedup effort could reduce that size by a bit. But taking the upper end: A WD Black 10 TB platter drive[0] costs $329, which means it would take only $2700 or so to have enough storage space to archive what's basically the sum total of human scientific knowledge. [0] - https:/…

Obvious copyright & TOS issues aside, how does this compare to the S3 bill?

Good question. It's about $0.023 per GB per month for a US East Coast instance, so some $1700/mo for 75TB. Plus you don't really control the data if it's on someone else's computer so there's a potential opportunity cost there as well.

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#26

Post by OkGoDoIt »

na85 wrote: post #25
andrewjl wrote: post #23

Earlier quoted context omitted.

Obvious copyright & TOS issues aside, how does this compare to the S3 bill?

Good question. It's about $0.023 per GB per month for a US East Coast instance, so some $1700/mo for 75TB. Plus you don't really control the data if it's on someone else's computer so there's a potential opportunity cost there as well.

On the other hand the data would be much more reliable in that scenario. A 10 GB consumer hard drive is likely to fail or at least corrupt some data at some point, and if you’re going to go with multiple redundancies then your cost goes up.

But yeah, it’s all kind of moot. I had AWS suspend my account once because someone complained they found a copyrighted image on one of my servers. If it’s in the cloud, you don’t really control it.

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#27

Post by na85 »

OkGoDoIt wrote: post #26
na85 wrote: post #25

Earlier quoted context omitted.

Good question. It's about $0.023 per GB per month for a US East Coast instance, so some $1700/mo for 75TB. Plus you don't really control the data if it's on someone else's computer so there's a potential opportunity cost there as well.

On the other hand the data would be much more reliable in that scenario. A 10 GB consumer hard drive is likely to fail or at least corrupt some data at some point, and if you’re going to go with multiple redundancies then your cost goes up. But yeah, it’s all kind of moot. I had AWS suspend my account once because someone complained they found a copyrighted image on one of my servers. If it’s in the cloud, you don’t…

You can buy "datacenter-grade" drives from WD on their website, which are presumably of higher quality.

Either way, though, this is going to cost five figures, annually, to store in S3. For half of one year of S3 you can have a fairly robust RAID setup, maybe replace 1 drive every two years, and enjoy the data any time you want it without fees-as-a-service.

High five for avoiding AWS!

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#28

Post by justinclift »

OkGoDoIt wrote: post #26
na85 wrote: post #25

Earlier quoted context omitted.

Good question. It's about $0.023 per GB per month for a US East Coast instance, so some $1700/mo for 75TB. Plus you don't really control the data if it's on someone else's computer so there's a potential opportunity cost there as well.

On the other hand the data would be much more reliable in that scenario. A 10 GB consumer hard drive is likely to fail or at least corrupt some data at some point, and if you’re going to go with multiple redundancies then your cost goes up. But yeah, it’s all kind of moot. I had AWS suspend my account once because someone complained they found a copyrighted image on one of my servers. If it’s in the cloud, you don’t…

> I had AWS suspend my account once because someone complained they found a copyrighted image on one of my servers. If it’s in the cloud, you don’t really control it.

Considering the material here, that's more likely to happen than not. So, unlikely to be "much more reliable" for this particular scenario. ;)

Re: Archivists Are Trying to Make Sure a ‘Pirate Bay of Science’ Never Goes Down (2019)

#29

Post by saagarjha »

CheezeIt wrote: post #13
na85 wrote: post #11

I've seen estimates of the total sci-hub/libgen size at anywhere from 20TB to 75+TB. Probably there are lots of dupes in there, and a dedicated dedup effort could reduce that size by a bit. But taking the upper end: A WD Black 10 TB platter drive[0] costs $329, which means it would take only $2700 or so to have enough storage space to archive what's basically the sum total of human scientific knowledge. [0] - https:/…

You can pick up an 8TB WD black external for $200, and I saw cheaper WD EasyStore 8TB’s at retail in Best Buy today at $180, and that’s just what I saw offhand earlier today.

Costco has a Seagate 8TB drive on sale for $120 until tomorrow: https://www.costco.com/seagate-backup-plus-hub-8tb-desktop-h...
Post reply on HN