Storing 25 petabytes of Megaupload data costs us $9,000 a day • Hacker News Board

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#81

Post by shrike » Thu, Mar 22, 2012, 10:44 PM UTC

The federal government does have a process for this sort of thing, if they seize an alleged drug dealer's house and that house has a mortgage the United States Marshals Service will pay the mortgage. If they seize cars, furniture, other assets the government is responsible for the storage of those items until the case has been resolved. [1]

I would guess that MegaUpload's lawyers will make the claim that the data on those servers is critical to their defense and must be maintained. That is probably an accurate claim, DotCom will want to present evidence of compliance with DMCA notices, counter the claim that a "majority" of the content was under copyright, etc. Best case for DotCom would probably be that his lawyers argue for retaining the data and the judge lets Carpathia destroy it anyway. That would give DotCom reasonable grounds for appeal.

[1] http://en.wikipedia.org/wiki/Asset_forfeiture

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#82

Post by VikingCoder » Thu, Mar 22, 2012, 10:48 PM UTC

njharman wrote: post #78

VikingCoder wrote: post #70

Help me out with the math here: 1 terabyte costs them $128.41 per year, right? Amazon S3 would cost them roughly $444 per year, if they were using the Reduced Redundancy Storage. The cheapest HD that I see on pcpartpicker (in terms of Price/GB) is the Western Digital Caviar Green 2.5 TB (5400 RPM) for $135.43, which is $0.054/GB. That's $54.17 per TB. If you want a single backup, that's $108.34 per TB. Two backups (3…

The hardware cost 1.25 million. It cost $9000/day in electricity/connectivity/rackspace.

Got it. So, yeah, I missed something completely obvious.

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#83

Post by devicenull » Thu, Mar 22, 2012, 10:48 PM UTC

samstave wrote: post #49

DanBC wrote: post #40

Earlier quoted context omitted.

25 petabytes of hard disc is a considerable cost. Some people have mentioned insurance. Would any policy cover you for a customer being investigated for possibly criminal activity?

Assuming 2TB internal HDD's at $120.00 per piece, you would need $1.5MM worth of drives to transfer this to. Not including the man-hours associated with doing so. The $3.2MM per year is actually sounding a lot more reasonable to keep it up and running.

Don't forget replacement RAID controllers! The RAID config is usually dependent on the exact drive/controller combination, so removing the drives and not saving the controller may result in them becoming inaccessible. At the very least, they would almost certainly not pass requirements to be used as evidence if that happened.

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#84

Post by brownbat » Thu, Mar 22, 2012, 10:49 PM UTC

njharman wrote: post #78

VikingCoder wrote: post #70

Help me out with the math here: 1 terabyte costs them $128.41 per year, right? Amazon S3 would cost them roughly $444 per year, if they were using the Reduced Redundancy Storage. The cheapest HD that I see on pcpartpicker (in terms of Price/GB) is the Western Digital Caviar Green 2.5 TB (5400 RPM) for $135.43, which is $0.054/GB. That's $54.17 per TB. If you want a single backup, that's $108.34 per TB. Two backups (3…

The hardware cost 1.25 million. It cost $9000/day in electricity/connectivity/rackspace.

True. I'm honestly surprised they didn't declare daily depreciation, for $1.25 million in assets that are obsoleted by new technology at Moore's pace, I'd expect that could arguably be quite high.

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#85

Post by brownbat » Thu, Mar 22, 2012, 10:58 PM UTC

The urgency is because Carpathia's lease has run out, they can't stay at the $9k/day facility.

Carpathia has to pay $65k to move the servers, then $37k per month to keep them in a climate controlled facility while powered down. Lost profits are still a relevant consideration. This is a doozy of a damages calculation. What's depreciation on assets that are rendered obselete by (something like) Moore's law?

I'd say Carpathia deletes the data and then supports the petitioners (those with lost data) in the takings clause case against the government. Carpathia claims indemnity against claims by pointing at MegaUpload and the Feds, but probably gets joined in a bunch of messy lawsuits. Real roll of the dice.

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#86

Post by johngalt » Thu, Mar 22, 2012, 11:11 PM UTC

I've dealt with e-discovery sets. No one really has answers to what to do when you have a litigation hold on data. Legislation commonly requires "retention of anything related to X case", but how do you know what's relevant and what isn't? When you are a third party the ambiguity increases. So you end up with an everything and kitchen sink data dump. Even with everything the data is commonly useless without context. You have files without access logs and logs referrencing local namespaces etc...

With a 25 petabyte discovery, I'm not surprised that everyone's scratching their heads on what to do next. This isn't just an MPAA/Megaupload problem. Even a smaller dataset like a 10-20TB discovery has numerous problems. Hosting/indexing/classifying/reviewing millions of documents is an open issue for the legal field. What do you do when there are multiple parties who all need to see "everything"? If everyone does their own thing how do you reference materials in a consistent manner across the interested parties? If you all agree to host the data in a neutral place who pays for it? What if the technology of that host benefits one party at the expense of another?

For years the legal field has had a "print it all out and have a team of paralegals go over it" viewpoint. Clients don't pay for computers, but they do pay for paralegal hours. Only recently has that become untenable. Discovery sizes are growing exponentially per year. It's common to have a new discovery set come in larger that every previous set combined, and the legal industry doesn't really know what to do about it.

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#87

Post by brown9-2 » Thu, Mar 22, 2012, 11:12 PM UTC

zevyoura wrote: post #69

jneal wrote: post #46

What's the big deal? Just delete the data. Customer pays for storage. Company stores. Customer stops paying for storage. Company deletes. Sure, a bunch of pissed off people will certainly be upset - but it's not the company's fault - they shouldn't have to bear this burden. I can't see how they could be sued by users for this, they didn't enter into any kind of agreement with the users, only with the customer.

The article also mentions that they need to hold onto the data because it may be used as evidence in the court case.

It boggles my mind that the court hasn't taken position of the hardware/data then.

If the hardware is left running, and not in official custody, how do any authorities know that the data isn't being tampered with?

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#88

Post by jellicle » Thu, Mar 22, 2012, 11:24 PM UTC

anon808 wrote: post #6

It sucks, but that's the price of doing business. They chose their customer, and now are (unfortunately) tied to consequences. Same thing happens to building owners who have a crime committed by a tenant, the leased space becomes a crime scene until the police/govt are done with their investigation.

But investigations take hours, while this case will take years.

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#89

Post by Tuna-Fish » Thu, Mar 22, 2012, 11:25 PM UTC

jlawer wrote: post #22

The costs of moving that amount of data is crazy. I am surprised the government hasn't seized the hardware, and chucked it in a warehouse. I did some back of an envelope calculations... and its absolutely crazy. Tape would require over 17,000 Ultrium tapes. Now you could De-dupe... but the hardware to process and dedupe that much data.... not really an option. Not to mention the time to write that many tapes... Somet…

> Now you could De-dupe... but the hardware to process and dedupe that much data.... not really an option.

FYI, the data is already comprehensively de-duped.

Re: Storing 25 petabytes of Megaupload data costs us $9,000 a day

#90

Post by astrodust » Thu, Mar 22, 2012, 11:53 PM UTC

johngalt wrote: post #86

I've dealt with e-discovery sets. No one really has answers to what to do when you have a litigation hold on data. Legislation commonly requires "retention of anything related to X case", but how do you know what's relevant and what isn't? When you are a third party the ambiguity increases. So you end up with an everything and kitchen sink data dump. Even with everything the data is commonly useless without context.…

If it's anything like the usual proceedings, the lawyers involved would probably prefer if they had several copies of the data, too.

I'm sure most of these drives were arrayed in such a fashion where they're unreadable unless in the proper equipment. It's not like you can just buy a pile of off-the-shelf external drives and start copying, either, as the contents might be unreadable unless the proper software is installed and configured correctly.