> In total the VMs use 182 GB of RAM and 94 CPU cores. The total storage capacity is 620 GB, but that’s not all used. That level of hardware/cores seems a bit over the top given what TPB does. When I was a boy we had this thing called 'Alta Vista'. It was the search engine before Bing! came along. Processors did not run at gigahertz speeds back then and a large disk was 2Gb. Nonetheless most offices had the internet…
The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
41–50 of 55 posts
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#42> In total the VMs use 182 GB of RAM and 94 CPU cores. The total storage capacity is 620 GB, but that’s not all used. That level of hardware/cores seems a bit over the top given what TPB does. When I was a boy we had this thing called 'Alta Vista'. It was the search engine before Bing! came along. Processors did not run at gigahertz speeds back then and a large disk was 2Gb. Nonetheless most offices had the internet…
IIRC there where (quite) a few before bing. More to the point google was the pinnacle of web searches long before bing came into existence.
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#43> In total the VMs use 182 GB of RAM and 94 CPU cores. The total storage capacity is 620 GB, but that’s not all used. That level of hardware/cores seems a bit over the top given what TPB does. When I was a boy we had this thing called 'Alta Vista'. It was the search engine before Bing! came along. Processors did not run at gigahertz speeds back then and a large disk was 2Gb. Nonetheless most offices had the internet…
They also didn't get as much traffic as TBP, since there wasn't that many connected back then.
I would also imagine that they didn't have to HIDE their services either.
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#44Earlier quoted context omitted.
It depends on what you mean by that. The only way to prevent a codebase from being seen by an adversary with physical access when the server is on is to not have the sensitive data on the server in the first place. Encryption (with the decryption key being gotten at boot from, say, a particular .onion address) would work against backups, but won't protect against an adversary with admin access to the server when the…
This would be a cool application for some kind of homomorphic encryption. Server gets encrypted search request and matches it against an encrypted index and returns the encrypted results.
Giving Joe Public access to the private key necessary for interpreting the query result allows attackers to inspect all of the intermediate states of the query finite state machine, which allows debugging and inspection just as if homomorphic encryption wasn't in use.
I suppose the routing proxy could hold the private key and decrypt the query result for the general public. However, the location of the routing proxy is almost certainly going to be compromised before the locations of the servers executing the queries, so in the decrypting proxy scenario, the attackers will almost certainly have the secret keys before they get access to the boxes executing the queries. There's also the problem that the messages being decrypted are the final states of finite state machines that executed the queries, so the messages to be copied over the network add up in size to at least the size of the dataset being queried. (The data can be sharded into many smaller databases, and almost certainly would be in order to speed up the homomorphic computation steps, but this doesn't cut down on the amount of network traffic necessary to retrieve all search results for a single query. A simple query on 1 TB of data, split into 10,000 databases each of 100 MB would require copying and remotely decrypting 10,000 messages, each over 100 MB in size.)
It might be possible to discover a homomorphic encryption scheme whereby knowledge of the private key allows one to devise a mapping from a higher dimensional finite state to a lower dimensional finite state machine, where the secret key for the smaller dimensional machine doesn't leak information about the secret key for the larger machine. In this case, it may be possible to perform some finishing operations on the query to prepare it for conversion to the smaller state machine and give the public the private key to the smaller state machine so that the query result could be read from the machine by the public without the public being able to observe intermediate states of the query computation. However, I believe this is far beyond our current mathematical understanding.
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#45> In total the VMs use 182 GB of RAM and 94 CPU cores. The total storage capacity is 620 GB, but that’s not all used. That level of hardware/cores seems a bit over the top given what TPB does. When I was a boy we had this thing called 'Alta Vista'. It was the search engine before Bing! came along. Processors did not run at gigahertz speeds back then and a large disk was 2Gb. Nonetheless most offices had the internet…
How many pages were there on "the entire internet" back then? How many reqs/second did AltaVista serve? How does that compare to the numbers for TPB?
Alta Vista started out with a modest size index of 20 million pages. Let's imagine those pages were all of 1Kb in size, then, 20 10^6 10^3 comes to 20 *10^9 or 20Gb. So, in terms of stuff indexed, that is considerably larger than TPB. Agreed?
Well, maybe not. They could have used compression to get the vastness of TPB onto that USB stick. Around that time - 2012 - they had 1.6 million torrents. That is some way off the Gb that AltaVista indexed, no matter how you bloat the maths. Sad to say, but, in the 1990's, the internet was actually larger than your porn collection.
How useful is reqs/second anyway? By that score Google probably does very badly as a search usually returns the answer on the first page. With old-style search engines you might need to go through scores of pages before getting what you want. I found TPB to be a bit like that too, wading through results pages more than necessary.
TPB is not 'safe for work' and in a lot of jurisdictions you cannot even access it from home. In the UK (which is a small but well populated country) it is not that easy to get onto TPB - you have to have hacker voodoo skills to do that or route through a VPN as none of the main ISPs will let you on. Most of the civilised world has the same need to protect citizens from the evils of TPB so places where it can be accessed are not that common. Even if you could access it, would you? Probably...
Meanwhile, back in 1998 - a year or two before the dotcom crash - plenty of people were using search engines such as AltaVista (which was the best back then) for actual work. Maybe not everyone, but enough people knew about computers and things like AOL disks, modems and what not. The internet was big.
Which reminds me of my main point, the one you thought so important to down vote rather than give kudos for being insightful. TPB uses a constellation of computers and consumes vastly more resources than the biggest search engine of the 1990's, yet, the utility of TPB is limited to only a few fortunate enough to live somewhere where TPB can be accessed. What can be searched for on TPB is a mere subset of what was on Altavista albeit different and not so useful stuff. I would say that with AltaVista they were doing far more with what they had, reaching a better audience, doing something more useful for the world (than serving weight loss adverts) and all together performing a miracle. TPB is a slouch in comparison.
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#46Earlier quoted context omitted.
How many pages were there on "the entire internet" back then? How many reqs/second did AltaVista serve? How does that compare to the numbers for TPB?
From what I remember the whole of TPB server + data could fit onto a 90Mb usb stick in 2012. Sure we have had many episodes of really important reality TV series and other great stuff that all needs pirating, yet, in 2014 I doubt that 90Mb has ballooned into peta bytes. We are still in the same range - let's say 1Gb might be a reasonable size USB stick to buy for it. Alta Vista started out with a modest size index of…
Running a top 100 site[1] on 21 VMs in 2014 is quite impressive.
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#47Interesting, so I'm presuming there's several VPNs involved between the load-balancer and all the discrete servers. I wonder if they use a VPN provider with a static IP and no-logs policy or if it's simply yet another VPS. I'd love to hear a little more about the architecture.
[1] https://www.ipredator.se/ [2] http://torrentfreak.com/pirate-bay-announces-ipredator-globa...
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#48Earlier quoted context omitted.
This would be a cool application for some kind of homomorphic encryption. Server gets encrypted search request and matches it against an encrypted index and returns the encrypted results.
The problem with homomorphic encryption in this case is that you need access to the private key in order to interpret the results of the computation. Joe Public encrypts queries using a public key, and the query runs on a possibly compromised server without the attackers learning anything about the database or the query, but then the result of the query looks like absolute gibberish to Joe Public. Giving Joe Public a…
This doesn't apply to TPB, but one could give each user of, say, an email webapp the private key to his/her own data while still facilitating server-side search.
> I suppose the routing proxy could hold the private key and decrypt the query result for the general public. However, the location of the routing proxy is almost certainly going to be compromised before the locations of the servers executing the queries.
This wouldn't be completely useless since it lets you offload much of the storage and computation onto commodity cloud providers without revealing what's on the machines, even if they're scanning your RAM. From the article it seems like TPB is getting some kind of utility out of such a scheme: "All virtual machines are hosted with commercial cloud hosting providers, who have no clue that The Pirate Bay is among their customers. All traffic goes through the load balancer, which masks what the other VMs are doing."
> There's also the problem that the messages being decrypted are the final states of finite state machines that executed the queries, so the messages to be copied over the network add up in size to at least the size of the dataset being queried.
In a homomorphic encryption scheme that supported querying, only the encrypted results would need to be relayed back from each search shard, no?
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#49Earlier quoted context omitted.
This would be a cool application for some kind of homomorphic encryption. Server gets encrypted search request and matches it against an encrypted index and returns the encrypted results.
The problem with homomorphic encryption in this case is that you need access to the private key in order to interpret the results of the computation. Joe Public encrypts queries using a public key, and the query runs on a possibly compromised server without the attackers learning anything about the database or the query, but then the result of the query looks like absolute gibberish to Joe Public. Giving Joe Public a…
Re: The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines
#50Earlier quoted context omitted.
If someone is paying the bill, do they really care?
"If someone is paying the bill, do they really care?" So you could cross reference names of the people raided with payment information of the VPS providers (usual suspects or top "n" providers let's say). Of course that could be hidden as well. Other issue is how does anyone know this isn't misinformation anyway and that the VPS providers don't play a role or not as much of a role as is indicated. Just because someon…
I doubt this will be useful as they're probably using Bitcoin or prepaid cards or things like that for payments.
> What advantage does it have for anyone (like this) to reveal anything
It could teach others how to setup websites that are harder to censor or more resilient to raids (plus it gets them free PR/traffic).