Live data from Hacker News

We have successfully completed our migration to RAM-only VPN infrastructure

mullvad.net

61–70 of 195 posts

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#61

Earlier quoted context omitted.

This has never made any sense to me. I'm surprised this isn't a massive red flag from anyone on HN. Running a production-grade service with zero metrics and logs? If there's an outage, or even something as mundane as a VM failing to provision, you're telling me that Mullvad developers just shrug and say "well, we can't do anything, because there's no logs!" I don't use a third party VPN, but if I wanted to, "we delib…

Who said they don't have metrics? The VPN servers don't have any storage, but they can still send metrics to an off-machine API, or a different server that does have storage can send requests to the VPN servers and do metrics that way. Ditto for logging. They claim to not log activity over the VPN itself, but I don't see any claims about not logging more mundane infra stuff like "a VM failed to provision". I think yo…

> but they can still send metrics to an off-machine API

And that would be the next most interesting post, imo. A post about how they metric and log in a RAM-only environment while obscuring or obfuscating the details that could lead to “compromise”. Even if the answer is something so simple like “we regex and discard this out”, I would feel more trusting of their services.

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#62
post #42

Earlier quoted context omitted.

> Some pour thousands of dollars into forcing influencers to say they care about security, Tangential to this, it always irks me how they talk about how they all act as if the majority of the websites their users are going to aren't HTTPS and they act like their main benefits are filling in the gaps that HTTPS actually fills in. HTTPS isn't a cure all by any means but most of the scare tactics that the big VPN compan…

I’d agree with you about HTTPS providing most of the benefit that VPN advertising focuses on if I hadn’t seen repeated direct evidence that even most technical users will blithely click through HTTPS errors’ “accept the risk” bypass. It’s as if knowledgeable users think “sure, this could be a man in the middle attack, but it’s most likely just a benign cert problem, because certs are hard.” Sigh.

To be frank that's also because the cause for an HTTPS certificate error ranges from "malicious hijack" to "misconfigured server setup" to "I lapsed the expiry date" to "I am using a self-signed certificate".

The degree of which these should be scares is not equivalent, yet browsers will treat all of these as equivalent even though they can distinguish between them in the error page. It just results in clickthrough fatigue, where technical users just ignore the warning because it's not worthwhile to deal with even when they really should.

Plus a VPN won't protect you from a malicious hijack, it just prevents them from grabbing your IP address.

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#63
post #10
post #4

Earlier quoted context omitted.

There is no disk in the servers, so there is no chance for user information to persist anywhere. I also wouldn’t be surprised if it’s a performance benefit, since RAM is far faster than any permanent storage. The cons are probably just that this is a pretty unusual architecture that they probably had to put some work into setting up and making it reliable.

You can still mount a remote networked file system to a dikless node. Lack of disks does not guarantee inability to persist data.

You can talk about hypothetically doing anything. You're not really making a point.

Hypothetically, you can rewrite the firmware for the IPMI with a backdoor and extract data.

Hypothetically, you could kidnap the family of a developer and force them to add a surreptitious side channel for exfiltrating data.

Hypothetically, you could run the universe in a simulator and use the simulator's controls to read data from RAM in the simulated Mullvad servers on the simulated Earth.

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#64
post #27

Earlier quoted context omitted.

If only the system were open source so you wouldn't have to wonder about that... But we do still have to trust that they are actually running the code they posted. Unless that code somehow contains some way to verify itself? I wonder if there is some way to do that? Have the code include a hash of itself and some way to query the running service that guarantees that the running service must be running the code you ar…

Homomorphic encryption: https://en.wikipedia.org/wiki/Homomorphic_encryption

I think the normal solution to this is all the prove you are the software you say you are calls is proxied to that software and all the normal services calls you want to log and duplicate or otherwise violate the contract are sent to the modified code.

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#65
post #42

Earlier quoted context omitted.

> Some pour thousands of dollars into forcing influencers to say they care about security, Tangential to this, it always irks me how they talk about how they all act as if the majority of the websites their users are going to aren't HTTPS and they act like their main benefits are filling in the gaps that HTTPS actually fills in. HTTPS isn't a cure all by any means but most of the scare tactics that the big VPN compan…

I’d agree with you about HTTPS providing most of the benefit that VPN advertising focuses on if I hadn’t seen repeated direct evidence that even most technical users will blithely click through HTTPS errors’ “accept the risk” bypass. It’s as if knowledgeable users think “sure, this could be a man in the middle attack, but it’s most likely just a benign cert problem, because certs are hard.” Sigh.

The problems with clicking past those errors are typically not due to network sniffing but with whatever crazy shit is on the page they are going to.

The only two valid usecases of big VPNs like these are

1. Very mild security increase over public wifi 2. Shifting your risk from the ISP spying to mullvad or the VPN provider spying or slightly anonymizing if mullvad rotates IPs.

(2) is a real benefit because ISPs are pretty terrible, but it's still pretty minor in the grand scheme of most people's threat models.

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#66
post #11
post #6

Earlier quoted context omitted.

It's essentially a PXE-boot diskless environment, what makes you think it is unusual and possibility of being unreliable?

I should say, unsuitable for certain use cases.

Not a best setup to run database, yes. But stateless farm of servers that essentially forward the traffic for you? Not that much downsides.

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#67
post #8

One thing that I always wondered from VPNs. Let's say a pedophile uses Mullvad to get forbidden images, isn't the VPN liable? I mean, the law enforcement will see that the IP was from Mullvad's office, so I assume they are the ones doing it? How do they avoid this? It is a real doubt. Maybe stupid, but real.

IANAL but my understanding of current case law is that it IP address does not automatically mean a particular person.

Case law in what country? Mullvad is Swedish, are you knowledgeable about Swedish case law?

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#68

I wonder about those VPNs that say "we don't log or store anything". That may be the case, but they probably just send a continuous stream of data to the law enforcement / intelligence services or whoever instead of storing it themselves. They can then correctly say "WE don't log".

I formerly worked for a somewhat-older mainstream consumer VPN provider for a few years, to the extent that you can take my word for it, this is not industry-standard practice at least as far as the provider is able to control it.

Commercial VPNs typically run on rental servers -- usually a mix of the major cloud providers and smaller hosting providers -- and in my former company's case, using dedicated hosting (bare metal where available). Steps were taken to restrict access for physical actors, but ultimately, the mantra's always that physical access basically guarantees data access on a long enough timeline if you assume there's a bad actor in the mix.

That said, to the best of my memory, there were no indications of this kind of data siphoning happening without our knowledge, and we absolutely didn't take part in it ourselves knowingly. Occasional requests would come in from various international law enforcement orgs, and every time they'd be replied to with a message about how we don't store user records (which was a truthful reply AFAIK).

The biggest challenge for us was competing with some of the newer actors in the space, taking advantage of deceptive marketing and engaging in (IMO) unethical business practices for the sector:

- Claims of "no logging," even backed up by audits, are only ever point-in-time measurements, and may not reflect reality if the VPN provider approaches the auditors in bad faith (say, with a sanitized code base); a good auditor in my experience will refuse to make this claim in the report

- Claims about having the corporate HQ in one country making it immune from the laws of countries they operate servers in (this is deceptive marketing; failure to comply with laws will get you shut down, and at my old employer we'd make calls about whether to just drop our server presence in a country entirely in response to local laws and political happenings)

- Commercial resale of user data is (allegedly) rampant among many of the newer providers you see constantly plugged on Youtube. This isn't helped by the massive consolidation of the VPN market under just 2 or 3 holding companies.

I won't name names for the companies I mentioned above, but my recommendation is to adjust your threat model from "nation-state level surveillance" to "commercial data resale just like every other web service."

As far as data collection went for my old company: we collected system metrics like resource usage over time, and kept minimal sanitized logs to help diagnose any production issues that'd come up -- basically the absolute minimum amount of data we needed to keep the service operating smoothly. I have every reason to believe this is an industry norm, since otherwise development and troubleshooting would be nearly impossible.

Anyway, there's also the looming "threat" (lol) of HTTPS and encrypted DNS proliferation and improvement making the core use case for commercial VPNs obsolete. I think anyone who's spent a bit of time in that industry realizes that the business model isn't long for this earth as a result, so I suspect many are trying to milk the industry for all it's worth. Personally, I'm all for HTTPS and encrypted DNS proliferation, and I'm also hoping more and more commercial public networks start using virtual private subnets and other device isolation features to make it even harder to abuse coffee shop Wi-Fi.

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#69
post #65

Earlier quoted context omitted.

I’d agree with you about HTTPS providing most of the benefit that VPN advertising focuses on if I hadn’t seen repeated direct evidence that even most technical users will blithely click through HTTPS errors’ “accept the risk” bypass. It’s as if knowledgeable users think “sure, this could be a man in the middle attack, but it’s most likely just a benign cert problem, because certs are hard.” Sigh.

The problems with clicking past those errors are typically not due to network sniffing but with whatever crazy shit is on the page they are going to. The only two valid usecases of big VPNs like these are 1. Very mild security increase over public wifi 2. Shifting your risk from the ISP spying to mullvad or the VPN provider spying or slightly anonymizing if mullvad rotates IPs. (2) is a real benefit because ISPs are…

Yeah, I hate my ISP. I am certain they sell every bit of data they can. Ergo, I use a VPN most of the time.

Re: We have successfully completed our migration to RAM-only VPN infrastructure

#70
post #42

Earlier quoted context omitted.

> Some pour thousands of dollars into forcing influencers to say they care about security, Tangential to this, it always irks me how they talk about how they all act as if the majority of the websites their users are going to aren't HTTPS and they act like their main benefits are filling in the gaps that HTTPS actually fills in. HTTPS isn't a cure all by any means but most of the scare tactics that the big VPN compan…

I’d agree with you about HTTPS providing most of the benefit that VPN advertising focuses on if I hadn’t seen repeated direct evidence that even most technical users will blithely click through HTTPS errors’ “accept the risk” bypass. It’s as if knowledgeable users think “sure, this could be a man in the middle attack, but it’s most likely just a benign cert problem, because certs are hard.” Sigh.

I haven't experienced an HTTPS error on a legitimate site that I would input any personal information into in years.

I couldn't imagine clicking past one of those warnings to login to my bank or even amazon.

Post reply on HN