Live data from Hacker News

The FastMail Security Mindset

blog.fastmail.com

241–250 of 301 posts

Re: The FastMail Security Mindset

#241
post #219

Earlier quoted context omitted.

Wow, that's a lot of questions, and I can't answer all of them without creating security risks! Our absolute focus is on minimizing the human factors. In the past year and a bit since that incident, we have improved our escalation policies and support training, as well as let some support staff go. But more importantly, we now have an automated account recovery system which can be used to verify ownership of the acco…

Wow, that's a lot of questions, and I can't answer all of them without creating security risks Questions like these are not unreasonable for a customer to ask a service provider with respect to identity management and protection of that customer’s proprietary and confidential information. With respect to the first question “Exactly which employees have the ability to alter recovery email settings.” Not being able to…

Which employees? At the time of this compromise, that list was all support staff as well as the technical staff in Melbourne. It is a specific role that's granted to specific people, to answer your question about having a procedure or policy.

Today, that role is granted to a much more limited set of senior security staff (currently 3 people). Regular support staff can not alter security-sensitive details about accounts. If your account is owned by someone else (e.g. family or business, or part of a resold package) then they can still alter recovery options, as they own the account.

In 2016 before we had automated account recovery, lost password was in the top 3 categories of ticket every single week! Every member of the support team dealt with multiple account-loss tickets per day, both forgotten password or stolen account.

Stolen account losses are way down now we have app passwords, we often only have to block a single app password and notify the user rather than locking the entire account. Forgotten passwords have not reduced, but most people are able to recover using the automated tooling.

---

In what fashion do we audit and track? A few ways - we log every API call at the lowest level. We log each override when the support person accesses user accounts against the ticket that they come in through, so we can see why they were accessing that user.

We could always do with better tooling to introspect logs, but the data is all captured and can be followed through after the fact. Support staff have no way to wipe their audit trail.

---

Training - in 2016 we didn't have much formal training for our support staff - they learned on the job from each other. We are very aware that this was a failing at that time.

We have more training now. We did a lot of work at unifying our support teams across the FastMail and Pobox/Listbox family throughout 2017, and that led to better training and induction materials, as well as better internal reference material for support staff to use.

Early in the induction process for all new support staff is a description of how social engineering works and a warning that urgency is often used in social engineering attempts, so when in doubt, slow down and get a second opinion (which leads to complaints about slow support, but that's the tradeoff here.)

---

Escalation process - as mentioned earlier, if you had 2fa enabled then it has always gone straight to the senior security team, which is based in Melbourne and consists of our most experienced and trusted people. Neil (author of the blog post this HN refers to) is of course one of these people.

With lost passwords no longer a highly common support request, all support tickets requesting manual account recovery are escalated to our senior team for review.

---

Support people have no incentive to close tickets quickly. Absolutely. That is a bad metric, and it's not a metric we have ever used.

Time to first response and time to followup responses are tracked, but there's no incentive to close tickets.

This answer is a no brainer and I should have answered it in the first response - sorry. I was still rushing through initial responses at that time, and there were too many points in that post to think about them all at once and still respond quickly. The real-time nature of this hacker-news medium encourages fast answers above complete answers. I hope this longer response helps clear up remaining questions, at least to those who see it!

---

There's another blog post coming soon about the account recovery system in particular, which addresses exactly how we've minimising human involvement in recovery decisions while not excessively punishing real human frailty amongst our customers.

Re: The FastMail Security Mindset

#242

Earlier quoted context omitted.

Good morning. I'm going to be here to answer specific questions, and I owe you a personal response to this as well, which I'm about to start working on! There is no doubt that in this specific case our human factor screwed up, and I'm really sorry about that. First I'm going to post the standard response that our team has written for any new support tickets that come in about this today, then write my own personal ap…

Exactly which employees in your organization have the ability to alter recovery email settings? How many of those employees are there? In what fashion do you audit and track the activities of those employees? What training are these employees given to avoid social engineering? What firm provides the courseware? What's the escalation process for complicated, non-no-brainer reset situations? If a support person isn't a…

Replying directly because I'm not sure if you'll get notified otherwise, but here's a longer response to this: https://news.ycombinator.com/item?id=15859024

Re: The FastMail Security Mindset

#243

Earlier quoted context omitted.

'I'm an Australian, and I'm a great fan of our "100 points of ID" system, which is designed to remove the human factor from identifying people.' According the page you linked, a birth certificate and bank statement (or even a 'Document issued by or registered corporations.') would be enough. So if I get your birth certificate and have an Australian corporation, I can issue a letter saying you're a customer for a year…

We don't use the 100 points of ID system of course, because we're an online service. The 100 points of ID is something that's used in person to decide whether you can open a new bank account using that name. The concept behind the 100 points of ID is that there's a fixed standard and it's not a per-time decision made by a human, it's a consistent set of rules applied without fear or favour.

It's a consistent, fixed standard that you hope you have trained your employees to adhere. Fingers crossed. /s

Re: The FastMail Security Mindset

#244
post #181

Earlier quoted context omitted.

If a well-resourced attacker was targeting me specifically, it wouldn't be too difficult for them to find out about my short-to-medium term travel plans. A bit of social engineering with the airlines could tell them exactly which flight I'm on. They could also compromise other people who need to know my plans and don't have the same security practises as me. I think about this stuff and minimise as best I can, but my…

If a "well-resourced attacker" was targeting you specifically, you're toast. Period.

I wish more people understood this.

Re: The FastMail Security Mindset

#245
post #165

Earlier quoted context omitted.

Wow, that's a lot of questions, and I can't answer all of them without creating security risks! Our absolute focus is on minimizing the human factors. In the past year and a bit since that incident, we have improved our escalation policies and support training, as well as let some support staff go. But more importantly, we now have an automated account recovery system which can be used to verify ownership of the acco…

I'm a FastMail customer. Your response is troubling to me in that it didn't answer most of tptacek's questions. It's troubling enough for me to start looking at other email providers. :-( I would like FM to provide something akin to Google's advanced protection program. Those of us who are careful not to lose our login credentials should not have to suffer a weak recovery process for the convenience of those who do.…

I have now responded in more detail - at the time I was busy trying to spread the love around, and also support my team as they dealt with the support requests and digesting the response on here.

Check out the longer response here:

https://news.ycombinator.com/item?id=15859024

Re: The FastMail Security Mindset

#246
post #219

Earlier quoted context omitted.

Wow, that's a lot of questions, and I can't answer all of them without creating security risks Questions like these are not unreasonable for a customer to ask a service provider with respect to identity management and protection of that customer’s proprietary and confidential information. With respect to the first question “Exactly which employees have the ability to alter recovery email settings.” Not being able to…

Which employees? At the time of this compromise, that list was all support staff as well as the technical staff in Melbourne. It is a specific role that's granted to specific people, to answer your question about having a procedure or policy. Today, that role is granted to a much more limited set of senior security staff (currently 3 people). Regular support staff can not alter security-sensitive details about accoun…

hi Bron, thank you for this response. Much clearer and I think this is what everyone wanted to see.

Can I just clarify some things for peace of mind?

1) When you say regular support staff cannot alter security-sensitive details. How is that done? Do they only perform changes through a limited set of UI?

2) When you say if 2fa is enabled it goes to senior security team, is that an automated process such that support staff don't see that ticket at all? The support ticket interface doesn't seem to have anything that helps to automatically route password reset requests.

3) Was the security incident involving ghouse through support tickets?

4) Do the senior security team have direct data access? i.e. do they also change things through a UI or do they have capability to directly change data?

Thanks

Re: The FastMail Security Mindset

#247
post #213

Earlier quoted context omitted.

Could you share some info/links on what makes it the Gold Standard?

They have one of the largest information security teams in the world, that team includes what is probably the best corporate vulnerability research team in the world. They're one of a small number of companies that is actively defining modern TLS and thus modern transport encryption; their operations and security teams are almost certainly the world's most sophisticated users of TLS. They ship the most secure browser…

They have every incentive to ensure the highest security possible. Their entire business model and most of their revenue is predicated on consumers and businesses moving not just some, but all of their data, straight over to Google's custody and control. Indeed, it damn well had better be secure.

But I think they're compromised by those same business models. Google wants to provide intelligence, and probably more important to them, marketing data. This requires that the consumer is an open book to them, and their business decisions incorporate that. Up until recently, they were actively scanning email for marketing insights. In addition, Google's operating complexity, both business and technical, increases the opportunity for failure. And their other business objectives compromise their security work. That's glaringly apparent for their Android platform. There's more surface. And in a Google world, the email account grants direct access to everything — location data, purchasing history, passwords, documents... everything.

For another dedicated email provider, what they have to protect is also simpler. There are fewer moving parts. There's less to protect, which means that there don't need to be as many engineers. That means a careful and well thought out email provider /can/ be as secure, by carefully limiting their exposure, doing one thing, and doing it well.

There's something to be said for careful application of open standards and open source software, a smaller and more responsive team, and not building a massive single point of failure. I am a current Fastmail customer, and hope to remain, depending on the outcome of this review.

Re: The FastMail Security Mindset

#248

Earlier quoted context omitted.

Now to write a more detailed response. If this winds up out of order later, I first posted: https://news.ycombinator.com/item?id=15856609 Again, ghouse, I'm really really sorry about what happened to your account. It was wrong and we screwed up. As other comments have already noted, it was during the transition to a new security system which was designed precisely to remove the human factor from decision making. I'm…

I use Fastmail and I like it but this is extremely disturbing. I appreciate you being relatively candid with us, but it doesn’t change the fact that you allowed a customer’s account to be compromised by the most basic attack out there. Complexity and vulnerability go hand-in-hand. A product providing a critical service like email should be opt-in to any form of recovery not requiring pure secrets provided directly by…

https://blog.fastmail.com/2017/12/06/security-account-recove... is probably what you're looking for. I've also made additional responses in this thread.

Re: The FastMail Security Mindset

#249
post #201

Earlier quoted context omitted.

Unless you have a set of objectives that are very different from what I consider "as secure as e-mail gets", please consider GSuite and not Protonmail. (I don't speak for 'tptacek, but I'm pretty sure he'd agree.) As a corollary: if you really care, use Signal for stuff you can't say over e-mail. Whatsapp's fine too. But they solve a very different security problem than the one you need e-mail to solve, which is most…

Just gonna drive by mention https://landing.google.com/advancedprotection/ , which is a physical-2fa-security-key-only version of gmail. To my knowledge it also disallows mail forwarding, and the account recovery procedure in the event of losing both second factors is intended to be a long process that involves proof of identity and multiple attempts to notify the account owner. (I work on gmail, but I'm not intimate…

The pricing is very unclear from even three clicks in from that link, only that it is some sort of add on service.

Re: The FastMail Security Mindset

#250

Earlier quoted context omitted.

Now to write a more detailed response. If this winds up out of order later, I first posted: https://news.ycombinator.com/item?id=15856609 Again, ghouse, I'm really really sorry about what happened to your account. It was wrong and we screwed up. As other comments have already noted, it was during the transition to a new security system which was designed precisely to remove the human factor from decision making. I'm…

The blog post was in 2014. This security bypass happened in 2016. I think what we're witnessing here is that despite best intentions and past experience, humans are going to be humans. I actually felt good after reading that blog post in 2014 thinking that you guys are going to be better than most companies here. Nope. But I think a lesson can be learned here. The lesson is simply that humans are the weakest link. As…

https://blog.fastmail.com/2017/12/06/security-account-recove... - it's currently 3 people who have that ability. It had to be more before we had the automated tooling, those three people couldn't handle 3 figures per day (I'm not kidding) of regular password losses by regular users.
Post reply on HN