Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning from a Year of Security Breaches

medium.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Learning from a Year of Security Breaches

#21

Post by ThePhysicist »

mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

Thanks for writing this, really insightful! A question: What's your advice on how to store secrets on the server-side?

Currently, I mainly use a seperate "secrets.yml" file that gets deployed via Ansible and is stored there encrypted using Ansible-Vault with a strong password. Is that a reasonable approach? What is your opinion about storing secrets in environment variables? It seems that some people advise this over storing them in files, but I have seen some cases where environment variables can be exposed to the web client as well.

Re: Learning from a Year of Security Breaches

#22

Post by user5994461 »

lvh wrote: post #11
user5994461 wrote: post #6

> I wasn’t roped into a single intrusion this year at any companies with completely role driven environments where secrets were completely managed by a secret store. > This can either mean one of a few things: These environments don’t exist at all, there aren’t many of them, or they don’t see incidents that would warrant involving IR folks like myself. What are these secrets store? Do they exist?

In general, secret stores "manage secrets so that you don't have to". That can mean a few things, depending on who's using the term. Sometimes, it's as simple as a shared password store (I've used one powered by GPG, for example). This is better than YOLO password policy, but not by much: humans still see individual keys. If you want to be really fancy, you authenticate the human and then decide what they get to do,…

Still need a secret to access the secret store... so steal the secret then steal the secrets in the store.

I fail to see how it is secured. (Though, I can understand that it is less bad than a YOLO policy).

> Many organizations get pretty close -- I'm told that the DoD pretty much authenticates everything with smart cards, for example.

I've been at a place with RSA SecurID (smart card and OTP) + active directory account as SSO authentication for everything (use one or both for 2FA). It was nice and well done.

Re: Learning from a Year of Security Breaches

#23

Post by lvh »

user5994461 wrote: post #22
lvh wrote: post #11

Earlier quoted context omitted.

In general, secret stores "manage secrets so that you don't have to". That can mean a few things, depending on who's using the term. Sometimes, it's as simple as a shared password store (I've used one powered by GPG, for example). This is better than YOLO password policy, but not by much: humans still see individual keys. If you want to be really fancy, you authenticate the human and then decide what they get to do,…

Still need a secret to access the secret store... so steal the secret then steal the secrets in the store. I fail to see how it is secured. (Though, I can understand that it is less bad than a YOLO policy). > Many organizations get pretty close -- I'm told that the DoD pretty much authenticates everything with smart cards, for example. I've been at a place with RSA SecurID (smart card and OTP) + active directory acco…

You made that point elsewhere in the comments; and I replied to it there; for the benefit of other people wondering why a secret store _isn't_ just robbing Peter to pay Paul: https://news.ycombinator.com/item?id=13224802

Re: Learning from a Year of Security Breaches

#24

Post by jasonlotito »

mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

You said this: "Rarely do I see a team eliminate all of their debt, but the organizations _that least respect_ their debt never get so far behind that they can no longer be helped in a breach."

Do you mean instead "that _at_ least respect"?

I ask only because they two have different meanings.

Re: Learning from a Year of Security Breaches

#25

Post by tptacek »

lvh wrote: post #15
acidbaseextract wrote: post #10

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking…

You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible. CloudWatch works fine too. CloudWatch comes integrated with AWS services out of th…

I've had good luck with Cloudwatch and, if you're on AWS, I'd recommend it over any other hosted log system (with the possible suggestion of a more elaborate ELK setup that you build yourself).

The trick to Cloudwatch is --- like most AWS services --- never using the web UI.

Re: Learning from a Year of Security Breaches

#26

Post by lvh »

tptacek wrote: post #25
lvh wrote: post #15

Earlier quoted context omitted.

You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible. CloudWatch works fine too. CloudWatch comes integrated with AWS services out of th…

I've had good luck with Cloudwatch and, if you're on AWS, I'd recommend it over any other hosted log system (with the possible suggestion of a more elaborate ELK setup that you build yourself). The trick to Cloudwatch is --- like most AWS services --- never using the web UI.

That's a good point! If you have someone consuming it that wants a (shared) web UI, you want Kibana. If they prefer to consume their text in a terminal and are fine with typing `aws logs` a bunch, CloudWatch is fine (and probably a little less twiddly than ELK).

Re: Learning from a Year of Security Breaches

#27

Post by mag00 »

ThePhysicist wrote: post #21
mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

Thanks for writing this, really insightful! A question: What's your advice on how to store secrets on the server-side? Currently, I mainly use a seperate "secrets.yml" file that gets deployed via Ansible and is stored there encrypted using Ansible-Vault with a strong password. Is that a reasonable approach? What is your opinion about storing secrets in environment variables? It seems that some people advise this over…

I don't like the idea of keeping secrets in ENV and limiting it to config, though it's the kind of thing I'd ask other folks about myself to understand any tradeoffs. I see Kubernetes and other things supporting secrets in env variables so unsure how common it is.

The big win is simply keeping secrets out of source code, out of an general engineer's copy/paste buffer, and with errors not going to a logging platform with single factor access. Your likelihood of a short term incident decreases dramatically. Especially if those secrets have well segmented access, (IE, not a single AWS key with `AdministratorAccess` everywhere).

Re: Learning from a Year of Security Breaches

#28

Post by mag00 »

jasonlotito wrote: post #24
mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

You said this: "Rarely do I see a team eliminate all of their debt, but the organizations _that least respect_ their debt never get so far behind that they can no longer be helped in a breach." Do you mean instead "that _at_ least respect"? I ask only because they two have different meanings.

Yep, fixing

Re: Learning from a Year of Security Breaches

#29

Post by mzzter »

ThePhysicist wrote: post #21
mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

Thanks for writing this, really insightful! A question: What's your advice on how to store secrets on the server-side? Currently, I mainly use a seperate "secrets.yml" file that gets deployed via Ansible and is stored there encrypted using Ansible-Vault with a strong password. Is that a reasonable approach? What is your opinion about storing secrets in environment variables? It seems that some people advise this over…

Have you heard of torus.sh keyrings? I don't know how well it works for an organization, but integrating torus into my side projects has been painless.

Re: Learning from a Year of Security Breaches

#30

Post by tptacek »

mag00 wrote: post #27
ThePhysicist wrote: post #21

Earlier quoted context omitted.

Thanks for writing this, really insightful! A question: What's your advice on how to store secrets on the server-side? Currently, I mainly use a seperate "secrets.yml" file that gets deployed via Ansible and is stored there encrypted using Ansible-Vault with a strong password. Is that a reasonable approach? What is your opinion about storing secrets in environment variables? It seems that some people advise this over…

I don't like the idea of keeping secrets in ENV and limiting it to config, though it's the kind of thing I'd ask other folks about myself to understand any tradeoffs. I see Kubernetes and other things supporting secrets in env variables so unsure how common it is. The big win is simply keeping secrets out of source code, out of an general engineer's copy/paste buffer, and with errors not going to a logging platform w…

If your code adopts a convention of reading secrets from the environment, you get a lot of flexibility in how they're actually stored; you can put them in protected files and export the contents of the file before running the service, or you can have a tool that works like "env" that populates from a secret store. Your secret storage system can get more sophisticated without your code having to change.

I wouldn't recommend putting them in /etc/environment or /etc/profile or /home/service/.profile where you'll forget about them, though.

Just as a strategy for passing secrets to code, I like the environment a lot.

Post reply on HN