Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning from a Year of Security Breaches

medium.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Learning from a Year of Security Breaches

#11

Post by lvh »

user5994461 wrote: post #6

> I wasn’t roped into a single intrusion this year at any companies with completely role driven environments where secrets were completely managed by a secret store. > This can either mean one of a few things: These environments don’t exist at all, there aren’t many of them, or they don’t see incidents that would warrant involving IR folks like myself. What are these secrets store? Do they exist?

In general, secret stores "manage secrets so that you don't have to". That can mean a few things, depending on who's using the term.

Sometimes, it's as simple as a shared password store (I've used one powered by GPG, for example). This is better than YOLO password policy, but not by much: humans still see individual keys.

If you want to be really fancy, you authenticate the human and then decide what they get to do, in a centralized fashion. This is often tricky to do, because you either don't have the funds to do that if you're small, or you have too many services to interact with if you're big. (Many organizations get pretty close -- I'm told that the DoD pretty much authenticates everything with smart cards, for example.)

Sometimes, it means a more automated system where software authenticates instead of a human, and it gets e.g. a certificate. Usually this is still always the same certificate, though; so the main difference is that it's a human versus a machine authenticating.

Sometimes, it means an HSM (hardware security module). These are secure physical devices that perform cryptographic operations for you, so that the key stays on the device.

Re: Learning from a Year of Security Breaches

#12

Post by ben_jones »

acidbaseextract wrote: post #10

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking…

I'm in the same boat. Looking for recommendations on strong, sturdy, buckets, for bailing water.

Re: Learning from a Year of Security Breaches

#13

Post by d4mi3n »

acidbaseextract wrote: post #10

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking…

TFA aside, centralized logging is super useful for debugging a variety of issues. There are a number of hosted options, and setting them up isn't too hard. It usually involves configuring you're application's log device to talk to the remote service, or configuring syslog on your app servers to forward logs to said service.

See https://logentries.com/ for an example

Re: Learning from a Year of Security Breaches

#14

Post by theluketaylor »

acidbaseextract wrote: post #10

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking…

If you want a pretty prepackaged solution you could do a lot worse than splunk. They even offer it SaaS

https://www.splunk.com/en_us/cloud.html

Re: Learning from a Year of Security Breaches

#15

Post by lvh »

acidbaseextract wrote: post #10

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking…

You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible.

CloudWatch works fine too. CloudWatch comes integrated with AWS services out of the box. It can be more annoying to get your logs into it than ELK (the latter seems overall more popular). Its alerting and the AWS CLI integration pretty slick, though.

You should also go turn on CloudTrail right now. It lets you automatically log side-effectful API calls. It is not a replacement for a centralized logging pipeline, but it's great high-signal data to put into one.

I appreciate that your complaint (totally valid!) was "this is a rabbit hole", and I just gave you two options, and that might not help your perception that it's a rabbit hole. If you find yourself paralyzed by choice, either choice is much better than deferring the choice! Just pick one. Heck, if you can't pick, let me help: pick AWS hosted Elasticsearch.

A lot of people (also in the security space) like Splunk. I find it annoying to deploy (I've heard rsyslog-in-front-of-forwarders as a canonical deployment method for just ingesting syslog more than once because reasons) and overpriced. YMMV.

Disclaimer: shameless plug! You're not the only one with your hair on fire. One of the first things we're doing for Latacora customers is setting up a centralized logging pipeline.

Re: Learning from a Year of Security Breaches

#16

Post by mag00 »

Hi, I wrote this!

To continue a discussion:

  - How does your engineering team track new "debt" after releasing code? (if at all, and why not)
  - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful?
  - Do you feel like your company is good at managing access when hiring / firing people?
Otherwise thanks for any feedback, I enjoy writing these!

Re: Learning from a Year of Security Breaches

#17

Post by dsacco »

mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

Thank you for writing these. These blog posts are my go-to resources when my client companies want to learn more about what they can do to improve their security posture long term. It's a really great series.

Re: Learning from a Year of Security Breaches

#18

Post by user5994461 »

dsp1234 wrote: post #9
user5994461 wrote: post #6

> I wasn’t roped into a single intrusion this year at any companies with completely role driven environments where secrets were completely managed by a secret store. > This can either mean one of a few things: These environments don’t exist at all, there aren’t many of them, or they don’t see incidents that would warrant involving IR folks like myself. What are these secrets store? Do they exist?

For example, Hashicorp Vault[0] [0] - https://www.vaultproject.io/intro/index.html

Then people need secrets to access the secret store and you're back at square one ;)

Re: Learning from a Year of Security Breaches

#19

Post by dsacco »

lvh wrote: post #15
acidbaseextract wrote: post #10

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking…

You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible. CloudWatch works fine too. CloudWatch comes integrated with AWS services out of th…

I second ELK, and I even stronger-second Splunk being overpriced (with the caveat that if you do deploy it, I think it's the best option, just not really worth it).

I think it's really important to internalize the idea that there is no Platonic ideal of a logging solution. It's a fundamentally frustrating manifestation of entropy that you're going to wrestle with, but it's a really necessary goal to work towards long term. Sort of a "the first step is admitting powerlessness" kind of deal.

Re: Learning from a Year of Security Breaches

#20

Post by lvh »

user5994461 wrote: post #18
dsp1234 wrote: post #9

Earlier quoted context omitted.

For example, Hashicorp Vault[0] [0] - https://www.vaultproject.io/intro/index.html

Then people need secrets to access the secret store and you're back at square one ;)

There are plenty of things a secret store still buys you.

- It knows how to encrypt and store secrets securely. Having one specialized application have an opinion on how to do that is much better than having a hundred ones that do it incidentally. The central one will be audited and monitored. The hundreds will invariably mess it up.

- It tracks who accessed a secret and when. This is critical information for remediation and ongoing scope reduction. Knowing who accessed what, when gives you the context for why; all three tell you how to further reduce the authority that application has.

- It can generate "minimal" credentials on-demand. I.e. a new key that only lets you access what you need and for a limited amount of time.

- It can encrypt things on behalf of the requester, such that the requester never sees the key. That is good, because it can be one-way. It is also good because if a service is compromised, the compromise may be detected and remediated (access revoked) before all data is dumped and compromised. Having the secret store lets you do e.g. rate limiting and centralized monitoring, for example.

- Secret stores can know how secrets are linked; making it easier to do revocation, and easier to determine the impact of a breach or misuse incident.

Post reply on HN