Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learning from a Year of Security Breaches

medium.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Learning from a Year of Security Breaches

#31

Post by haser_au »

mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

Can only speak about my corner of a very large organisation;

- Technical debt of custom coded solutions is a known issue across our organisation. New strategy is to move to market solutions, therefore outsourcing the risk to organisations with (hopefully) better code management than we have. For my corner, we don't have technical debt measured accurately enough for my liking.

- Yes, we pay for an use centralised logging. We've actually been through two solutions, and are now moving to a third due to various factors (cost, integrations, speed, out-of-the-box metrics). Integration into the centralised logging system is part of our Request for Tender marking criteria.

- Relatively good at disabling access after someone leaves. We integrate as much as possible to a central repository. It's just the outliers that tend to last beyond someone in the organisation. Critical systems are absolutely shutdown within 24 hours of a leaver departing (usually immediately if they're a bad leaver).

Edit: Formatting

Re: Learning from a Year of Security Breaches

#32

Post by tptacek »

haser_au wrote: post #31
mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

Can only speak about my corner of a very large organisation; - Technical debt of custom coded solutions is a known issue across our organisation. New strategy is to move to market solutions, therefore outsourcing the risk to organisations with (hopefully) better code management than we have. For my corner, we don't have technical debt measured accurately enough for my liking. - Yes, we pay for an use centralised logg…

Which logging systems did you like/not like?

Re: Learning from a Year of Security Breaches

#33

Post by capkutay »

acidbaseextract wrote: post #10

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking…

Please excuse the shameless plug but since you are asking for one, Striim is a good out of the box centralized logging solution. We use Kafka as our messaging layer (you can either install on your own Kafka or use our internal) and we use Elasticsearch as the storage layer.

We also have streaming log parsers to connect your data. That whole thing about 'creating new alerts in minutes' is trivial in our platform since everything is based in SQL.

Unlike Splunk or ELK, our solution is based on in-memory streams so you don't have to wait for data to be indexed to fire off alerts on anomalous activity. Feel free to message me to find out more or simply download the product from http://www.striim.com/

Re: Learning from a Year of Security Breaches

#34

Post by [deleted] »

tptacek wrote: post #25
lvh wrote: post #15

Earlier quoted context omitted.

You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible. CloudWatch works fine too. CloudWatch comes integrated with AWS services out of th…

I've had good luck with Cloudwatch and, if you're on AWS, I'd recommend it over any other hosted log system (with the possible suggestion of a more elaborate ELK setup that you build yourself). The trick to Cloudwatch is --- like most AWS services --- never using the web UI.

[deleted]

Re: Learning from a Year of Security Breaches

#35

Post by jonstewart »

It's interesting to see press leaks highlighted here as a pattern for insider threat. I don't doubt the author that this is so for the limited scope of organizations considered (SFBA tech companies), but I've worked on several insider cases, had insight into many more, and it's almost always an employee or ex-employee, with an axe to grind, taking trade secret information to a new job at a competitor. In many instances, the competitor has no idea and is pissed when they find out.

One piece of advice that I'd give out with such cases is to listen to your Spidey Sense. A lot of organizations will say, after the fact, "well... something didn't seem right with Bob...". If you sense something isn't right, prepare to secure evidence and analyze it. Don't put IT assets back into circulation if there's doubt, and don't sit on it.

Re: Learning from a Year of Security Breaches

#36

Post by [deleted] »

acidbaseextract wrote: post #10

Where do I start on centralized logging? I'm primarily an application developer, deployment isn't my strong suit. My hair is on fire at my current startup. There's a ton to do, we're trying to launch several new major efforts in January. What's a good plug and play solution that I don't have to think about? Are there hosted installs of Elasticsearch/Logstash/Kibana? Is ELK even what I want? Every time I start looking…

[deleted]

Re: Learning from a Year of Security Breaches

#37

Post by andrewflnr »

tptacek wrote: post #30
mag00 wrote: post #27

Earlier quoted context omitted.

I don't like the idea of keeping secrets in ENV and limiting it to config, though it's the kind of thing I'd ask other folks about myself to understand any tradeoffs. I see Kubernetes and other things supporting secrets in env variables so unsure how common it is. The big win is simply keeping secrets out of source code, out of an general engineer's copy/paste buffer, and with errors not going to a logging platform w…

If your code adopts a convention of reading secrets from the environment, you get a lot of flexibility in how they're actually stored; you can put them in protected files and export the contents of the file before running the service, or you can have a tool that works like "env" that populates from a secret store. Your secret storage system can get more sophisticated without your code having to change. I wouldn't rec…

Article and discussion re storing secrets in environment variables: https://news.ycombinator.com/item?id=8826024

The gist seems to be that it's easy to accidentally leak environment variables (which is why I think the top comment is off-base). tptacek, do you think this risk is overblown?

Re: Learning from a Year of Security Breaches

#38

Post by spydum »

mag00 wrote: post #16

Hi, I wrote this! To continue a discussion: - How does your engineering team track new "debt" after releasing code? (if at all, and why not) - Do you pay anyone for centralized logging, or wish you didn't? Are you making it useful? - Do you feel like your company is good at managing access when hiring / firing people? Otherwise thanks for any feedback, I enjoy writing these!

Great article, shared it on with my coworkers.

- poorly, really.

- for network, and security stuff, absolutely: splunk is the bees knees. For apps, each team tends to run their own mix (graylog2/elk/custom). Have pushed for more security type events from apps into splunk for correlation, but it just costs too damm much.

- depends on the region. I find US / UK do okay, but the more emerging/growth markets where we have employees, the worse it gets.

Re: Learning from a Year of Security Breaches

#39

Post by bm98 »

Ugh. A good and scary reminder of what's lurking around the corner for any of us at any time - including holidays and vacations (Linode's holiday attack last December comes to mind). IMHO, the emotional impact of breaches on the staff who respond to them is under-discussed. The author touches on it here:

> The discovery of a root cause is an important milestone that dictates the emotional environment an incident will take place in, and whether it becomes unhealthy or not.

> A grey cloud will hover over a team until a guiding root cause is discovered. This can make people bad to one another. I work very hard to avoid this toxicity with teams. I remember close calls when massive blame, panic, and resignations felt like they were just one tough conversation away.

Re: Learning from a Year of Security Breaches

#40

Post by tptacek »

andrewflnr wrote: post #37
tptacek wrote: post #30

Earlier quoted context omitted.

If your code adopts a convention of reading secrets from the environment, you get a lot of flexibility in how they're actually stored; you can put them in protected files and export the contents of the file before running the service, or you can have a tool that works like "env" that populates from a secret store. Your secret storage system can get more sophisticated without your code having to change. I wouldn't rec…

Article and discussion re storing secrets in environment variables: https://news.ycombinator.com/item?id=8826024 The gist seems to be that it's easy to accidentally leak environment variables (which is why I think the top comment is off-base). tptacek, do you think this risk is overblown?

It's good to be aware of the fact that environments are inherited by child processes (as are file descriptors), but I don't think that's a good reason to avoid using the environment.
Post reply on HN