31–40 of 92 posts
Can you explain the nginx/HAproxy config a little more?
Nginx is great for...pretty much everything else.
Great post, but this part scares me a bit... I think a lot of services (even banks!) have serious security problems and seem to be able to weather a small PR storm. So figure it out if it really is important to you (are you worth hacking? do you actually care if you’re hacked? is it worth the engineering or product cost?) before you go and lock down everything. Just because you can "afford" to be hacked, doesn't mean…
Systems get compromised, it happens. Organizations with weak security architectures can become so compromised that cleanup becomes a nightmare because it is difficult to isolate the threat(s) without serious disruption in services. A strong security architecture is not so much to ensure breaches never happen but to limit the amount of damage likely to occur when breaches do happen.
And yes, this happens even to organizations that think they have nothing worth hacking.
Great post, but this part scares me a bit... I think a lot of services (even banks!) have serious security problems and seem to be able to weather a small PR storm. So figure it out if it really is important to you (are you worth hacking? do you actually care if you’re hacked? is it worth the engineering or product cost?) before you go and lock down everything. Just because you can "afford" to be hacked, doesn't mean…
Yeah, that point significantly underestimates the cost of cleaning up once your systems have been penetrated. By the time you notice that one system has been compromised, there is no guarantee that every system at your company is not compromised, particularly if so little effort is put into a robust security architecture. I've seen companies that took the attitude the author does and ended up paying for it down the r…
Having internal firewalls between servers that don’t need to talk to each other — again a good idea. But if your service doesn’t actually need this, don’t necessarily do it
I can not think of any reason why "your service doesn't actually need this" and "don't necessarily do it". I understand that it costs money to do these things, but setting up a firewall is relatively cheap, significantly less than the cost of the additional cleanup if the breach is not contained.
Security, in a way, can be compared to insurance. Sure, if you are young and live a healthy life style you may not necessarily see the need to spend $100+ a month for a health insurance policy, you can save a bunch of money... but if an accident does happen, you can rest assured it will cost you significantly more than if you had just bought the insurance in the first place.
This, in a sense, is the security tradeoff.
I think really smart engineers who are well versed in security can know where security needs to be, and yes it is possible to go overboard, but I think this is the exception rather than the rule. Advising readers that it's ok to not worry too much about security because:
lot of services (even banks!) have serious security problems
is absolutely ridiculous and is horrible advise.
Earlier quoted context omitted.
Says an ex "Product Manager at Dropbox". Edit: Thanks for the downvotes. My point is, just make it unambiguous to everyone in your comment so we don't have to click through your profile. Context matters. e.g.: "I was Product Manager at Dropbox and worked with Rajiv (the OP). He's awesome, you should listen to him." Much better.
which means his opinion counts at least 100 times more than yours does.
I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have Why not take the extra half a second to make those random strings meaningful and hidden behind a DEBUG log level?
Probably most of their logging _is_ meaningful, but deciding how to professionally phrase each and every log message will eventually get you to decision fatigue. The point that he was making with this was that over-logging is a good thing - this probably wasn't something the initial author thought was going to be terribly informative, hence the random string. And yet it ended up diagnosing a real world problem. In a…
The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you…
I actually purposely used to set all the various clocks at home ahead by anywhere from 0 - 15 minutes. At first, I could remember which ones were ahead by how much, but then soon I started to forgot and had to just assume they were running at the right time. It worked great. After a few years of this, I set them all back to right time and found that I had trained myself to just leave at the right time, with no more t…
Once I had three wake up alarms, at different points at the bedroom. Didn't work.
Being late is lame. Suffering its consequences is the best teacher one can have.
I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have Why not take the extra half a second to make those random strings meaningful and hidden behind a DEBUG log level?
Let he who has never written a frustrated, nonsense, print statement throw the first stone, if you will.
Earlier quoted context omitted.
Probably most of their logging _is_ meaningful, but deciding how to professionally phrase each and every log message will eventually get you to decision fatigue. The point that he was making with this was that over-logging is a good thing - this probably wasn't something the initial author thought was going to be terribly informative, hence the random string. And yet it ended up diagnosing a real world problem. In a…
I don't know how many new logging statements you commit to production code every day, but I can't imagine it averages out to more than one or two. If you can't take the time to phrase them both professionally and meaningfully then you're doing yourself and your team a disservice.
At the time of writing the code, you're hopefully thinking through "how could this fail?"
There's your log message.