Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling lessons learned at Dropbox, part 1

eranki.tumblr.com

Post reply on HN
Print view View original

21–30 of 92 posts

Re: Scaling lessons learned at Dropbox, part 1

#21

Post by ephemeralgomi »

akent wrote: post #14

I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have Why not take the extra half a second to make those random strings meaningful and hidden behind a DEBUG log level?

Probably most of their logging _is_ meaningful, but deciding how to professionally phrase each and every log message will eventually get you to decision fatigue.

The point that he was making with this was that over-logging is a good thing - this probably wasn't something the initial author thought was going to be terribly informative, hence the random string. And yet it ended up diagnosing a real world problem.

In a perfect world, by all means properly write out your messages - but if you're stalling on a log message because you're not sure how to phrase it, you may get concrete benefit from just dropping a FUUUCCKKKKKasdjkfnff and moving on.

Re: Scaling lessons learned at Dropbox, part 1

#22

Post by joshma »

brc wrote: post #11

The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you…

It's not exactly the same, though. If you're late 5 minutes by your watch you're on time naturally, without any action. If you're "overloaded" on your servers, you at least have to consciously decrease the extra load or risk having real consequences.

Re: Scaling lessons learned at Dropbox, part 1

#23

Post by apu »

brc wrote: post #11

The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you…

I actually purposely used to set all the various clocks at home ahead by anywhere from 0 - 15 minutes. At first, I could remember which ones were ahead by how much, but then soon I started to forgot and had to just assume they were running at the right time. It worked great.

After a few years of this, I set them all back to right time and found that I had trained myself to just leave at the right time, with no more trickery needed.

Re: Scaling lessons learned at Dropbox, part 1

#24

Post by Game_Ender »

brc wrote: post #11

The idea of running extra load - it sounds good in theory but I can't help thinking that it's a bit like setting your watch forwards to try and stop being late for things. Eventually you know your watch is 5 minutes fast so start compensating for it. I wonder if this strategy starts to have the same effect - putting fixes off because you know you can pull the extra load before it becomes critical. In the same way you…

This is just an adaption of the "margin" system commonly used in systems engineering. Basically you never spec everything out to perfectly match the load it's expected to bare, always leave some headroom.

For example if you are designing an aircraft your first design is never perfect. So you when you do the initial design, you do it as if the aircraft has to weight 70% of what it really will. As errors are corrected in your original design (or features creep in) you will slowly eat away at that 30% margin. Hopefully by the time you finish the have some left, or the aircraft will never get off the ground.

Re: Scaling lessons learned at Dropbox, part 1

#25

Post by christoph »

ephemeralgomi wrote: post #21
akent wrote: post #14

I noticed that a particular “FUUUCCKKKKKasdjkfnff” wasn’t getting printed where it should have Why not take the extra half a second to make those random strings meaningful and hidden behind a DEBUG log level?

Probably most of their logging _is_ meaningful, but deciding how to professionally phrase each and every log message will eventually get you to decision fatigue. The point that he was making with this was that over-logging is a good thing - this probably wasn't something the initial author thought was going to be terribly informative, hence the random string. And yet it ended up diagnosing a real world problem. In a…

So, so true.

When the problem occurs, it's pretty quick for the guy who needs to fix it at 2am, to find where it exploded in the code base, while the original developer is (maybe) passed out in a bar somewhere.

Not much else matters. He could of just done :( x 10 and had the same result. The main thing is, it's easily traceable!

Re: Scaling lessons learned at Dropbox, part 1

#26

Post by akent »

christoph wrote: post #25
ephemeralgomi wrote: post #21

Earlier quoted context omitted.

Probably most of their logging _is_ meaningful, but deciding how to professionally phrase each and every log message will eventually get you to decision fatigue. The point that he was making with this was that over-logging is a good thing - this probably wasn't something the initial author thought was going to be terribly informative, hence the random string. And yet it ended up diagnosing a real world problem. In a…

So, so true. When the problem occurs, it's pretty quick for the guy who needs to fix it at 2am, to find where it exploded in the code base, while the original developer is (maybe) passed out in a bar somewhere. Not much else matters. He could of just done :( x 10 and had the same result. The main thing is, it's easily traceable!

Sure, for fatal errors, random (greppable) strings aren't so bad, but the OP made it sound like his FFFFFFFFFFFUUUUUU message was expected all the time rather than an exception. If you're going to print something all the time in normal operation, make it meaningful.

Re: Scaling lessons learned at Dropbox, part 1

#27

Post by bestes »

nl wrote: post #16

I wish he'd left the security advice out. The whole post was excellent, but all the useful points will now be overshadowed by the armchair quarterbacking about security by people who mostly don't understand that ALL security is a compromise, and it is as important to understand and make deliberate decisions about your security as it is to try to make a secure system in the first place.

I'm glad he put the security notes in. It is so hard to get true facts about how things are actually done.

Re: Scaling lessons learned at Dropbox, part 1

#28

Post by stratos2 »

akent wrote: post #12
ivankirigin wrote: post #5

Rajiv is awesome, you should listen to him

Says an ex "Product Manager at Dropbox". Edit: Thanks for the downvotes. My point is, just make it unambiguous to everyone in your comment so we don't have to click through your profile. Context matters. e.g.: "I was Product Manager at Dropbox and worked with Rajiv (the OP). He's awesome, you should listen to him." Much better.

which means his opinion counts at least 100 times more than yours does.
Post reply on HN