How Facebook Ships Code • Hacker News Board

Re: How Facebook Ships Code

#61

Post by dot » Thu, Jul 05, 2012, 1:23 PM UTC

alttab wrote: post #54

yawgmoth wrote: post #43

I like the idea of encouraging a high-performance culture, but I don't think the 'perform or die' atmosphere would be healthy for many engineers. I know, idolizing 'rockstar programmers' is a sort of new hotness and I understand that a company like Facebook wants to have super-talented developers, but developers grow and learn new tricks as they mature, and they might take more than six months to do so.

True - but those engineers don't work at Facebook. This isn't no-child left behind. This isn't hand-hold time. This is the most expensive and expansive internet application in the world. If they need time to ramp up, they can do it on someone else's product and come to Facebook when they're ready.

the most expensive internet application? what do you mean?

Re: How Facebook Ships Code

#62

Post by sreyaNotfilc » Thu, Jul 05, 2012, 1:42 PM UTC

alttab wrote: post #51

sreyaNotfilc wrote: post #40

This may be my favorite quote... "Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything t…

I agree. Specialists are really just full stack engineers who haven't fulfilled their potential. Why hire a self-proclaimed UX guy? Unless I have all the amazing engineers I need for my team (99% not the case), I'm looking for someone who can do multiple things. The best ones are ones that can do schema design, query design, system and model architecture, front-end wireframes, javascript, and implementation. That sai…

What I mean by "tightly woven" is that you know exactly what values/components/types need to communicate with each other whether its the front end or the back end. You're not wasting time trying to match a spec sheet or fulfill a precondition to an existing back-end procedure (cause honestly those things just don't fit right in). That is, you're not spending time trying to put a square peg into a round hole.

Re: How Facebook Ships Code

#63

Post by wpietri » Thu, Jul 05, 2012, 1:44 PM UTC

alttab wrote: post #54

yawgmoth wrote: post #43

I like the idea of encouraging a high-performance culture, but I don't think the 'perform or die' atmosphere would be healthy for many engineers. I know, idolizing 'rockstar programmers' is a sort of new hotness and I understand that a company like Facebook wants to have super-talented developers, but developers grow and learn new tricks as they mature, and they might take more than six months to do so.

True - but those engineers don't work at Facebook. This isn't no-child left behind. This isn't hand-hold time. This is the most expensive and expansive internet application in the world. If they need time to ramp up, they can do it on someone else's product and come to Facebook when they're ready.

That sounds impressively macho, but it's an attitude that has long-term organizational costs. HN just had a great article on how Microsoft's internal competition deeply harmed the company. The number one complaint I hear from departing Google engineers is the absurd internal promotions system.

Years ago I did a gig at eBay, and I thought their macho attitude was a giant source of problems. Plenty of good, sane people were driven off (or driven mad) by artificially high-pressure situations. Every email about a promotion mentioned how somebody had worked all night to get something done; they were promoting more for drama than for skill. And a "this isn't hand-hold time" attitude was common among senior technical staff, which meant that people often hid their weaknesses rather than getting the help they needed.

The lesson I learned from that is that software companies that take normal circumstances with the intensity of emergencies are gradually cutting their own throats.

That's a lesson reinforced for me spending a lot of time with a family member in hospitals last year. Even when survival was on the line, the best doctors and nurses proceeded with patience and kindness, working to train staff and improve systems as they went. In an actual emergency they moved like it was an emergency. But only then. If they can be serene and thoughtful while dealing with brain tumors, I don't think there's any reason that people at Facebook have to puff themselves up with self-importance.

Re: How Facebook Ships Code

#64

Post by Silhouette » Thu, Jul 05, 2012, 1:49 PM UTC

wilfra wrote: post #50

Silhouette wrote: post #27

Earlier quoted context omitted.

I'm not arguing for unrealistic quality levels, and I will acknowledge immediately that my experience could be atypical. However, in fairness, if I look at all of the software that I use regularly in a professional capacity today, then it is clearly Google products that are the most buggy, and by a very wide margin. For example, I have a client who uses Google Docs/Drive. We have rarely managed to hold a meeting with…

Where HTML5 is concerned, Chrome is by far the most buggy browser. It's unfortunate because over 50% of our users are using it now.

Where HTML5 is concerned, Chrome is by far the most buggy browser.

I agree, though FWIW I'd say it still takes more work to support Safari on iOS because of all the odd quirks. Apple might not consider them bugs. They're entitled to their opinion. ;-)

It surprises me that we've reached this point, but several of the web-based projects for business users that I work on professionally now recommend IE9 and won't officially support either Firefox or Chrome, and I agree with them. The trendy culture of frequently pushing subtle changes and half-baked new features and then hoping that if anything breaks you can fix it fairly quickly might work for places like Facebook, but it just doesn't cut it for professionals.

Re: How Facebook Ships Code

#65

Post by wpietri » Thu, Jul 05, 2012, 1:49 PM UTC

gaius wrote: post #58

gioele wrote: post #48

Earlier quoted context omitted.

Boundary value analysis? Machine state transitions testing? Equivalence partitioning? There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.

LOL! You think someone who's drunk the unit testing kool-aid has even heard of any of that?

You have to start somewhere. If I have to choose between a world where people never test and one where they test but aren't very good at it yet, I'll definitely take the latter.

Of course, I'm still filled with rage when I find a cargo-cult suite of unit tests. But it's so much easier to convince those people to write smarter tests than it is to deal with the team that has no time to write tests because they're spending all their time debugging.

Re: How Facebook Ships Code

#66

Post by gouranga » Thu, Jul 05, 2012, 1:59 PM UTC

alttab wrote: post #46

rimantas wrote: post #30

Earlier quoted context omitted.

No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.

By being able to design software that doesn't need or allow crazy data configurations. My whole mantra is don't manage complexity: avoid it.

You nailed it - thanks for the absolutely spot on comment.

Design is the key word here. It requires thought, it requires intelligence and it requires multiple people's input.

Re: How Facebook Ships Code

#67

Post by wpietri » Thu, Jul 05, 2012, 1:59 PM UTC

mgkimsal wrote: post #38

TazeTSchnitzel wrote: post #14

Earlier quoted context omitted.

Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.

A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc). Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email repor…

I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..."

Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume.

Madness, sure, but common madness.

Re: How Facebook Ships Code

#68

Post by gouranga » Thu, Jul 05, 2012, 2:03 PM UTC

jorgeleo wrote: post #36

gouranga wrote: post #2

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…

But Facebook is not a financial institution, and I would put the responsability in the user if financial data is on the news feed. One size (of engineering practices) fit all mentality is not the correct approach either.

You are correct - it is not a financial organsiation.

It does know who you are, what you eat, where you've been, who you're friends with, what you're interested in, what your political allegiances are, what your bowel movements are like etc.

A list of investments or your mortgage statement is way less important.

Re: How Facebook Ships Code

#69

Post by gouranga » Thu, Jul 05, 2012, 2:05 PM UTC

mgkimsal wrote: post #38

TazeTSchnitzel wrote: post #14

Earlier quoted context omitted.

Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.

A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc). Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email repor…

> Adding on top of that was my own dynamic language background not mapping well to the JVM

There's your problem...

Re: How Facebook Ships Code

#70

Post by gouranga » Thu, Jul 05, 2012, 2:06 PM UTC

its_so_on wrote: post #34

This is the real Facebook secret sauce in convenient flowchart form: What's the most evil thing we can think of doing? V Candidate ------->^ V (yes) ^ Is it legal? ^ V (yes/no) ^ Can we get away with it? (abs. not)--->^ V (yes/maybe/prob. not) ^ Keep shipping! ^ V ^ Did we get in trouble for it? (no) ^ V (yes) V ^ Claim it was a mistake! V ^ V V ^ Still in trouble? (no)-> Keep feature->^ V (kind of) ^ Being sued for…

That's great - thanks for posting it :)