How Facebook Ships Code • Hacker News Board

Re: How Facebook Ships Code

#51

Post by alttab » Thu, Jul 05, 2012, 12:46 PM UTC

sreyaNotfilc wrote: post #40

This may be my favorite quote... "Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything t…

I agree. Specialists are really just full stack engineers who haven't fulfilled their potential. Why hire a self-proclaimed UX guy? Unless I have all the amazing engineers I need for my team (99% not the case), I'm looking for someone who can do multiple things. The best ones are ones that can do schema design, query design, system and model architecture, front-end wireframes, javascript, and implementation.

That said, your comment about the front end being "tightly woven in with the business logic of the module" scares me a little. The front end is presentation and interaction. Unless most of your businesses mojo is client side, even a full stack engineer should design each layer as fully cohesive and encapsulated components. I may have misunderstood you, however.

Re: How Facebook Ships Code

#52

Post by alttab » Thu, Jul 05, 2012, 12:50 PM UTC

DigitalSea wrote: post #37

Don't get me wrong a developer-led company in theory sounds like an awesome idea because the developers know the product better than any product manager ever could, but that isn't a good thing. I've worked with companies that have ultra-strict testing of features, where many eyes see and use the code before being pushed out and while that process works it can get in the way of progress when the politics come out to p…

I would argue the opposite. Facebook is in prime position to move quickly and break things. They have enough traction, gravity, and brand recognition they can afford to make mistakes.

If you have a small user base or are just getting started, delivering the best product experience is the #1 thing you can do for success. Is iterating faster and making more mistakes and giving the early adopters (ones that champion your service the most to others) a sub-par experience worth the speed? Sometimes, yes. All the time? No.

Facebook can change your privacy settings, steal your e-mail address, recognize you automatically in photos, watch your browsing activity outside of their walled garden, and sell your data to advertisers. And yet they still have 500million+ users.

Re: How Facebook Ships Code

#53

Post by Goladus » Thu, Jul 05, 2012, 12:51 PM UTC

rimantas wrote: post #32

bnr wrote: post #9

Earlier quoted context omitted.

You don't. Reproduce the issue on a development instance.

How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?

For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.

Re: How Facebook Ships Code

#54

Post by alttab » Thu, Jul 05, 2012, 12:52 PM UTC

yawgmoth wrote: post #43

I like the idea of encouraging a high-performance culture, but I don't think the 'perform or die' atmosphere would be healthy for many engineers. I know, idolizing 'rockstar programmers' is a sort of new hotness and I understand that a company like Facebook wants to have super-talented developers, but developers grow and learn new tricks as they mature, and they might take more than six months to do so.

True - but those engineers don't work at Facebook. This isn't no-child left behind. This isn't hand-hold time. This is the most expensive and expansive internet application in the world. If they need time to ramp up, they can do it on someone else's product and come to Facebook when they're ready.

Re: How Facebook Ships Code

#55

Post by gaius » Thu, Jul 05, 2012, 1:09 PM UTC

gouranga wrote: post #2

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…

Right. FB doesn't get that all they have is the user experience, and it is bad user experience if I drop out of a conversation I'm having because their DBA-free database has decided to simply, silently drop my last comment. Or if it takes me several attempts to post a photo. Or if I invite someone to an event, and they never get the invite.

You can do all sorts of amazing things if you just don't care if your code actually works or not. Grown-up companies have full-time professional DBAs, and not just for separation of concerns.

Re: How Facebook Ships Code

#56

Post by polyfractal » Thu, Jul 05, 2012, 1:11 PM UTC

sreyaNotfilc wrote: post #40

This may be my favorite quote... "Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything t…

Eh, I dunno. I don't want the carpenter designing my house, and I don't want the architect cutting the main support beams. I want both of them doing what they do best, and working together to do it.

Re: How Facebook Ships Code

#57

Post by wpietri » Thu, Jul 05, 2012, 1:11 PM UTC

Silhouette wrote: post #10

Am I the only person who thinks this whole approach is broken? We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on. From the outside, I see systems…

"Broken" seems strong. I'd agree with "has flaws", but every system has flaws.

If I were Zuckerberg, I'd be terrified of slipping into a premature complacency or a heavyweight process that prevents change. After years of social network king-of-the-hill, Facebook seems to be set up for the generational dominance of a Microsoft or a Google. But it's not locked in yet.

Sure, current practices risk irritating users or a severe FTC beatdown. But those are obvious risks, easy to measure and mitigate. The real harm to Microsoft didn't come from annoying UI or regulatory intervention. It came from long-term shifts that they were too culturally isolated and too inflexible to understand or adapt to, much less drive.

There are certainly downsides to shipping early and often and then seeing what happens to your user metrics. But bottom-up power to ship means that a lot more good, novel ideas will see the light of day.

Re: How Facebook Ships Code

#58

Post by gaius » Thu, Jul 05, 2012, 1:13 PM UTC

gioele wrote: post #48

rimantas wrote: post #30

Earlier quoted context omitted.

No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.

Boundary value analysis? Machine state transitions testing? Equivalence partitioning? There are heaps of techniques one can use to identify which data points and combinations are useful to test. Some of these techniques even take a peek at your code to highlight the possible pain points.

LOL! You think someone who's drunk the unit testing kool-aid has even heard of any of that?

Re: How Facebook Ships Code

#59

Post by Goladus » Thu, Jul 05, 2012, 1:13 PM UTC

The facebook system actually sounds really solid, especially the "boot camp" thing that so many companies fail to have, however it's probably pretty expensive. Choosing between fast, cheap, and good, Facebook is choosing fast and good. For many, cheap and good but slow is more desirable.

One of the risks to consider with a "devops" oriented approach is that you may become more dependent on it than you want.

Often, applications split things into a few different categories depending on how tweakable they need to be. There's code, configuration, app administration, and data. Code shouldn't need to change often. Configuration may need to change when the environment changes. App administration (eg creating new accounts) needs to change often, and data is always changing (or at least growing).

The risk is that developers will design the system so that only developers can administer it. Configuration, the settings that may need to be tweaked by sys admins long after the original developers have left the project, may wind up in the code or sometimes lumped into the database alongside end-user options.

It's not a reason not to take this approach, just something to consider when developing internal processes and culture.

Re: How Facebook Ships Code

#60

Post by wpietri » Thu, Jul 05, 2012, 1:22 PM UTC

Silhouette wrote: post #27

jrockway wrote: post #15

Earlier quoted context omitted.

I wouldn't lump Google in there with Facebook. We have very strict controls on access to user data (we can't even see email addresses in logs), and we have not adopted the motto "move fast and break things". We do extensive automated testing and have release processes that are designed to minimize problems in the event of a bad push. Yes, bugs happen from time to time, but it's software -- there is no known practical…

I'm not arguing for unrealistic quality levels, and I will acknowledge immediately that my experience could be atypical. However, in fairness, if I look at all of the software that I use regularly in a professional capacity today, then it is clearly Google products that are the most buggy, and by a very wide margin. For example, I have a client who uses Google Docs/Drive. We have rarely managed to hold a meeting with…

I saw one interesting analysis of service bugs. I can't find it now, but the basic point was that people should stop focusing on MTBF (median time between failures) and instead look at MTTR (median time to recovery).

The notion was that it was much better to be able to notice and fix bugs quickly than it was to add process delays that reduced bug frequency. That especially makes sense to me when you think about the value integral. The longer your release cycle, the longer you keep people away from improvements. It also makes sense if you think about companies as learning organizations; speed of learning is limited by the length of your feedback loops.

So I'd say that Google's mistake isn't going for a release-early, release often approach. It's that they don't pay sufficient attention after release to user feedback or user metrics. Hell, I've stopped reporting bugs to Google, and I must know 10 people who work there. Even with a good backchannel I just have no faith that anything would come of a report.