Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Facebook Ships Code

framethink.wordpress.com

Post reply on HN
Print view View original

31–40 of 116 posts

Re: How Facebook Ships Code

#31

Post by TezzellEnt »

michaelmartin wrote: post #3

I really like seeing this approach listed: "resourcing for projects is purely voluntary. A PM lobbies group of engineers, tries to get them excited about their ideas. Engineers decide which ones sound interesting to work on." That sounds exactly the same as how Github's engineers work. It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. And if you can't get anyone interest…

This sounds to me exactly how Valve and other organizations operate (http://blogs.valvesoftware.com/abrash/valve-how-i-got-here-w...).

I think that if you get buy-in from the engineers working on the project it can induce ownership and a willingness to watch it succeed and work towards it's success. The issue occurs when there isn't adequate a/b testing (or something similar), which could create the fiasco such as facebook's switching of email addresses.

Companies such as Google allow 20% time to work on projects that might not make it to production, or get killed after acquisitions. I think that policies such as those work better than code being committed on the fly.

Re: How Facebook Ships Code

#32

Post by rimantas »

bnr wrote: post #9
TazeTSchnitzel wrote: post #4

Earlier quoted context omitted.

It's a little irresponsible, but how else do you expect them to debug live code?

You don't. Reproduce the issue on a development instance.

How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?

Re: How Facebook Ships Code

#33

Post by jacques_chester »

angstrom wrote: post #23
jacques_chester wrote: post #12

Earlier quoted context omitted.

> It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. Eventually people lose interest. It's only human. But The System has been written, it is in production, and it has accrued a healthy stock of user data. One day, The System breaks. "Only" tens of millions of users -- less than a percent of all Facebook users -- rely on the The System. But they rely on it utterly. Who wi…

> That's boring, and there are no incentives for fixing bugs in obscure features because only launching successful new features gains visibility from higher-ups. Companies really should find a way to incentivize this. Rewrites are about as silly. "Write the next generation xyz." Translates to "No one here has any idea how xyz was written, so it's being rewritten instead of modified. We look forward to rediscovering t…

> Companies really should find a way to incentivize this.

It can't be done from the top. You need to start with engineers who care about quality over the long term. It has to come from within each engineer.

When Zuckerberg talked about younger programmers being "better", he probably meant "more like me". But old farts are just young farts with expensively-acquired scar tissue.

Most of our most treasured software development lore comes from the lesson that "move fast, break things" just devolves into "fuck, everything is broken, fix it fast".

Re: How Facebook Ships Code

#34

Post by its_so_on »

This is the real Facebook secret sauce in convenient flowchart form:

    What's the most evil thing we can think of doing?
     V 
    Candidate ------->^
     V (yes)                               ^
    Is it legal?                           ^
     V (yes/no)                            ^
    Can we get away with it? (abs. not)--->^
     V (yes/maybe/prob. not)               ^
    Keep shipping!                         ^
     V                                     ^
    Did we get in trouble for it? (no)     ^
     V (yes)                        V      ^
    Claim it was a mistake!         V      ^
     V                              V      ^
    Still in trouble? (no)-> Keep feature->^ 
     V (kind of)                    ^
    Being sued for it? (no) --->--->^
     V (yes)                        ^
    Throw money at lawsuit          ^
     V                              ^
    Did we lose? (yes/no) ----->--->^

Re: How Facebook Ships Code

#35

Post by [deleted] »

piggity wrote: post #18
gouranga wrote: post #2

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…

You finance and billing guys will never understand. These are rockstars . They work for facebook. They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';

[deleted]

Re: How Facebook Ships Code

#36

Post by jorgeleo »

gouranga wrote: post #2

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…

But Facebook is not a financial institution, and I would put the responsability in the user if financial data is on the news feed.

One size (of engineering practices) fit all mentality is not the correct approach either.

Re: How Facebook Ships Code

#37

Post by DigitalSea »

Don't get me wrong a developer-led company in theory sounds like an awesome idea because the developers know the product better than any product manager ever could, but that isn't a good thing. I've worked with companies that have ultra-strict testing of features, where many eyes see and use the code before being pushed out and while that process works it can get in the way of progress when the politics come out to play.

The whole move fast, make mistakes mantra is a bad approach when you're a high-trafficked site like Facebook that risks jeopardising your revenue. A second of down time can be very costly. Knowingly pushing out half or completely untested features might be acceptable if you have a small user base, you're a new Internet startup or your logo says alpha or beta, but certainly not an established brand and company it's stupid.

This is all my own opinion, of course.

Re: How Facebook Ships Code

#38

Post by mgkimsal »

TazeTSchnitzel wrote: post #14
damncabbage wrote: post #11

Earlier quoted context omitted.

I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure. (Poe's Law says hi.)

Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.

A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc).

Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports).

Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L.

Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops.

Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.

Re: How Facebook Ships Code

#39

Post by batgaijin »

Silhouette wrote: post #27
jrockway wrote: post #15

Earlier quoted context omitted.

I wouldn't lump Google in there with Facebook. We have very strict controls on access to user data (we can't even see email addresses in logs), and we have not adopted the motto "move fast and break things". We do extensive automated testing and have release processes that are designed to minimize problems in the event of a bad push. Yes, bugs happen from time to time, but it's software -- there is no known practical…

I'm not arguing for unrealistic quality levels, and I will acknowledge immediately that my experience could be atypical. However, in fairness, if I look at all of the software that I use regularly in a professional capacity today, then it is clearly Google products that are the most buggy, and by a very wide margin. For example, I have a client who uses Google Docs/Drive. We have rarely managed to hold a meeting with…

Hey man, don't anger the hive.

Re: How Facebook Ships Code

#40

Post by sreyaNotfilc »

This may be my favorite quote...

"Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves."

I actually like this idea. Building my own website as well as working as the senior engineer during my day job forces me to be involved in all facets of web developing. The jobs are not abstracted. You are expected to know what you're doing from the front end to the back end. If not, nothing get done in a timely manner. I like this method because the front end is tightly woven in with the business logic of the module that you're working on. In other words, you know what the code is doing inside and out.

Post reply on HN