Live data from Hacker News

How Facebook Ships Code

framethink.wordpress.com

21–30 of 116 posts

Re: How Facebook Ships Code

#21
post #2

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…

>I build BIG financial software and we have certain audit requirements, access control requirements, data protection requirements etc and that is exactly how it should be.

Funny, I had the same argument with a colleague today. Since FB is listed and their business model is based on customer data they should follow the same legal requirements as customer finance companies.

Re: How Facebook Ships Code

#22

Am I the only person who thinks this whole approach is broken? We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on. From the outside, I see systems…

As a simple user I sometimes find annoying all that continuous tweeking and changing stuff in facebook. Maybe there is a reason in user behaviour and adds conversions, but if doesn't please don't make all that uselss changes just for fun!. The last timeline mayor update is specially dull, I find it hard to navigate with that splited view. Of course I am only a user, surely the cohort stats show how all the millions of people rave about each and every change... In my opinion is far more important fixing bugs like the one at the Ipad app. When you visit some comments in a picture you have previously commented, facebook will open another random picture or comment page of the user who posted the picture. It 's the same on the older iphone app( I have the 3G and I am not updating it, so it's my own selfinflicted bug), but I just downloaded the ipad app two weeks ago, and there it is, safe and sound the same ol' bug. Sorry for the final rant, but maybe they must have some department curating the user experience beyond the engineer centered culture. Maybe is better the google way, work on stablished problems, and then give people some time to work on side proyects and give them the oportunity to grow.

Re: How Facebook Ships Code

#23

I really like seeing this approach listed: "resourcing for projects is purely voluntary. A PM lobbies group of engineers, tries to get them excited about their ideas. Engineers decide which ones sound interesting to work on." That sounds exactly the same as how Github's engineers work. It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. And if you can't get anyone interest…

> It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. Eventually people lose interest. It's only human. But The System has been written, it is in production, and it has accrued a healthy stock of user data. One day, The System breaks. "Only" tens of millions of users -- less than a percent of all Facebook users -- rely on the The System. But they rely on it utterly. Who wi…

> That's boring, and there are no incentives for fixing bugs in obscure features because only launching successful new features gains visibility from higher-ups.

Companies really should find a way to incentivize this. Rewrites are about as silly. "Write the next generation xyz." Translates to "No one here has any idea how xyz was written, so it's being rewritten instead of modified. We look forward to rediscovering the same problems we did the first time."

Re: How Facebook Ships Code

#24
post #2

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…

It's a little irresponsible, but how else do you expect them to debug live code?

Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire.

It just works.

In the last 5 years, we've never hooked a debugger to production instance or given access to a developer.

In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 billion page hits (most of our stuff is backend non OLTP).

That's how it's done.

Re: How Facebook Ships Code

#25
post #16

Am I the only person who thinks this whole approach is broken? We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on. From the outside, I see systems…

> Am I the only person who thinks this whole approach is broken? Broken? Depends on your perspective. If you care first and foremost about your users' data, it sure is broken. But FB is not a hosting company. If you care the most most about correctness of your code and getting everything right the first time (VMS style rather than UNIX `worse is better' style), then again, it will seem broken. But FB is not an academ…

I think Facebook is successful in spite of their engineering culture, not because of it. They started with the densest social graph of any social network and that made the difference in the long run.

Re: How Facebook Ships Code

#26
post #7
post #2

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…

Note that it doesn't say whether they all have write access, or whether all the data they have access to is encrypted in some way. And any of those statements should be interpreted as one response he got from talking to lots of different people, and be subject to some amount of skepticism.

Either way, it simply doesn't matter.

Re: How Facebook Ships Code

#27

Am I the only person who thinks this whole approach is broken? We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on. From the outside, I see systems…

I wouldn't lump Google in there with Facebook. We have very strict controls on access to user data (we can't even see email addresses in logs), and we have not adopted the motto "move fast and break things". We do extensive automated testing and have release processes that are designed to minimize problems in the event of a bad push. Yes, bugs happen from time to time, but it's software -- there is no known practical…

I'm not arguing for unrealistic quality levels, and I will acknowledge immediately that my experience could be atypical.

However, in fairness, if I look at all of the software that I use regularly in a professional capacity today, then it is clearly Google products that are the most buggy, and by a very wide margin.

For example, I have a client who uses Google Docs/Drive. We have rarely managed to hold a meeting without one of our small team struggling to see either a word processor document or a spreadsheet properly, and that's just the minor cosmetic or browser incompatibility bugs that keep appearing along with all those minor UI changes. We have also experienced some much more serious problems, including corruption/data loss and seeing the entire change history for some files become inaccessible for no apparent reason. In other words, it's not just minor UI errors that crept in as the product evolved, there are evidently fundamental flaws in the underlying architecture that don't store the data robustly.

Another example: I spent a couple of days last week trying to figure out why a site that had been working fine for users until recently and had not been changed at all on our side was suddenly generating bug reports. It turned out that recent Chrome builds have broken some HTML5 features in multiple ways. There have been related bug reports in some cases, but they have been closed as the specific example given no longer seemed to be a problem. Again, the nature of the problems makes it obvious that these are not just one-liner issues but fundamental flaws, typically where Chrome is so aggressive with its cache usage and/or trigging redrawing/relayout that it just plain doesn't work. And even though the bugs had been reported, obviously the correct root cause was never identified and fixed.

Another example: I recently spent some time looking into how the Closure Tools are coming along. Have you tried the examples/demonstrations for the Closure Library recently? Many of them simply don't work in Gecko-based browsers, and that would be obvious if anyone working on the project had made even a cursory attempt to test them for five minutes.

I will finish by once again acknowledging that my experiences here may be atypical, and that the particular projects I've mentioned that I happen to be using may not reflect the wider Google culture. But on the evidence before me, I see an organisation that keeps breaking things in its rush to push new features out and that demonstrably lacks robust architectures that keep data safe, effective testing processes before pushing code into production, and proper issue resolution processes when defects do get reported.

Re: How Facebook Ships Code

#28
post #18
post #2

What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…

You finance and billing guys will never understand. These are rockstars . They work for facebook. They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';

Of course not. Much faster to type

    UPDATE users SET email = username || '@facebook.com';
Then let the users sort out the exceptions.

Re: How Facebook Ships Code

#29

Earlier quoted context omitted.

It's a little irresponsible, but how else do you expect them to debug live code?

Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire. It just works. In the last 5 years, we've never hooked a debugger to production instance or given access to a developer. In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 bi…

You're right.

Now that I think about it, I doubt Facebook is that careful, considering how often they break things.

Re: How Facebook Ships Code

#30

Earlier quoted context omitted.

It's a little irresponsible, but how else do you expect them to debug live code?

Thou who uses unit tests, UI tests, precondition, postcondition checks, heavy logging, knows the language and toolchain and knows arse from elbow NEVER debugs in production out of necessity or desire. It just works. In the last 5 years, we've never hooked a debugger to production instance or given access to a developer. In that time, we've pushed approximately 912 billion SQL transactions through the system and 24 bi…

No please tell me, how did you learn to write unit test which cover all the crazy data configurations users are able to come with.
Post reply on HN