11–20 of 116 posts
I really like seeing this approach listed: "resourcing for projects is purely voluntary. A PM lobbies group of engineers, tries to get them excited about their ideas. Engineers decide which ones sound interesting to work on." That sounds exactly the same as how Github's engineers work. It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. And if you can't get anyone interest…
Eventually people lose interest. It's only human. But The System has been written, it is in production, and it has accrued a healthy stock of user data.
One day, The System breaks. "Only" tens of millions of users -- less than a percent of all Facebook users -- rely on the The System. But they rely on it utterly.
Who will fix The System?
Of the five engineers who wrote it:
* Jack and Mandeep left to launch myfornaxisnatrr.com
* Wei Li has moved to a different group
* John doesn't want to touch it, he only came on board to help Jack
* Michael was an intern but has since taken a job with Google.
Uh oh! There's nobody around to voluntarily fix an existing system. That's boring, and there are no incentives for fixing bugs in obscure features because only launching successful new features gains visibility from higher-ups.
Guess we'll need some mean old managers to round up a posse.
If they care enough.
Meanwhile, The System has acquired millions of users, cost millions of dollars to develop and operate and will now abruptly cause tens of millions of customers to become incredibly frustrated. And at no point has anybody stopped and asked:
"Was this the Right System to build?"
What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…
It's a little irresponsible, but how else do you expect them to debug live code?
I've actually spent quite a long time writing code where, for various legal reasons, you are never going to get direct access to the production data. (And indeed some cases where accessing the data in question would be a serious criminal offence).
Makes you really keen on defensive programming and comprehensive logging - although you do then have to be careful about what you log.
I actally found it kind of fun to try and debug complex systems based only on a log file and QA with production operations folks.
[NB Of course, none of this would be applicable to Facebook]
Earlier quoted context omitted.
You don't. Reproduce the issue on a development instance.
I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure. (Poe's Law says hi.)
Am I the only person who thinks this whole approach is broken? We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on. From the outside, I see systems…
Am I the only person who thinks this whole approach is broken? We have seen the rise of "devops" recently, and big name web companies like Facebook and Google seem to be very proud of how engineer-led they are, how empowered their developers are, how their product managers don't have much real authority, how they push code to production ten minutes before it's even written, and so on. From the outside, I see systems…
Broken? Depends on your perspective.
If you care first and foremost about your users' data, it sure is broken. But FB is not a hosting company.
If you care the most most about correctness of your code and getting everything right the first time (VMS style rather than UNIX `worse is better' style), then again, it will seem broken. But FB is not an academic excercise.
If, on the other hand, you felt like a stakeholder in the company and cared most about developing the business, then no, `moving fast and breaking things' is means to that end.
If, alternatively, you enjoy experimentation and pushing frontiers, then again, no, `moving fast and breaking things' is about the only modus operandi that can accomodate that.
Emphasis on product management and testing is for serving your customers -- but users aren't FB's customers. I cannot assure you, but I would bet a lot FB does indeed heavily test and manage projects related to serving ads -- which is what their customers pay for.
I really like seeing this approach listed: "resourcing for projects is purely voluntary. A PM lobbies group of engineers, tries to get them excited about their ideas. Engineers decide which ones sound interesting to work on." That sounds exactly the same as how Github's engineers work. It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. And if you can't get anyone interest…
> It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. Eventually people lose interest. It's only human. But The System has been written, it is in production, and it has accrued a healthy stock of user data. One day, The System breaks. "Only" tens of millions of users -- less than a percent of all Facebook users -- rely on the The System. But they rely on it utterly. Who wi…
What a crock of shit: after boot camp, all engineers get access to live DB I can understand on a startup or small org but an organisation of that size, there should be very tight access control. Despite what anyone says, the probability that someone does something bad increases in larger groups. Security should be on a simple need-to-know basis and nothing else. I build BIG financial software and we have certain audi…
These are rockstars.
They work for facebook.
They would never ever type UPDATE users SET email = username || '@facebook.com'; WHERE username == 'john.smith';
Earlier quoted context omitted.
I'm curious - what happens when an engineer wants to propose an idea? Do they need to get a PM on-board to lead the project? Who decides if the idea is good? The PM/engineer divide feels uncomfortable to me, but I have limited experience with these roles, so it's likely simply for lack of knowledge. I hope someone can clarify this for me.
From the article at least, it seems like there aren't a whole lot of PMs at Facebook. If an engineer has an idea, they seem to be more encouraged to just find some people to work with, build it, and then roll it out to a very small section of Facebook's users and see how they react to it. Good ideas might then be taken further, but bad ones (Or I guess, ones people just don't care about) won't be. (I don't work there…
That's not an entirely small amount for Facebook since they don't have a zillion different products and features that need to be owned, like a company like Microsoft or Google.
Earlier quoted context omitted.
> It's an awesome concept; no-one can justifiably be bored with their projects if they chose them. Eventually people lose interest. It's only human. But The System has been written, it is in production, and it has accrued a healthy stock of user data. One day, The System breaks. "Only" tens of millions of users -- less than a percent of all Facebook users -- rely on the The System. But they rely on it utterly. Who wi…
The world doesn't work like this. In addition to doing-whatever-they-want-24-7, people are also trying to get promoted. That means there is always an incentive to do some task that nobody else will do. This kind of causes talent to even out between high-profile new projects and not-as-fun legacy projects.
In places where the goal is to make engineers happy and to impress other engineers, there are incentives for behaviour that are perverse from the customer's point of view.