Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google could have killed Facebook with the flick of a switch

shaneosullivan.wordpress.com

Post reply on HN
Print view View original

131–140 of 145 posts

Re: Google could have killed Facebook with the flick of a switch

#131

Post by shaneos »

grecy wrote: post #99
esprehn wrote: post #94

This story is a fun read, but the "Chrome could ruin us at any moment" angle is more story telling hyperbole than reality. At that time on Chrome we were just figuring out a process to remove features and were pretty conservative. We debated a lot about the right way to remove things without causing damage to the web ecosystem. In an early attempt we had failed trying to remove a legacy feature and needed to revert i…

.. and even if Chrome had removed WebSQL back then, couldn't that have just kept using whatever version of Chrom e they were already using?

[author here] Unfortunately no - we had many thousands of customers, most of whom had no control over their machines (low level employees at ad agencies). As a company, Facebook has no control over what browsers their customers use. The product would simply have broken overnight

Re: Google could have killed Facebook with the flick of a switch

#132

Post by shaneos »

esprehn wrote: post #101
grecy wrote: post #99

Earlier quoted context omitted.

.. and even if Chrome had removed WebSQL back then, couldn't that have just kept using whatever version of Chrom e they were already using?

I suspect a lot of FB customers are using a browser in a business setting where IT manages the Chrome updates. FB could have used a polyfill though, or used Safari, or shipped a desktop wapper around an older Chrome temporarily. Electron was brand new at the time so it probably wasn't a great option, but FB certainly had a lot of emergency options.

[author] There were some contingency plans, but we had no time to get them ready in advance. If Google had actually pulled the plug, we wouldn't been down for everyone for many weeks, and given that the vast majority of our large customers are non-tech-savvy (low paid employees at ad agencies), a large percentage would have remained with a broken experience forever. They'd have just given up and moved their money to Google.

Re: Google could have killed Facebook with the flick of a switch

#133

Post by shaneos »

Osiris wrote: post #7

If the product was responsible for 25% of the company revenue, why was only half of one engineer assigned to it? And after the vulnerability was discovered, they still only had 5 people assigned to it while it increased in importance to 50% of revenue?

[author here] Getting 5 engineers was actually a huge ask. Just a year earlier the entire Ads Interfaces org had 5 engineers in it (I was #6). That was 5 engineers for all API and front end code for a $4B/year revenue company. Facebook has a history of using tiny teams to lift impossible mountains - e.g. FB Photos was originally written by two engineers and overtook Flickr, the current leader, with just that investment.

Re: Google could have killed Facebook with the flick of a switch

#134

Post by shaneos »

ska wrote: post #92
Osiris wrote: post #7

If the product was responsible for 25% of the company revenue, why was only half of one engineer assigned to it? And after the vulnerability was discovered, they still only had 5 people assigned to it while it increased in importance to 50% of revenue?

> If the product was responsible for 25% Being a (however necessary) part of the toolchain is importantly different than being responsible for the revenue. Such tools often fall under "it ain't broke, don't mess with it". Until they are broke, of course.

[author] There were a number of critical parts of the chain responsible for FB Ads revenue. It all started at the UI or API (which were 75% and ~25% of FB revenue when I left in 2017). Then you had the DB layer, managed by another team, the AI/Targeting layer and one or two others.

If any had failed fundamentally, FB was in big trouble. However, other than Power Editor and Chrome, the rest all ran in our data centers and were under our control.

Re: Google could have killed Facebook with the flick of a switch

#135

Post by shaneos »

Cthulhu_ wrote: post #38
Osiris wrote: post #7

If the product was responsible for 25% of the company revenue, why was only half of one engineer assigned to it? And after the vulnerability was discovered, they still only had 5 people assigned to it while it increased in importance to 50% of revenue?

I hope those 13 engineers got paid millions. So much weight and responsibility on their shoulders! They were responsible for nearly $2 billion in 2013 (25% of $7.87bn), and nearly $14bn in 2016 when the work was finished (50% of $27.63bn)

[author] Many did and it was well earned. The team was awesome and had great camaraderie, but the work was horrible. Everything that broke led to a SEV for our team, even though 80% of the time it was not our code (somewhere deep in the API that had insufficient logging and observability). All our customers hated the product since it was so buggy, and our internal customers (sales ppl at FB) hated it for the same reason. They didn't see the 16 hour days, nights and weekend that we were all putting in to fight fires, do a fundamental rewrite from two JS frameworks to an untested new one (React), and add hundreds of new product features to keep up with FB's exploding Ads org. It was three years of hell to be frank, but very rewarding if you're into that kind of thing. It took a special type of person to join that team, knowing how long it would be before any payoff was really visible in the product, all while being vilified inside and outside the company even though you didn't build the thing, you just agreed to jump in and help fix it.

Re: Google could have killed Facebook with the flick of a switch

#136

Post by shaneos »

esprehn wrote: post #87
neodymiumphish wrote: post #60

> This became a closely held secret in Facebook Ads leadership. We didn’t want to take any chance that word our this vulnerability could get back to Google. Considering the tracking Google does with Chrome, I suspect they recognized the traffic still occurring over WebSQL and left it active for exactly this reason. It's highly unlikely that nobody at Google was able to recognize this. Alternatively, maybe Google didn…

At the time this story takes place Chrome didn't have a way to collect web feature usage metrics broken down by site. It was aggregated across all sites to protect user privacy. That came years later with Rappor: https://www.chromium.org/developers/design-documents/rappor WebSQL usage was too high at the time to remove though, so while the story is fun to read Chrome never would have actually turned it off.

[author] Hard to say. If Google leadership had heard of this vulnerability, right as they were launching Google+, there would at least have been a serious conversation about it. They spent $Bs to build Google+, changed their whole company structure to support it, and were losing hundreds of people to FB every month. I doubt bending one little rule about removing a deprecated technology would have bothered them too much.

Re: Google could have killed Facebook with the flick of a switch

#137

Post by shaneos »

whalesalad wrote: post #120

Could have killed you with the flick of a switch - but you took three years and a team of 5 people to perform the rewrite?

[author] Unfortunately yes. It was not the only thing happening at the time. FB had just gone IPO and this put a huge amount of pressure to grow revenue. When I joined the Ads org in 2012, it was about 100 people I think, with just a few products. When I left in 2017 it was well over 1000, and my job had changed to managing relationships with 55 other teams, all of whom depended on my teams product to ship their individual features (think Video ads, Carousel ads, hundreds of others).

So, while doing the rewrite, we had to transition from being a product org, where a few engineers built all features for a bunch of back end teams, to an infra+product team, where we built all sorts of support for all the new teams that seemed to spring up weekly.

If we had done only the technical rewrite, and not enabled all these other teams to ship their products on the UI surface that our customers actually used, FB's revenue would not have gone from $4B when I started to around $30B when I left.

These things are never that simple. I wish....

Re: Google could have killed Facebook with the flick of a switch

#138

Post by shaneos »

waterside81 wrote: post #104

Anybody have insight into why WebSQL was chosen in the first place? I don't think I've even heard of it prior to this article.

[author] Power Editor was originally written by a brilliant engineer named Vladimir Kolesnikov (yes, all by one engineer), and he was solving a very specific problem of letting customers scale their ad spend. So he chose a technology that let them download all of their ad account locally to disk, do all their work, then upload it again.

WebSQL was flexible, fast and easy to work with. It's a real shame it was deprecated to be honest

Re: Google could have killed Facebook with the flick of a switch

#139

Post by shaneos »

disgruntledphd2 wrote: post #15
simonh wrote: post #10

Earlier quoted context omitted.

I've seen this sort of thing happen many times. What usually happens is the original team that built it leave the company or move on to other teams, and the one engineer left actually has a different job as well now but is stuck supporting it part time as a favour. It's not that you couldn't assign other engineers to help support it, but they wouldn't have any idea how to do so because there's probably no documentati…

It was because Facebook practiced open allocation for engineers at the time. Essentially, you had a choice of teams and the manager would need to convince the engineer to work on the project. At the time, ads was not a high status part of FB engineering, and as such, it was very difficult to get people to work on ads products.

[author] This is very accurate. As the manager of the team I had to spend a huge amount of time trying to hire internally for it. No one came to FB to work on ads, they wanted to work on stuff that their friends used. I had to develop a smooth song-and-dance to get the right type of people interested. In the end, this served me very well in my career, being well practiced at getting engineers excited to join your team is a valuable skill.

Re: Google could have killed Facebook with the flick of a switch

#140

Post by shaneos »

throwawayboise wrote: post #74
disgruntledphd2 wrote: post #15

Earlier quoted context omitted.

It was because Facebook practiced open allocation for engineers at the time. Essentially, you had a choice of teams and the manager would need to convince the engineer to work on the project. At the time, ads was not a high status part of FB engineering, and as such, it was very difficult to get people to work on ads products.

Hard to imagine that anything responsible for 25% of revenue (billions of dollars, at FB scale) would not be seen as "high status" if not by the engineers, then by management.

[author] It wasn't. The plan at the time was to deprecate Power Editor and build a new version of Ads Manager. We undertook both projects simultaneously, with the plan being to keep Power Editor going until AM v2 was ready. Unfortunately that took too long, and since Power Editor was the only product that could ship all the new ads features, by the time it came that Ads Manager V2 was ready, Power Editor had 50% of revenue share and Ads Manager about 20%. So, in effect we rebadged PE as Ads Manager, pulled over some of the new features from AM V2 and called it done.
Post reply on HN