Meta developer tools: Working at scale • Hacker News Board

Re: Meta developer tools: Working at scale

#111

Post by jacksnipe » Wed, Jun 28, 2023, 5:35 PM UTC

zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

Strongly not my experience. The tooling at meta was so good compared to my previous 5 jobs that I left with changed opinions about the tradeoffs of investing in tooling.

Re: Meta developer tools: Working at scale

#112

Post by LoulouMonkey » Wed, Jun 28, 2023, 5:42 PM UTC

zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

As a Meta employee as well, and working in the data analytics / engineering space, I'm finding tooling to be pretty high standards actually.

Though we arguably rely on a lot of Apache products, whatever we use that's internal only is great to work with. Daiq** recently started supporting notebooks, which has been a game changer for us as well as for the teams we work with. Phabricator is great as well, and makes shipping stuff super easy. Only Ben**, the internal notebook solution, I find meh. Especially compared to Google Colab. But the rest has simply been a joy to work with.

For those interested, a former DE made this nice repo that maps internal tools against "real world" products: https://github.com/thijsessens/xmeta2external

Re: Meta developer tools: Working at scale

#113

Post by jonathankoren » Wed, Jun 28, 2023, 5:44 PM UTC

morgannewman wrote: post #72

jonathankoren wrote: post #36

I stopped reading when I reached that they made their own CVS. This is a solved problem, and even if it isn’t, there’s entire communities dedicated to it. There’s literally no reason why Facebook needs to be in the business of reinventing the wheel from scratch beyond some dude trying to show “impact”. It’s a distraction from core business problems.

You might be surprised to learn that it was not and still is not a solved problem for companies like Meta. Here is an earlier write-up about this topic from Meta: https://engineering.fb.com/2014/01/07/core-data/scaling-merc...

You might be surprised to learn that they're solving the wrong problem.

Re: Meta developer tools: Working at scale

#114

Post by jonathankoren » Wed, Jun 28, 2023, 5:44 PM UTC

TechBro8615 wrote: post #100

morgannewman wrote: post #72

Earlier quoted context omitted.

You might be surprised to learn that it was not and still is not a solved problem for companies like Meta. Here is an earlier write-up about this topic from Meta: https://engineering.fb.com/2014/01/07/core-data/scaling-merc...

Also keep in mind they've been in hypergrowth mode since 2010 or earlier. Regardless of whether Git is the clear solution today, it definitely wasn't at that time. So it makes sense they invested in building their own tooling.

And now it's 2023, and they're inventing the wheel again.

Re: Meta developer tools: Working at scale

#115

Post by SilverBirch » Wed, Jun 28, 2023, 5:44 PM UTC

yawnxyz wrote: post #12

This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"

Isn't this the Google curse? Companies hire an engineer who was at Google and suddenly find the Google engineer is building out all the Google internal infrastructure within the new company and because You Aren't Google, it sucks.

Re: Meta developer tools: Working at scale

#116

Post by ahahahahah » Wed, Jun 28, 2023, 5:46 PM UTC

cletus wrote: post #53

I've worked for both Facebook and Google so can make informed comments on this with two exceptions: Buck2 came after I left and I'm honestly not sure what sapling is. Is it some Mercurial-like re-implementation a bit like how Google's Piper is a re-implementation of Perforce? The tl;dr is that Google's developer tooling and ifnrastructure is superior in almost every way. Examples: - When I started at FB we used Nucli…

Everyone reading this should assume it's talking about FB around 2014 or so, I don't know how it could otherwise be so wrong.

> FB's code search is just grep. It's literally called BigGrep.

This hasn't been true for a long time, codesearch at FB is more complex than just grep and has some semantic understanding. Here's some discussion here when some of this infrastructure was open sourced: https://news.ycombinator.com/item?id=28365880

> FB C++ might take 2 minutes just to load the DAG before it builds anything.

Yeah, this is pretty terrible, but is mostly a description that doesn't apply to buck2.

> This is essentially instant at Google because a lot of infrastructure has been built to solve this problem. This is a combination of SrcFS and ObjFS.

SrcFS and ObjFS aren't what solve this problem at Google. And to the extent that they do, FB's sapling and buck integration do the same.

> All non-www builds at FB are local builds. Nothing at Google (on Google3 at least) is built locally, including mobile apps. This is way faster because of build artifact cachcing and you have beefier build machines.

This is wrong since at least 2015 FB's build system had the build artifact caching, and you can see in buck's git history that they've had remote execution just like Google (in fact, using the same RE api as bazel) since like 5 years ago.

Re: Meta developer tools: Working at scale

#117

Post by [deleted] » Wed, Jun 28, 2023, 5:53 PM UTC

epage wrote: post #13

arnon wrote: post #2

Surprised to see them use Phabricator (I know it came out of there, but I basically already forgot it existed) I used it briefly but couldn't get most people to adopt it widely enough.

I used Phab at a job and it was a complete mess. Stacked Diffs only sort-of worked, CI integration was bad, notifications were so noisy everyone tuned them out, etc. Talking with a Meta person, it sounds like Phab really needs Mercurial to work well, at least for Stacked Diffs because you need to be able to identify commits independent of their location in history to properly maintain the Stacked Diff associations.

[deleted]

Re: Meta developer tools: Working at scale

#118

Post by kevan » Wed, Jun 28, 2023, 5:56 PM UTC

optymizer wrote: post #60

zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

For what it's worth, as a Meta employee, I generally disagree with this comment. I mean, calling it 'inferior' is just laughable. It's way better than what open source projects have access to and it's way better than Amazon internal tooling (is that 'industry standard' enough?). The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's…

>and it's way better than Amazon internal tooling (is that 'industry standard' enough?).

It's slowly changing but I wouldn't consider Amazon's tooling to be industry standard by any definition.

Re: Meta developer tools: Working at scale

#119

Post by baggiponte » Wed, Jun 28, 2023, 5:59 PM UTC

I am genuinely curious about wasabi, the python LSP they announced on Meta open source but is not available anywhere. Would love to try that out, there is not enough competition in the LSP space in Python and it would foster new development https://developers.facebook.com/blog/post/2022/07/18/enablin...

Re: Meta developer tools: Working at scale

#120

Post by sangnoir » Wed, Jun 28, 2023, 6:04 PM UTC

jonathankoren wrote: post #76

rigelbm wrote: post #39

Earlier quoted context omitted.

> This is a solved problem. Hosting a gigantic monorepo for 25K concurrent users is so far from a solved problem.

[flagged]

For a moment, let's assume they do abandon the monorepo. What solution would you recommend for managing code dependencies and coordinating releases between thousands of teams (at a modest 5 repos per team) - git tags?