Meta developer tools: Working at scale • Hacker News Board

Re: Meta developer tools: Working at scale

#121

Post by likpok » Wed, Jun 28, 2023, 6:06 PM UTC

blitz_skull wrote: post #4

Sapling looks quite cool! I've used git extensively in my career and consider myself as having a slightly-more-advanced-than-typical understanding of how to use it just based on conversations with colleagues. However, one thing that's always been very limiting with git has been stack-based PR reviews, and as they mentioned amending deep commits. It's not impossible, but it makes it awkward enough that I usually avoid…

When we first switched from git to hg internally I was real cranky. Git was the clear industry winner, and hg was adding a whole bunch of churn for no good reason.

Now, hg is amazing and when I leave I will be extremely sad to go back to git. The UX is well thought out, with commands mapping to operations (how do you undo a commit? hg uncommit vs git playing around with the reflog).

Amending deep commits is pretty good -- it's still tricky and absorb works on some pretty limited heuristics. But the merge tooling is pretty good around moving around in a stack, and the general UX over interacting with the stack is way better: hg histedit edits history, while hg rebase moves the stack around.

Re: Meta developer tools: Working at scale

#122

Post by Hermitian909 » Wed, Jun 28, 2023, 6:24 PM UTC

zug_zug wrote: post #29

taeric wrote: post #24

Earlier quoted context omitted.

I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had…

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview. Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who nee…

> If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it

Hear me, nameless internet stranger, that you might learn from my mistakes: This is not true. Efficiency is only worth money if it increases profits - concretely this means your efficiency gain must result in the following:

1. Delivering features faster

2. Delivering features with meaningfully higher quality

3. Delivering the same features with lower headcount costs

(2) is hard to measure, so you can generally only sell on (1) and (3). During the 80s the business world learned the hard way you can throw away a lot of money on useless efficiency. If that interests you I recommend "The Goal" by Goldratt for a fictionalized account of those learnings.

* There's also a subtler point that corporate finances may mean that even if efficiency is perfectly captured, a 1% efficiency increase may only be worth 0.1% of your salary.

Re: Meta developer tools: Working at scale

#123

Post by Spooky23 » Wed, Jun 28, 2023, 6:24 PM UTC

zug_zug wrote: post #29

taeric wrote: post #24

Earlier quoted context omitted.

I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had…

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview. Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who nee…

Splunk can save money, as long as you’re intelligent about processing data. I can think of dozens of security and operational incidents where splunk saved the day.

I’ve been through a few savings exercises where everything is gonna go open source dashboards etc. Once the guy who understands how it all works disappears, it migrates back to splunk.

One security incident response will pay for the entire splunk infrastructure.

Re: Meta developer tools: Working at scale

#124

Post by Solvency » Wed, Jun 28, 2023, 6:42 PM UTC

MikeTheRocker wrote: post #71

optymizer wrote: post #60

Earlier quoted context omitted.

For what it's worth, as a Meta employee, I generally disagree with this comment. I mean, calling it 'inferior' is just laughable. It's way better than what open source projects have access to and it's way better than Amazon internal tooling (is that 'industry standard' enough?). The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's…

Having also written Android code at Meta, I 100% agree.

What's the short version of why Meta did this, wrapping so much up in this internal nonstandard format?

Re: Meta developer tools: Working at scale

#125

Post by cletus » Wed, Jun 28, 2023, 6:45 PM UTC

aseipp wrote: post #95

cletus wrote: post #53

I've worked for both Facebook and Google so can make informed comments on this with two exceptions: Buck2 came after I left and I'm honestly not sure what sapling is. Is it some Mercurial-like re-implementation a bit like how Google's Piper is a re-implementation of Perforce? The tl;dr is that Google's developer tooling and ifnrastructure is superior in almost every way. Examples: - When I started at FB we used Nucli…

Buck2 is IMO much better than even Bazel is from a design POV, because it actually cleanly separates all user rules from the build engine, and has a coherent modern design around a sound theoretical basis. Neil, one of the leads and author of this post, has written many build systems, so it's not like he's unaware of Bazel; his taxonomy of build systems and ones like Bazel in "Build Systems a la Carte" is worth readi…

I hope Buck2 is better. Like I said, I have no experience with it.

My issue is more organizational. Meta, as a company, in my experience, does not put in sufficient investment to build a mature, robust open source project with few exceptions. Even things like the www test infra, which are core to the company when I was there had ~1 FTE SWE. That's not open source but you get my point. Buck (and Thrift) seem to have been woefully underinvested for years. Thrift was originally an intern project at a time when Stubby (Google's protobuf-based RPC) was not open source.

Re: Meta developer tools: Working at scale

#126

Post by dnissley » Wed, Jun 28, 2023, 6:48 PM UTC

castlecrasher2 wrote: post #51

rangledangle wrote: post #42

Earlier quoted context omitted.

No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. I always attributed that to their internal tools, since most of the other companies seem to be using the same crap. Slack is straight up painful compared to their chat system. And don't get me started on the how good the task tool is compared to literally any other ticketing system out there. Everything is behind th…

>Slack is straight up painful compared to their chat system. As someone who vastly prefers Slack over all its competitors, I'm very interested in hearing how Meta's compares. What makes it better than Slack for you?

Don't believe the hype -- I think workchat probably beats slack for speed in the web client, but the desktop/android clients which I use regularly are truly sluggish and constantly refusing to load new messages if I'm not constantly using them, forcing me to restart them in order to do so.

Re: Meta developer tools: Working at scale

#127

Post by dnissley » Wed, Jun 28, 2023, 6:51 PM UTC

giraffe_lady wrote: post #80

phyrex wrote: post #67

Earlier quoted context omitted.

It's also doing a lot of good. It's in the nature of any broadcasting and communications platform

"drunk driving may kill a lot of people, but it also helps a lot of people get to work on time, so, it;s impossible to say if its bad or not,"

"driving helps a ton of people get around, but sometimes people drive drunk and kill other people, so it's definitely unforgivingly bad"

Re: Meta developer tools: Working at scale

#128

Post by throwaways885 » Wed, Jun 28, 2023, 6:55 PM UTC

rangledangle wrote: post #42

zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. I always attributed that to their internal tools, since most of the other companies seem to be using the same crap. Slack is straight up painful compared to their chat system. And don't get me started on the how good the task tool is compared to literally any other ticketing system out there. Everything is behind th…

Anyone who's used both, how does it compare to Google's internal tooling?

Re: Meta developer tools: Working at scale

#129

Post by cletus » Wed, Jun 28, 2023, 6:59 PM UTC

ahahahahah wrote: post #116

cletus wrote: post #53

I've worked for both Facebook and Google so can make informed comments on this with two exceptions: Buck2 came after I left and I'm honestly not sure what sapling is. Is it some Mercurial-like re-implementation a bit like how Google's Piper is a re-implementation of Perforce? The tl;dr is that Google's developer tooling and ifnrastructure is superior in almost every way. Examples: - When I started at FB we used Nucli…

Everyone reading this should assume it's talking about FB around 2014 or so, I don't know how it could otherwise be so wrong. > FB's code search is just grep. It's literally called BigGrep. This hasn't been true for a long time, codesearch at FB is more complex than just grep and has some semantic understanding. Here's some discussion here when some of this infrastructure was open sourced: https://news.ycombinator.co…

As for Glean, you concede "some semantic understanding". All I know is as of 2 years ago I couldn't do "Find Usages" of a particular C++ method where I could Google cs 6+ years ago.

I did spend way more time doing www than C++ though and Hack was specifically written to facilitate regex searches like being able to search for "SomeClassName::someFunctionName" to find usages as well as the other examples of prohibiting namespacing and type aliasing (in Hack). Google cs doesn't have that constraint.

> SrcFS and ObjFS aren't what solve this problem at Google

It's a mix. You can't just pull out one piece of the Google dev infra because it's all connected. In a P4 client, you'd list some paths that were "local" allowing local modifications. Forge, SrcFS, ObjFS, TAP and Sponge are all pieces of this puzzle.

> This is wrong since at least 2015 FB's build system had the build artifact caching

First, there's more to FB builds than infra C++, most notably iOS and Android, which are all built locally. It's why iOS/Android engineers have big, chunky machines like the iMac Pro or the trash can. If this has changed, it's a fairly recent change. Google builds mobile apps on Forge. There are literally racks of Mac Minis to build iOS. With this you can build artifact cachincg like you do with, say, Google3 Java or C++.

Second, "local" requires some further explanation. Typically, things are built on a devserver (although this was transitioning to on-demand VMs, for which the Google equivalent was CitC). But a devserver build required a full checkout and build with artifact caching. It could then be incremental until an hg pull forced a larger rebuild. There were sparse checkouts but I think the support was pretty limited and only worked on certain infra projects.

Either way, the whole FB C++ build experience was fairly primitive and even worse for iOS/Android.

Re: Meta developer tools: Working at scale

#130

Post by spookthesunset » Wed, Jun 28, 2023, 7:05 PM UTC

jonathankoren wrote: post #113

morgannewman wrote: post #72

Earlier quoted context omitted.

You might be surprised to learn that it was not and still is not a solved problem for companies like Meta. Here is an earlier write-up about this topic from Meta: https://engineering.fb.com/2014/01/07/core-data/scaling-merc...

You might be surprised to learn that they're solving the wrong problem.

I don't think you fully understand how big their codebase is and how many different teams are working on it at any given time.

There are very fundamental differences between a mono-repo and a bunch of repos for each "service" or whatever. Lots of tradeoffs. I've worked both and I can see the reasons for huge monorepos. They make a lot of things that were previously hard much simpler... The tradeoff is your tooling needs to be able to scale with growth of the company. And for a company the size of FB, dedicating an entire team to improving the tooling for their monorepo is well worth it.