Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta developer tools: Working at scale

engineering.fb.com

Post reply on HN
Print view View original

211–220 of 228 posts

Re: Meta developer tools: Working at scale

#211

Post by crakhamster01 »

castlecrasher2 wrote: post #51
rangledangle wrote: post #42

Earlier quoted context omitted.

No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. I always attributed that to their internal tools, since most of the other companies seem to be using the same crap. Slack is straight up painful compared to their chat system. And don't get me started on the how good the task tool is compared to literally any other ticketing system out there. Everything is behind th…

>Slack is straight up painful compared to their chat system. As someone who vastly prefers Slack over all its competitors, I'm very interested in hearing how Meta's compares. What makes it better than Slack for you?

Slack tries to satisfy a number of requirements, all through a chat interface. 1:1 chatting, group chatting, team/project discussions, company updates, etc. I think Slack is as good or superior to Workplace just considering chat functionality, but as a tool for inter-company collaboration Workplace is much better.

Once your company reaches a sufficient size (maybe 300+ employees?), Slack channels just become so clunky and tedious for trying to keep up with projects and discussions. Using Slack keyword notifications helps cut through the noise, but I really think applying Facebook's ranking/Groups expertise to work-related content has been the best solution I've seen so far. You're no longer inundated with unread channels that may or may not have messages relevant to you, Workplace just surfaces the relevant content (and obviously there's chat/mentions for more pressing issues).

Re: Meta developer tools: Working at scale

#212

Post by vlovich123 »

erik_seaberg wrote: post #210
vlovich123 wrote: post #205

Earlier quoted context omitted.

What makes you think that’s not the case?

Maybe I misinterpreted “flaky test detection, suppression” and “not going to fail builds.” You retry until a confirmed success before merging?

I think flaky tests would be always retried and if they start failing 100% then it’s marked as a pure regression. But they don’t block merges if they fail (ie it might be done only after the merge). Not 100%. But generally yes - the culture is to incentivize people to fix their flaky tests and it’s one of the things EMs are judged on.

> Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.

Re: Meta developer tools: Working at scale

#213

Post by ak67 »

quadrifoliate wrote: post #90
randall wrote: post #81

Earlier quoted context omitted.

https://sapling-scm.com/ It did work out.

I know that, I meant the specific reasons why Sapling diverged from standard Mercurial. https://sapling-scm.com/docs/introduction/differences-hg/ lays out some of the differences; but not why they exist. One guess that I can make from Sapling docs is that perhaps FB/Meta needed an ability to pull in Git repos where needed; and I can see Mercurial devs not being super enthusiastic about that being a first-class workfl…

> I know that, I meant the specific reasons why Sapling diverged from standard Mercurial

The needs of a large internal code base are often quite different than what open source projects need.

Anther issue is that tools like Git and Mercurial need to be compatible with positively ancient repositories and can't really break backwards compatibility whereas within Meta, it is easier to move faster and deal with breaking format changes since developers's checked out repos on their laptops can be silently upgraded since the machines are well managed with Chef and can be upgraded behind the scenes. In the outside world, you can't assume people will be running the latest version and can't upgrade them.

Finally, Google is also moving away from Mercurial. Perhaps the JJ developers want to respond why (and check out Jujitsu -- it's a novel and very interesting system y'all should check out).

Re: Meta developer tools: Working at scale

#214

Post by ak67 »

roland35 wrote: post #206
jarjoura wrote: post #163

Earlier quoted context omitted.

As an ex-meta mobile engineer, I 100% agree that I found Eden to be impossible to use productively. I always had to pre-warm the cache, pretty much defeating its stated goals. Maybe it has gotten better since I left a year ago, but I had 2 TB of local storage, and mercurial's sparse checkouts were more than enough for me and far more reliable. However, all the other tooling around code were just superior in every way…

Eden definitely works better on a dev server and sucks on a laptop, especially arm based mac's (it's slowly getting better but still requires occasional reboots). +1 on the coding tools being great

Please reach out to the Eden team about your M1 issues; they'd love to hear from you - from what I've heard, it works better on M1s.

Re: Meta developer tools: Working at scale

#215

Post by ak67 »

likpok wrote: post #121
blitz_skull wrote: post #4

Sapling looks quite cool! I've used git extensively in my career and consider myself as having a slightly-more-advanced-than-typical understanding of how to use it just based on conversations with colleagues. However, one thing that's always been very limiting with git has been stack-based PR reviews, and as they mentioned amending deep commits. It's not impossible, but it makes it awkward enough that I usually avoid…

When we first switched from git to hg internally I was real cranky. Git was the clear industry winner, and hg was adding a whole bunch of churn for no good reason. Now, hg is amazing and when I leave I will be extremely sad to go back to git. The UX is well thought out, with commands mapping to operations (how do you undo a commit? hg uncommit vs git playing around with the reflog). Amending deep commits is pretty go…

You can always use Sapling. Many x-Meta people are still contributing to it on Github and use it for their git projects.

Re: Meta developer tools: Working at scale

#216

Post by ak67 »

jonathankoren wrote: post #179
spookthesunset wrote: post #130

Earlier quoted context omitted.

I don't think you fully understand how big their codebase is and how many different teams are working on it at any given time. There are very fundamental differences between a mono-repo and a bunch of repos for each "service" or whatever. Lots of tradeoffs. I've worked both and I can see the reasons for huge monorepos. They make a lot of things that were previously hard much simpler... The tradeoff is your tooling ne…

Sit down. I worked at FB.

One year in 2014/2015?

Saying that Meta "reinvented CVS" means you either know very little about source control or you are just being purposely misleading. Neither is a very good look.

Re: Meta developer tools: Working at scale

#217

Post by martinvonz »

ak67 wrote: post #213
quadrifoliate wrote: post #90

Earlier quoted context omitted.

I know that, I meant the specific reasons why Sapling diverged from standard Mercurial. https://sapling-scm.com/docs/introduction/differences-hg/ lays out some of the differences; but not why they exist. One guess that I can make from Sapling docs is that perhaps FB/Meta needed an ability to pull in Git repos where needed; and I can see Mercurial devs not being super enthusiastic about that being a first-class workfl…

> I know that, I meant the specific reasons why Sapling diverged from standard Mercurial The needs of a large internal code base are often quite different than what open source projects need. Anther issue is that tools like Git and Mercurial need to be compatible with positively ancient repositories and can't really break backwards compatibility whereas within Meta, it is easier to move faster and deal with breaking…

> Finally, Google is also moving away from Mercurial. Perhaps the JJ developers want to respond why (and check out Jujitsu -- it's a novel and very interesting system y'all should check out).

Sure. I presented about that at Git Merge 2022. https://github.com/martinvonz/jj#disclaimer has links to the slides and the recording from there. I'll summarize the problems we have with Mercurial here:

1. Performance/scalability. That's partly because Python is slow. Both the Mercurial project and Meta have rewritten many parts of it in C and Rust as a result. Maybe more importantly, there are many assumptions in Mercurial's design that don't scale well. We have extensions for downloading only a slice of the repo. We slice it in both file space and in version space. However, it can get very expensive to change afterwards. For example, checking out an old revision that they user hasn't previously downloaded is very slow (it requires rewriting all local revisions after that point).

2. Consistency. Mercurial was designed for local file systems, so when we store repos in our distributed file system, we run into write races that can corrupt repos.

3. Integrations. We integrate with Mercurial by running the `hg` binary and parsing the output. That's unnecessarily complicated and slow.

We also see several opportunities by switching to jj (in addition to hopefully fixing the problems above):

1. Simpler workflows. Things like: working-copy commit (no "dirty working copy" errors, for example), undo, first-class conflicts (no interrupted rebases, for example). See the GitHub project for details.

2. Cloud-based repos. The repos will be stored in a database instead of being stored in files on top of a distributed file system. That makes them much easier for our server to work with, and it opens up for many kinds of integrations that were not feasible before.

3. Simpler architecture. We designed jj from the beginning to be easy to integrate with our internal systems, so there should be much fewer workarounds.

4. Simpler code base. You can typically add a command without worrying about concurrent commands, a dirty working copy, or conflicts. An example I like to mention is how I spent about two weeks trying to implement a command for amending into an ancestor commit in Mercurial. Then I implemented a more powerful version of that (can move changes from any commit to any other commit) in an hour in jj.

Re: Meta developer tools: Working at scale

#218

Post by ben_w »

rangledangle wrote: post #142
ben_w wrote: post #134

Earlier quoted context omitted.

> No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. This is a curious thing to see. As a mere user , and speaking purely about FB not any other Meta IP… Other than unbreaking things that break when the OS and browsers change under you, has Facebook shipped even one thing in the last five years? Don't get me wrong, I know it takes a lot of effort to stand still — fo…

Most of the open source Ai things are Meta. React, GraphQL, pytorch, rocksdb, docusaurus, prophet, a whole ton of internal tools that aren’t public knowledge. Full disclosure I haven’t worked there since 2019 though, so not sure since then.

None of that looks like it's about the Facebook website itself?

But I guess that implies you didn't mean the website itself when you wrote "FB" in the comment I replied to? :)

(One of the most well known consumer brands shipping tooling rather than stuff the consumers use directly is, ah, apt for a company called "meta").

Re: Meta developer tools: Working at scale

#219

Post by wilde »

lanza wrote: post #194
wilde wrote: post #160

Earlier quoted context omitted.

> Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. You’re measuring the wrong thing. The user actions are slow and take several seconds to complete usually. No one gives a shit how big the repo is. They care how long their operation takes.

There isn't a tool that can do this job faster. git takes a few seconds once you're at half a million commits. Facebook has many repos checked in in their entirety that are bigger than that.

That’s like saying cargo ships are fast because they carry lots of stuff. It’s still slow in absolute terms to get my stuff from China.

Re: Meta developer tools: Working at scale

#220

Post by wilde »

optymizer wrote: post #209
wilde wrote: post #160

Earlier quoted context omitted.

> Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. You’re measuring the wrong thing. The user actions are slow and take several seconds to complete usually. No one gives a shit how big the repo is. They care how long their operation takes.

Compared to what? Not having a monorepo? Sure, but you run into different issues with that approach. Eden is the best solution there is for a monorepo of this size.

Yeah compared to not having a monorepo and buying more SSDs. The separate repos are much nicer to work in, but harder for the infra folks to manage. I miss having dataswarm as a separate repo. I will miss IG’s.

I understand the tradeoff. That doesn’t mean it’s “fast”. It’s slow but the other alternatives are slower.

Post reply on HN