Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta developer tools: Working at scale

engineering.fb.com

Post reply on HN
Print view View original

181–190 of 228 posts

Re: Meta developer tools: Working at scale

#181

Post by sangnoir »

jonathankoren wrote: post #178
sangnoir wrote: post #120

Earlier quoted context omitted.

For a moment, let's assume they do abandon the monorepo. What solution would you recommend for managing code dependencies and coordinating releases between thousands of teams (at a modest 5 repos per team) - git tags?

This coordinating releases across teams is not a unique a problem. In fact, every large software organization solves this problem. They don’t usually do it in a assbackwards way due to institutional blindness.

You're deflecting. What solution would you recommend, since you disapprove of monorepo as a solution to this problem we both agree exists.

If you're not simultaneously updating the code and all it's references (i.e. a monorepo), you will need a version dependency graph system (with integrated with your build system). I'm yet to encounter one such tool that isn't awful to use[1]: monorepos are an improvement when you grow beyond a couple dozen repos. Git submodules aren't a good solution either. If you familiar with a decent tool/workflow that is not "institutionally blind", I'd love to learn more about it.

1. Gradle, Android's "repo", home-grown git-submodule-based build systems.

Re: Meta developer tools: Working at scale

#182

Post by rangledangle »

no_wizard wrote: post #177
rangledangle wrote: post #42

Earlier quoted context omitted.

No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. I always attributed that to their internal tools, since most of the other companies seem to be using the same crap. Slack is straight up painful compared to their chat system. And don't get me started on the how good the task tool is compared to literally any other ticketing system out there. Everything is behind th…

Its not the tools, its the culture. Meta cut red tape in its early years especially. I know there is more "process" now, but they empowered teams and individuals to just make decisions and move on and the ecosystem around everything supports this. They'd be just as fast with Slack or whatever, for the most part. EDIT: as someone posted below: > The open-ness of code, visibility, diffs. It was perfection. Something br…

100% agree. The culture is a huge factor as well, but in some of the environments that I encountered before and after, that culture was there, but the tools didn’t allow the same cohesion.

To your point, other experiences have shown me the ego that rears it’s ugly head when trying to move that way in an env that didn’t have that culture. At FB there was a lot of candor, but in other environments I feel like I’m going to hurt someone’s feelings in code review, or even just watching what I say in slack messages. People take work too seriously in some companies. It’s so rewarding to have fun with it.

They used to say “Nothing at FB is someone else’s problem”

I hope they still do.

Re: Meta developer tools: Working at scale

#183

Post by throwaway1777 »

counters wrote: post #14
mrweasel wrote: post #6

Someone is going to read this and start to retooling their five person developer organisation because: "Facebook uses it". It's funny that the Sapling command is "sl", that's going to conflict with installations of "stream locomotive".

lol I've suffered under that before... except it was an ex-Googler forcing bazel on us, all of a 10 person dev team working on a codebase that was probably less than 15,000 LOC across three or four packages that just _had_ to be packaged into a monorepo.

Once it was setup was it really that painful? I’ve worked with buck and for day to day usage it didn’t really affect much.

Re: Meta developer tools: Working at scale

#184

Post by rangledangle »

no_wizard wrote: post #177
rangledangle wrote: post #42

Earlier quoted context omitted.

No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. I always attributed that to their internal tools, since most of the other companies seem to be using the same crap. Slack is straight up painful compared to their chat system. And don't get me started on the how good the task tool is compared to literally any other ticketing system out there. Everything is behind th…

Its not the tools, its the culture. Meta cut red tape in its early years especially. I know there is more "process" now, but they empowered teams and individuals to just make decisions and move on and the ecosystem around everything supports this. They'd be just as fast with Slack or whatever, for the most part. EDIT: as someone posted below: > The open-ness of code, visibility, diffs. It was perfection. Something br…

And just to add that last part. Moving fast was so damn cool. That shared understanding of responsibilities and freedom of making changes. In most cases of course they were added as reviewers to the revert diff review, and in some cases they knew it could adversely affect us and they’d tag us for our perspective before pushing, but the open-ness of it all. The freedom to bring me in at a later stage or for me to do the same to them.

These days at different companies it’s all planning and hashing out all those details before implementing, and then guess what? Surprises happen anyway. And in the end we become master planners and never actually build anything. Those blueprints sure do look great though.

Re: Meta developer tools: Working at scale

#185

Post by throwaway1777 »

cletus wrote: post #125
aseipp wrote: post #95

Earlier quoted context omitted.

Buck2 is IMO much better than even Bazel is from a design POV, because it actually cleanly separates all user rules from the build engine, and has a coherent modern design around a sound theoretical basis. Neil, one of the leads and author of this post, has written many build systems, so it's not like he's unaware of Bazel; his taxonomy of build systems and ones like Bazel in "Build Systems a la Carte" is worth readi…

I hope Buck2 is better. Like I said, I have no experience with it. My issue is more organizational. Meta, as a company, in my experience, does not put in sufficient investment to build a mature, robust open source project with few exceptions. Even things like the www test infra, which are core to the company when I was there had ~1 FTE SWE. That's not open source but you get my point. Buck (and Thrift) seem to have b…

But Meta has had a lot of cool open source projects over the years. React, cassandra, graphql, etc

Re: Meta developer tools: Working at scale

#186

Post by c_crank »

HumanOstrich wrote: post #180
c_crank wrote: post #30

Earlier quoted context omitted.

GitHub's gotten so much worse since Microsoft bought it. The drop in quality compared to when I used it in school is remarkable.

Can you provide some specific examples?

The token system used instead of passwords now was horribly explained. I had to look at several online resources to figure out how to actually use it.

I've noticed that the system seems to repeatedly throttle me where it never did before. Logging in often results in being given an error until I try it enough times.

Sometimes attempting to pull recent changes from a repo will fail, and tell me that I pulled the recent changes even though I am stuck on an old commit. I tried several things to fix this and the only thing that seems to work is doing a hard reset on the latest commit id. I never noticed github doing this before.

Re: Meta developer tools: Working at scale

#188

Post by vlovich123 »

erik_seaberg wrote: post #169
vlovich123 wrote: post #159

Earlier quoted context omitted.

GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta. I found the tools at Meta made it a lot smoother and faster to write and review code. Also, their automated flaky test detection, suppression, & ticket issuance/resolution is no joke. One of my favorite features. Their feature flagging system was also on point. Also, Workplace is a fantastic replacement of Wiki tha…

Suppressing flaky tests is a little terrifying. Google would just retry them a few times, which is slow but only delays requesting a code review.

It determines flakiness in aggregate across all builds. It also does automatic retries but it does so for the purposes of marking a run as flaky which it then remembers. And it's your team's responsibility for making your tests stable. If they're not stable, they're not going to fail builds which is the correct response (a ticket is filed & it's your team's job to keep those tickets under control). The neat thing is also that if you fix the test and it's no longer flaky, it gets automatically added back in & the ticket is closed (in case you forgot).

https://engineering.fb.com/2020/12/10/developer-tools/probab...

Re: Meta developer tools: Working at scale

#189

Post by xuancanh »

shepherdjerred wrote: post #148
yodsanklai wrote: post #103

Earlier quoted context omitted.

> If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world" SWEs need to learn new tools all the time. This shouldn't be a concern. Besides, it's not like "the real world" only use a single tool. Each company has their own tweaks, even those using open source stuff.

Big tech has _way_ better tools than what most companies use. At Amazon so many concerns were taken care of for you like: * How do I setup a new project? * How do I build code? * How do I pull internal dependencies? * How do I publish artifacts? Every company makes a horrible copy of these systems with off-the-shelf tools. At Amazon there were certainly flaws in the internal tools, but overall it was much better than…

I'm not quite sure if it is true that those tools are better than "the real world". There are many options for each of the steps that you are mentioned, old systems mixed with new solutions, so developers have to learn all of them and onboard to each of the system that they decided to use. While they provide some templates and scripts here and there to solve these problems, most of them are not production ready, and doesn't provide a holistic experience. I work for smaller companies before and the platform teams did a much better job to provide a unified experience across those steps.
Post reply on HN