Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta developer tools: Working at scale

engineering.fb.com

Post reply on HN
Print view View original

191–200 of 228 posts

Re: Meta developer tools: Working at scale

#191

Post by ketchupdebugger »

zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

Every tool Meta has, Google also has. Google open source theirs so the industry gravitates towards that, but Meta keeps theirs internal. So is Meta's tooling better than Industry/Google open source tools? probably not. The only tool I was impressed by was scuba. But Meta basically build it out of gold (thousands of machines with TB's of ram)

Re: Meta developer tools: Working at scale

#192

Post by zug_zug »

vlovich123 wrote: post #159
zug_zug wrote: post #158

Earlier quoted context omitted.

>> Amazon internal tooling (is that 'industry standard' enough?). I'm not sure, maybe? When I say industry-standard tools I mean "Best you can purchase." So I mean a UI as good as github, a platform as good as AWS (meta's doesn't hold a candle to AWS), full-text log searching as good as splunk, chat as fast/searchable as slack (workplace doesn't hold a candle), video chat as fast/clear/clean as Zoom (workplace doesn'…

GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta. I found the tools at Meta made it a lot smoother and faster to write and review code. Also, their automated flaky test detection, suppression, & ticket issuance/resolution is no joke. One of my favorite features. Their feature flagging system was also on point. Also, Workplace is a fantastic replacement of Wiki tha…

>> GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta.

??

Re: Meta developer tools: Working at scale

#194

Post by lanza »

wilde wrote: post #160
optymizer wrote: post #60

Earlier quoted context omitted.

For what it's worth, as a Meta employee, I generally disagree with this comment. I mean, calling it 'inferior' is just laughable. It's way better than what open source projects have access to and it's way better than Amazon internal tooling (is that 'industry standard' enough?). The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's…

> Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. You’re measuring the wrong thing. The user actions are slow and take several seconds to complete usually. No one gives a shit how big the repo is. They care how long their operation takes.

There isn't a tool that can do this job faster. git takes a few seconds once you're at half a million commits. Facebook has many repos checked in in their entirety that are bigger than that.

Re: Meta developer tools: Working at scale

#195

Post by rsdbdr203 »

vlovich123 wrote: post #159
zug_zug wrote: post #158

Earlier quoted context omitted.

>> Amazon internal tooling (is that 'industry standard' enough?). I'm not sure, maybe? When I say industry-standard tools I mean "Best you can purchase." So I mean a UI as good as github, a platform as good as AWS (meta's doesn't hold a candle to AWS), full-text log searching as good as splunk, chat as fast/searchable as slack (workplace doesn't hold a candle), video chat as fast/clear/clean as Zoom (workplace doesn'…

GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta. I found the tools at Meta made it a lot smoother and faster to write and review code. Also, their automated flaky test detection, suppression, & ticket issuance/resolution is no joke. One of my favorite features. Their feature flagging system was also on point. Also, Workplace is a fantastic replacement of Wiki tha…

>Their feature flagging system was also on point.

Which one... there are like 3 (or more)? The ones I worked with were the single biggest cause of SEVs. I wrote some tooling around trying to make it better, but it honestly needed to be completely replaced.

>Workplace is a fantastic replacement of Wiki that surfaces relevant & interesting content to you.

Yea? I found it horrible to have to read through post-after-post-after-post to try and keep up-to-date with what's going on. Not my way of ingesting information.

>Their chat tools are way better than Google Chat / Microsoft Teams (not sure how it stacks up against Slack since I'm not a huge user of them).

100% disagree... telling my dog to bark at the neighbor because I want something from them is better than Workplace chat. Constantly switching between internal beta and production to try and get the features I wanted, with something stable. Go ahead and send me a link to that message in a chat... oh yea, you can't :-|

Re: Meta developer tools: Working at scale

#196

Post by rsdbdr203 »

Hermitian909 wrote: post #122
zug_zug wrote: post #29

Earlier quoted context omitted.

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview. Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who nee…

> If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it Hear me, nameless internet stranger, that you might learn from my mistakes: This is not true. Efficiency is only worth money if it increases profits - concretely this means your efficiency gain must result in the following: 1. Delivering features faster 2. Delivering features with meaningfully higher quality 3. Deliverin…

100% this... profit, or it's not worth it

Re: Meta developer tools: Working at scale

#197

Post by rsdbdr203 »

Spooky23 wrote: post #123
zug_zug wrote: post #29

Earlier quoted context omitted.

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview. Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who nee…

Splunk can save money, as long as you’re intelligent about processing data. I can think of dozens of security and operational incidents where splunk saved the day. I’ve been through a few savings exercises where everything is gonna go open source dashboards etc. Once the guy who understands how it all works disappears, it migrates back to splunk. One security incident response will pay for the entire splunk infrastru…

This isn't a great argument... sure, Splunk "saved the day" because it caught that really bad security thing... but what other tools, at half the cost, could have also "saved the day"? If most of them could have, then it's not really worth the price... right?

Re: Meta developer tools: Working at scale

#198

Post by rsdbdr203 »

findjashua wrote: post #56
zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

phabricator & scuba are far better than anything i've used anywhere else.

I was very impressed by Scuba's ability to ingest and search data... and very unimpressed by the interface. I found it terribly unintuitive, and often didn't get you what you wanted without having to run multiple queries.

Phabricator was pretty sweet... and the internal version of mercurial was a dream!

Re: Meta developer tools: Working at scale

#199

Post by rsdbdr203 »

jonathankoren wrote: post #76
rigelbm wrote: post #39

Earlier quoted context omitted.

> This is a solved problem. Hosting a gigantic monorepo for 25K concurrent users is so far from a solved problem.

[flagged]

I disliked a lot of things about working at Meta, the monorepo was not one... it was truly amazing, and very well done.

Re: Meta developer tools: Working at scale

#200

Post by erik_seaberg »

vlovich123 wrote: post #188
erik_seaberg wrote: post #169

Earlier quoted context omitted.

Suppressing flaky tests is a little terrifying. Google would just retry them a few times, which is slow but only delays requesting a code review.

It determines flakiness in aggregate across all builds. It also does automatic retries but it does so for the purposes of marking a run as flaky which it then remembers. And it's your team's responsibility for making your tests stable. If they're not stable, they're not going to fail builds which is the correct response (a ticket is filed & it's your team's job to keep those tickets under control). The neat thing is…

If a test goes from “flaky” to “never succeeds,” I absolutely want to fail builds. To me it seems pretty common for large integration tests to be flaky, and I don’t know that it’s always feasible to avoid without mocking so much that the test becomes meaningless.
Post reply on HN