Live data from Hacker News

Meta developer tools: Working at scale

engineering.fb.com

21–30 of 228 posts

Re: Meta developer tools: Working at scale

#21
post #12

This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"

It's true. Exiting FB's incredible in-house information & developer tools culture is for many people one of the hardest things about leaving. Ex-fb groups maintain lists of potential "in the wild" replacements for each tool but very few are up to the task (pun intended).

Re: Meta developer tools: Working at scale

#22
post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

It's interesting to hear the contrast between Meta and my experience talking with ex-Googlers who complain that open-source or industry standard infrastructure is far inferior to what they were used to at Google. Why such a big chasm between the internal tool quality at Meta and Google?

Re: Meta developer tools: Working at scale

#23
post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

Yeah, I've worked at another large tech company who had accumulated a lot of in house tooling with a lot of invested SMEs who had been there a long time and missed that the state of the art out in the world had passed them by.

Re: Meta developer tools: Working at scale

#24
post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had to mirror any repository system locally and learn how to set that up. Probably getting it wrong in the process.

Mentioning splunk, I'm assuming you are using more paid industry tools. I suspect that opens things up a bit more, but realize that most of the industry doesn't use those due to prohibitive pricing. And I don't even know what the appropriate paid tool for building would be.

We seem remarkably primed to just hate whatever tooling we have at our disposal in ways that baffles me.

Re: Meta developer tools: Working at scale

#25
post #24
post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had…

I remember when Cisco mentioned potential acquisition of Splunk for $20B, the jokes about whether it was to buy the company or to pay the running Splunk bill wrote themselves.

Re: Meta developer tools: Working at scale

#26
post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

Yeah, that's fair. I think another thing is that a lot of these internal tooling projects date back 5-10 years when open source alternatives may not have been viable yet. For example, my company uses an in-house written time series database, which probably made sense at the time because when the project started, Prometheus wasn't 1.0 yet. Now it's reaching growing pains as we've scaled, and at this point it's a bit of a sunk cost fallacy to not migrate off of it.

Another thing is that sometimes it's truly impossible to use one-size fits all for some tools if your company scale is large enough. I've heard from friends at Amazon that the dev experience and tools are totally different in the hardware space than it is for those working in the retail or AWS orgs.

Re: Meta developer tools: Working at scale

#27
post #22
post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

It's interesting to hear the contrast between Meta and my experience talking with ex-Googlers who complain that open-source or industry standard infrastructure is far inferior to what they were used to at Google. Why such a big chasm between the internal tool quality at Meta and Google?

FWIW I am a Meta-to-Google transplant and I feel the opposite way. There’s a lot of internal tooling and functionality I miss dearly. Most of all, Workplace.

Re: Meta developer tools: Working at scale

#28
post #21
post #12

This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"

It's true. Exiting FB's incredible in-house information & developer tools culture is for many people one of the hardest things about leaving. Ex-fb groups maintain lists of potential "in the wild" replacements for each tool but very few are up to the task (pun intended).

It doesn't sound different than working anywhere else for an extended time. You get used to a certain way of doing things.

Re: Meta developer tools: Working at scale

#29
post #24
post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had…

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview.

Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal.

So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors.

Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.

Re: Meta developer tools: Working at scale

#30
post #9

Sapling looks quite cool! I've used git extensively in my career and consider myself as having a slightly-more-advanced-than-typical understanding of how to use it just based on conversations with colleagues. However, one thing that's always been very limiting with git has been stack-based PR reviews, and as they mentioned amending deep commits. It's not impossible, but it makes it awkward enough that I usually avoid…

Probably not, if only for reasons of inertia. Git plus third-party review tools (like GitHub) is more than "good enough" for most purposes. I used to work at FB, and while sapling is quite nice to use in practice, without the internal version of Phabricator to do code review (and, in all likelihood, mononoke), I don't think I'd pick it up again.

GitHub's gotten so much worse since Microsoft bought it. The drop in quality compared to when I used it in school is remarkable.
Post reply on HN