Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta developer tools: Working at scale

engineering.fb.com

Post reply on HN
Print view View original

31–40 of 228 posts

Re: Meta developer tools: Working at scale

#31

Post by stcroixx »

voidfunc wrote: post #16
yawnxyz wrote: post #12

This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"

No see, what happens is you leave then go to some other company and complain about how shitty their tools are then build a crappy half-baked version of whatever you had at $BigTechCorp and when shit hits the fan you boomerang back to $BigTechCorp for a sweet promo and raise.

I avoid hiring people coming from places like this for reasons similar. Extends past tools into libraries and frameworks, databases and other middleware, developer and business workflows. Only knowing proprietary stuff is a handicap. Assuming that proprietary stuff is superior because the big ad companies prefer it is even worse.

Re: Meta developer tools: Working at scale

#32

Post by nicce »

ksec wrote: post #18
yawnxyz wrote: post #12

This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"

Unpopular opinion. May be this is a feature not a bug? They want to filter out some Resume-Driven developers.

… lock developers into the ecosystem and prevent them leaving, since their knowledge is not so valuable elsewhere.

A bit cynical, but maybe not so unpopular opinion.

Meta as business is all about locking users to the ecosystem.

Re: Meta developer tools: Working at scale

#33

Post by quadrifoliate »

zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

I don't know anything about tooling at Meta, but I have appreciated custom-built tooling at all the jobs I have worked at so far -- much smaller companies ranging from about 100 to 5000 people.

Some of this is just having worked in the industry for a while. Guess what, 15 years ago "Put all your code in a web service running on AWS" was not nearly as slam-dunk a proposition as it is now; and tools optimized for on-prem were faster and more reliable.

The other thing is that you don't always have to reinvent things to the point that Meta does. You can just wrap around an Open Source project, or even contribute to it. This is what all of my previous jobs did -- make custom tooling from industry-standard OSS building blocks; and contribute back when it made sense for the larger community.

I actually think Meta was actually trying to do this with Mercurial a while ago [1], you probably have a much better idea than us as to why that didn't work out.

----------------------------------------

[1] https://engineering.fb.com/2014/01/07/core-data/scaling-merc...

Re: Meta developer tools: Working at scale

#34

Post by taeric »

zug_zug wrote: post #29
taeric wrote: post #24

Earlier quoted context omitted.

I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had…

I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview. Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who nee…

Even in a fang, most money spent on log storage is fully wasted. The article yesterday about monitoring being a pain was spot on.

More amusing is when you see someone build a giant Elastic Search pipeline to a Kibana dashboard so that they can get what would have been table stakes metrics if they had used the normal service templates. Without a ridiculously large bill.

Or folks that think they can warp around the high cardinality traps of making a metric out of everything. Assuming if you can make a system that works for the testing environment, of course it will work when you open the floodgates.

Seen people argue that sampling shouldn't be used, "because you could miss data?" Reservoir sampling is a thing, for very real reasons.

At any rate, I don't mean to just yell about splunk, from all I've heard it is nice. I am annoyed that folks seem to ignore the OLAP and OLTP divide, such that they think your metrics system should somehow be optimized for both. At the same time.

Pulling back to builds, though. What, exactly, is the gold standard in industry? I have yet to see it. Python builds, in particular, strike me as not good.

Re: Meta developer tools: Working at scale

#36

Post by jonathankoren »

I stopped reading when I reached that they made their own CVS.

This is a solved problem, and even if it isn’t, there’s entire communities dedicated to it. There’s literally no reason why Facebook needs to be in the business of reinventing the wheel from scratch beyond some dude trying to show “impact”. It’s a distraction from core business problems.

Re: Meta developer tools: Working at scale

#37

Post by ericbarrett »

Macha wrote: post #23
zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

Yeah, I've worked at another large tech company who had accumulated a lot of in house tooling with a lot of invested SMEs who had been there a long time and missed that the state of the art out in the world had passed them by.

I left FB in the early-mid 2010s and can attest to this. It took a year or so to adjust to the "real world." Things like, no, you don't have a fancy scheduler; you get SSH, ansible, and systemd timers. Docker, Jenkins, and Kubernetes (RIP Mesos) were all new to me when other devs/devops folks had years of experience.

Re: Meta developer tools: Working at scale

#38

Post by patrec »

skizm wrote: post #15
yawnxyz wrote: post #12

This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"

Most (all?) of these are open source and can be used outside of Meta infra. https://github.com/facebook/sapling

> Mononoke is the server-side component of Sapling SCM.

> While it is used in production within Meta, it currently does not build in an

> open source context and is not yet supported for external usage.

Re: Meta developer tools: Working at scale

#39

Post by rigelbm »

jonathankoren wrote: post #36

I stopped reading when I reached that they made their own CVS. This is a solved problem, and even if it isn’t, there’s entire communities dedicated to it. There’s literally no reason why Facebook needs to be in the business of reinventing the wheel from scratch beyond some dude trying to show “impact”. It’s a distraction from core business problems.

> This is a solved problem.

Hosting a gigantic monorepo for 25K concurrent users is so far from a solved problem.

Re: Meta developer tools: Working at scale

#40

Post by bertil »

zug_zug wrote: post #19

I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree. Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only…

I would say it’s quite uneven: most tools were better than the state of the art when they were introduced.

For many, the world has moved on a lot since, and those tools feel obsolete but so embedded in practices that it’s unthinkable to apply better approaches. One example of that is tracking data sources: legacy is slow, dysfunctional, and even fairly straightforward questions time out because you have to load so much meta-data. That’s because no one really uses it: most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. They benefit from it through comments they copy in their review, so fixing it isn’t anyone’s priority.

Others have evolved because the internal demand pushed things forward. PMs want Deltoid to work, so that system has moved to be state-of-the-art or interestingly unique in many ways: scaling, not implementing MCC, integration with the metric definitions.

Post reply on HN