Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking developer build times to decide if the M3 MacBook is worth upgrading

incident.io

Post reply on HN
Print view View original

361–370 of 432 posts

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#361

Post by FunnyLookinHat »

lawrjone wrote: post #81
Aurornis wrote: post #64

This is a great write-up and I love all the different ways they collected and analyzed data. That said, it would have been much easier and more accurate to simply put each laptop side by side and run some timed compilations on the exact same scenarios: A full build, incremental build of a recent change set, incremental build impacting a module that must be rebuilt, and a couple more scenarios. Or write a script that…

I totally agree with your suggestion, and we (I am the author of this post) did spot-check the performance for a few common tasks first. We ended up collecting all this data partly to compare machine-to-machine, but also because we want historical data on developer build times and a continual measure of how the builds are performing so we can catch regressions. We quite frequently tweak the architecture of our codeba…

I think there's something to be said for the fact that the engineering organization grew through this exercise - experimenting with using telemetry data in new ways that, when presented to other devs in the org, likely helped them to all level up and think differently about solving problems.

Sometimes these wandering paths to the solution have multiple knock-on effects in individual contributor growth that are hard to measure but are (subjectively, in my experience) valuable in moving the overall ability of the org forward.

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#362

Post by nightski »

vessenes wrote: post #18
lawrjone wrote: post #8

Earlier quoted context omitted.

This is pretty much aligned with our findings (am the author of this post). I came away feeling that: - M1 is a solid baseline - M2 improves performance by about 60% - M3 Pro is marginal on the M2, more like 10% - M3 Max (for our use case) didn’t seem that much different on the M3 Pro, though we had less data on this than other models I suspect Apple saw the M3 Pro as “maintain performance and improve efficiency” whi…

Yeah, agreed. I'll say I do use the M3 Max for Baldur's gate :). On LLMs, the issue is largely that memory bandwidth: M2 Ultra is 800GB/s, M3 Max is 400GB/s. Inference on larger models are simple math on what's in memory, so the performance is roughly double. Probably perf / watt suffers a little, but when you're trying to chew through 128GB of RAM and do math on all of it, you're generally maxing your thermal budget…

H100 is kind of a poor comparison. There are much cheaper ways to get to decent memory without that. Such as 2 A6000s.

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#363

Post by dimask »

Aurornis wrote: post #64

This is a great write-up and I love all the different ways they collected and analyzed data. That said, it would have been much easier and more accurate to simply put each laptop side by side and run some timed compilations on the exact same scenarios: A full build, incremental build of a recent change set, incremental build impacting a module that must be rebuilt, and a couple more scenarios. Or write a script that…

> This is a great write-up and I love all the different ways they collected and analyzed data.

> [..] due to inherent biases in the sample set [..]

But that is an analysis methods issue. This serves as a reminder that one cannot depend on AI-assistants when they are not themselves enough knowledgeable on a topic. At least for the time being.

For once, as you point, they conducted a t-test on data that are not independently sampled, as multiple data points were sampled by different people, and there are very valid reasons to believe that different people would have different tasks that may be more or less compute-demanding, which confound the data. This violates one of the very fundamental assumptions of the t-test, which was not pointed out by the code interpreter. In contrast, they could have modeled their data with what is called "linear mixed effects model" where stuff like person (who the laptop belongs to) as well as possibly other stuff like seniority etc could be put into the model as "random effects".

Nevertheless it is all quite interesting data. What I found most interesting is the RAM-related part: caching data can be very powerful, and higher RAM brings more benefits than people usually realise. Any laptop (or at least macbook) with more RAM than it usually needs has most of the time its extra RAM filled by cache.

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#364

Post by minimaul »

smoldesu wrote: post #358
minimaul wrote: post #349

Earlier quoted context omitted.

Being Mac only can be an advantage - I’ve been on both sides of trying to maintain & use non-trivial dev environments and the more OSes you bring in for people to work on, the harder it gets. Bringing in Windows or Linux has a set up cost and a maintenance cost that may exclude it from even being considered. Edit: plus, Macs are ARM, other options are inevitably x86. So it’s also two CPU architectures to maintain sup…

My biggest issue with Mac-only shops is that almost nobody actually deploys to Mac. The majority of Mac-only firms I've worked at deploy to x86 Linux and develop in a VM on their Macbook (even pre-M1). Unless your business is writing Mac-native apps, MacOS is probably going to be a second-class deployment platform for you. Even in an ideal scenario where your app already works on ARM, you will be dealing with OS-spec…

These are fair points, and definitely a rough spot.

Eg at work we use M1/M2 macs and dev on those using docker - so that’s a Linux VM essentially with some nice tooling wrapped around it.

We certainly see differences - mostly around permissions (as docker for Mac doesn’t really enforce any access checks on files on the host), but we also mostly deploy to ARM Linux on AWS.

We went Mac only from a mix of Linux, Windows and Mac as we found the least overall friction there for our developers - Windows, even with WSL, had lots of problems, including performance issues. Linux we had issues finding nice laptops, and more support issues (developers are often not *nix experts!). Mac was a nice middle ground in the end.

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#365

Post by euos »

gizmo wrote: post #228
euos wrote: post #223

I am ex-core contributor Chromium and Node.js and current core contributor to gRPC Core/C++. I am never bothered with build times. There is "interactive build" (incremental builds I use to rerun related unit tests as I work on code) and non-interactive build (one I launch and go get coffee/read email). I have never seen hardware refresh toggle non-interactive into interactive. My personal hardware (that I use now and…

Somehow programmers have come to accept that a minuscule change in a single function that only result in a few bytes changing in a binary takes forever to compile and link. Compilation and linking should be basically instantaneous. So fast that you don't even realize there is a compilation step at all. Sure, release builds with whole program optimization and other fancy compiler techniques can take longer. That's fin…

I am firmly in test-driven development camp. My test cases build and run interactively. I rarely need to do a full build. CI will make sure I didn’t break anything unexpected.

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#366

Post by euos »

cole-k wrote: post #276
gizmo wrote: post #228

Earlier quoted context omitted.

Somehow programmers have come to accept that a minuscule change in a single function that only result in a few bytes changing in a binary takes forever to compile and link. Compilation and linking should be basically instantaneous. So fast that you don't even realize there is a compilation step at all. Sure, release builds with whole program optimization and other fancy compiler techniques can take longer. That's fin…

I mean yeah I've come to accept it because I don't know any different. If you can share some examples of large-scale projects that you can compile to test locally near-instantly - or how we might change existing projects/languages to allow for this - then you will have my attention instead of skepticism.

That’s why I write test first. I don’t want to build everything.

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#367

Post by euos »

white_dragon88 wrote: post #234
euos wrote: post #223

I am ex-core contributor Chromium and Node.js and current core contributor to gRPC Core/C++. I am never bothered with build times. There is "interactive build" (incremental builds I use to rerun related unit tests as I work on code) and non-interactive build (one I launch and go get coffee/read email). I have never seen hardware refresh toggle non-interactive into interactive. My personal hardware (that I use now and…

5+ year old i7 are potato and would be a massive time waster today. Build times matter.

I don’t notice myself sitting and waiting for a build. I don’t want to waste my time setting up a new workstation so why bother?

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#368

Post by chiefalchemist »

majikandy wrote: post #147
chiefalchemist wrote: post #134

Earlier quoted context omitted.

Spot on. The mind often needs time and space to breathe, especially after it's been focused and bearing down on something. We're humans, not machines. Creativity (i.e., problem solving) needs to be nurtured. It can't be force fed. More time working doesn't translate to being more effective and more productive. If that were the case then why are a disproportionate percentage of my "Oh shit! I know what to do to solve…

I love those moments. Your brain has worked on it in the background like a ‘bombe’ machine cracking the day’s enigma code. And suddenly “ding… the days code is in!”

You might like the book "Your Brain on Work" by Dr David Rock. In fact, I'm due for a re- read.

https://davidrock.net/about/

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#369

Post by euos »

syntaxing wrote: post #229
euos wrote: post #223

I am ex-core contributor Chromium and Node.js and current core contributor to gRPC Core/C++. I am never bothered with build times. There is "interactive build" (incremental builds I use to rerun related unit tests as I work on code) and non-interactive build (one I launch and go get coffee/read email). I have never seen hardware refresh toggle non-interactive into interactive. My personal hardware (that I use now and…

Aren’t M series screen and storage speed significantly superior to your Intel MBP? I transitioned from an Intel MBP to M1 for work and the screen was significantly superior (not sure about storage speed, our builds are all on a remote dev machine that is stacked).

I only use laptop screen in emergencies. Storage is fast enough.

Re: Tracking developer build times to decide if the M3 MacBook is worth upgrading

#370

Post by euos »

steeve wrote: post #301
euos wrote: post #223

I am ex-core contributor Chromium and Node.js and current core contributor to gRPC Core/C++. I am never bothered with build times. There is "interactive build" (incremental builds I use to rerun related unit tests as I work on code) and non-interactive build (one I launch and go get coffee/read email). I have never seen hardware refresh toggle non-interactive into interactive. My personal hardware (that I use now and…

This is because you’ve been spoiled by Bazel. As was I.

One day I will learn cmake. But not today :)
Post reply on HN