I'm looking forward to developers setting up LLM prompts to make their code seem more complex and like it required more effort.
Show HN: Weave - actually measure engineering productivity
31–40 of 40 posts
Re: Show HN: Weave - actually measure engineering productivity
#32Let me just ignore my natural distain to the whole thing (as a engineer and a manager) > We’ve developed a custom model that analyzes code and its impact directly... This is a bold claim all things considering. Don't you need to fine tune this model for every customer as their business metrics likely vastly different? How do you measure the impact of refactoing? What about regressions or design mistakes that surface…
> Let me just ignore my natural distain to the whole thing (as a engineer and a manager) I totally get this - that's how I felt initially, but I was shocked to find that the vast majority of orgs are using bad metrics like LOC or commit counts anyway. Our belief is that replacing those with something much more accurate can help the entire industry.
I worked as a manager in a big tech company that used metrics such as amount of PRs done, amount of PR reviews and etc. At least with those when there were questions based on those metrics - I was able to contextualize and explain to upper management why for a particular eng they show a dip even when their performance is stellar.
How am I going to do that with your blackbox metrics when this need arises?
Also I don't have a Google account, so I can't even get pass your frontpage that has no info?
Re: Show HN: Weave - actually measure engineering productivity
#33If you build something that doesn't solve problems with impact to the business, your real productivity is zero. How does this account for that? https://blog.pragmaticengineer.com/the-product-minded-engine...
You're totally right! And we do not account for it at all. We don't give any insights into product decisions, just engineering output. If engineering output is high but business results are bad, that indicates there's a problem somewhere else in the business. But it's still good to know that engineering output is high!
Re: Show HN: Weave - actually measure engineering productivity
#34What do you see as the major threats to validity for your approach?
Re: Show HN: Weave - actually measure engineering productivity
#35"Hello Jane, please have a seat. We need to talk about your productivity. Yes, I know you helped the team through a crunch and delivered the new feature, which works flawlessly and is loved by our users. And our balance sheet is much healthier after you found that optimization that saves us $1mm/year. We also appreciate that younger teammates look to you for guidance and learn a lot from you. But you see, the AI scor…
LOL - I shudder at the idea of a manager making HR decisions based solely on this one metric! To be clear we're not claiming this is 1 number to holistically evaluate an entire engineer. Rather we're giving a much more accurate picture of output, which most orgs are already measuring (with terrible accuracy). It should be an important part of the picture but certainly not the whole story! And fwiw I think the scores…
Shudder is a fair reaction. Except the manager is now forced by hr to make a hr decision because that way too level "leadership" wouldnt be culpable!
Re: Show HN: Weave - actually measure engineering productivity
#36How did you come up with those magic correlation numbers? Is this generally just sniffing surface quality and quantity of written code, or is consideration given to how architecturally sound the system is built, whether the features introduced and their implementations make sense, how that power is exposed to users and whether the UI is approachable and efficient, user-feedback resulting from the effort, long-term su…
> How did you come up with those magic correlation numbers? Evaluated on a proprietary data set of manually labelled PRs > Is this generally just sniffing surface quality and quantity of written code... Somewhere in between the two :) a PR with a poorly and quickly implemented login will have a lower output score than a PR with a robust, well-designed and tested login, simply because the latter is more effort. But th…
A hotfix PR that was 2 lines long, prevents (or managed) an outage incident, but was merged with no title, description, or review because time was of the essence.
Some of such changes have been my most impactful ones.
Re: Show HN: Weave - actually measure engineering productivity
#37Earlier quoted context omitted.
I'd like to add that you need way more information on the landing page before I'm going to do much more than let you have my email address (if that.) Right now its a black box that takes in data(?) and spits out... something?
Check out our main landing page: https://workweave.dev/ Let me know if you have any questions that aren't answered there!
I just want to inform you that the pricing section is effed up. It talks about FramerBite pricing - which I guess is the thing you used to throw this landing page together. That seems very low effort and I would estimate the output metric of that to be 1.03 with a correlation of 0.96.
Re: Show HN: Weave - actually measure engineering productivity
#38If you build something that doesn't solve problems with impact to the business, your real productivity is zero. How does this account for that? https://blog.pragmaticengineer.com/the-product-minded-engine...
Agreed. Focusing on outputs leads to busy bodies and doesn't necessarily mean better productivity, just more busyness. "We generated all this output over the past sprint! Nevermind the outcome that had new business value only required a few hours and a days to delivery... But we managed to take 5 times longer to get that to the customer!" I'd rather engineers slow down and think before just producing noise to show they were thinking. I see knowledge work management and business leadership is falling back to vanity again.
Re: Show HN: Weave - actually measure engineering productivity
#39Earlier quoted context omitted.
You are describing a feature factory where people ship features just to ship features. Shitloads of code gets shipped and yet there's no real forward momentum because none of it has any impact. It is a bad place to be.
Wouldn't this be considered a very high performing engineering team with a very poor product team? We're only evaluating engineering - not the product decisions behind that engineering.
Sure, a siloed organization. Some folks care about the whole process not just the work in front of them. In fact, one of the biggest demotivators and morale destroying things is doing work that isn't meaningful (and the person isn't just about making money for themselves which would make the work meaningful to them (some of us don't care to be mercenaries...).
Re: Show HN: Weave - actually measure engineering productivity
#40Earlier quoted context omitted.
That assumes all or almost all the work is writing the code, with no time allotted to actually using the app with that code written, benchmarking or other measurements, research about possible alternatives, etc.
Not at all! The algorithm is calibrated with real human effort. So find/replacing something 1000 times will have nowhere near the same value as adding 1000 lines of new code. And given 1000 lines of new code, you'll get the same value for implementing the same functionality in 100 lines instead. What we don't capture is any product or communication overhead - however our platform has other metrics which can help find…
In a complex, mature system, a high impact bug could have a very small fix that is highly non-obvious. Your metric assumes that the person shitting out 1000 lines of a new feature no one wants is equally as productive as a distributed systems wizard who can fix bugs no one else can figure out adding a 3 line fix for an issue that customers have been complaining about for years. It is inherently biased towards adding new features and against maintenance and system quality improvement.