Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asana cleared 5 years of engineering work in 2 weeks with Codex

openai.com

Post reply on HN
Print view View original

61–70 of 103 posts

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#61

Post by eqvinox »

helloplanets wrote: post #58
eqvinox wrote: post #53

Earlier quoted context omitted.

A model isn't "aware" of anything. It's trained on things, and that training doesn't include time estimates for working with AI because that didn't exist yet.

There's pre and post training. What I was going after with "aware", is that the actual people working at the companies, training the models, are aware that people aren't mostly going to be implementing the plan by hand, if they've already made the plan in Claude Code or Codex. As for a specific Claude / GPT instance, "aware" would definitely be the wrong word choice there, but the instance does have its stats and env…

In all honesty I don't think it's possible to put this into the model because it's not actually sufficiently understood yet by anyone or anything. AI capabilities are changing too fast, and so is the way AI is used (worst case, even including financial limitations that are now increasingly appearing.)

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#62

Post by tossandthrow »

ank wrote: post #59
tossandthrow wrote: post #28

Earlier quoted context omitted.

I think there might be a mixup here. You see: normally one reads for recreation or enlightenment. If that same one would approach their job as a recreational activity, or solely for enlightenment, you might not have that job for very long.

1. I am not sure what your profession is, but I spend most of my reading time at work, with code, emails and so on. Unfortunately I don't have much time to read for recreation these days. 2. It's a joke! Relax.

Yes, is you see my response to one of the parents, it should be apparent that it was clear to me.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#63

Post by helloplanets »

eqvinox wrote: post #61
helloplanets wrote: post #58

Earlier quoted context omitted.

There's pre and post training. What I was going after with "aware", is that the actual people working at the companies, training the models, are aware that people aren't mostly going to be implementing the plan by hand, if they've already made the plan in Claude Code or Codex. As for a specific Claude / GPT instance, "aware" would definitely be the wrong word choice there, but the instance does have its stats and env…

In all honesty I don't think it's possible to put this into the model because it's not actually sufficiently understood yet by anyone or anything . AI capabilities are changing too fast, and so is the way AI is used (worst case, even including financial limitations that are now increasingly appearing.)

Every part of training a model have the timing data observed + saved on multiple different axis, it's one of the most inherent parts of the training process.

When the post training run is nearing its cutoff point, there's a massive amount of data on how long coding tasks take to complete by that model in the golden format of "task -> time task took to complete", separable to whatever amount of subtasks, in the same format. With the parts from the end of the dataset being useful for evaluating the finished model's capabilites, whether that data is then fed back into another step in post training or not.

Completely separate from even the actual training: If you have a model proactively giving estimates that are an order of magnitude wrong, you can already fix the worst of it as of this moment by just changing the system prompt. It's a dirty fix, but it's the type of fix that has been used by Anthropic and OpenAI since forever when a model is dishing out blatantly wrong outputs.

Not sure if I'm misunderstanding your point?

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#64

Post by eqvinox »

helloplanets wrote: post #63
eqvinox wrote: post #61

Earlier quoted context omitted.

In all honesty I don't think it's possible to put this into the model because it's not actually sufficiently understood yet by anyone or anything . AI capabilities are changing too fast, and so is the way AI is used (worst case, even including financial limitations that are now increasingly appearing.)

Every part of training a model have the timing data observed + saved on multiple different axis, it's one of the most inherent parts of the training process. When the post training run is nearing its cutoff point, there's a massive amount of data on how long coding tasks take to complete by that model in the golden format of "task -> time task took to complete", separable to whatever amount of subtasks, in the same f…

The misunderstanding might be that I'm thinking about how long it takes a human + an AI to get something done, not an AI by itself. My current understanding is that it's still not generally viable to have LLMs perform architecture/design tasks (it might work in the short run but degrades roughly exponentially in the long run, accruing serious tech debt). Most things I'd consider actual "projects" involve at least some level of design work as well, and that'd need to be included in the estimate.

(And also prompts will need to be refined, etc., unless you have amazing prompt skills it won't immediately deliver what you wanted. Even if it did, the work to put together the AI inputs also needs to be included.)

⇒ I don't think it's easily possible right now to give a time estimate for **the full picture of** a project to be implemented with AI assistance.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#65

Post by cpinto »

firefoxd wrote: post #15

Unfortunately, this is the type of statements we can't verify. I'm not sure why these types of news are still coming out when we all have AI at work. Whenever someone does such a huge drastic change like this, it's by ignoring a large chunk of code that most people were afraid to touch for good reasons. Now, that code is gone, AI is celebrated, things will break, people will work very hard in the background to fix it…

But you _can_ verify if you had bothered: https://asana.com/inside-asana/migrating-off-enzyme-2-weeks

Key statement:

> But at the rate we were going, we were still roughly five years from finishing.

Everyone has seen how this sort of thing comes about: it's meaningful work for engineering but never business/product critical so it just drags along.

Seems like this time around someone just went "I wonder if we could do it this way" and it worked. Perfect example of ditching sunk-cost and starting from scratch. Great outcome for them.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#67

Post by zombot »

kzrdude wrote: post #47
ttul wrote: post #37

If you ask Sol or Claude how much time it will take to implement a plan they just came up with, they usually advise a timeframe in the weeks or months - assuming, I suppose, that human programmers will be building it. And then you ask the model to just "do it" and it takes an hour or two. I always find this entertaining.

Funny but shows how it doesn't have a useful world model

It does, assuming "world" is made up of humans (which it is, in its training data). It does however not have any model whatsoever of itself, apart from some superficial "I am a helpful assistant built by $X".

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#68

Post by tossandthrow »

greggoB wrote: post #52
tossandthrow wrote: post #22

Earlier quoted context omitted.

> where afraid to touch for good reasons This appears entirely unreasonable. Normally the reason is not good. The reason is that unit testing is missing or that downstream effects are not entirely mapped out. Exactly activities that traditional software developers are loathing because they are boring and mentally straining.

> This appears entirely unreasonable. Seems a bit strong. Unit testing can get you some of the way, but its not a full-on all-case guarantee. Sometimes the code is encapsulating some particularly complex system/behaviour. Sometimes the reason is interop/compatilibity issues or some kind of politics. P.S. you managed to introduce a typo in your quote (were –> where)

Oh, I did notice t introduce the typo. The commenter corrected after quoted.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#69

Post by Volundr »

cpinto wrote: post #65
firefoxd wrote: post #15

Unfortunately, this is the type of statements we can't verify. I'm not sure why these types of news are still coming out when we all have AI at work. Whenever someone does such a huge drastic change like this, it's by ignoring a large chunk of code that most people were afraid to touch for good reasons. Now, that code is gone, AI is celebrated, things will break, people will work very hard in the background to fix it…

But you _can_ verify if you had bothered: https://asana.com/inside-asana/migrating-off-enzyme-2-weeks Key statement: > But at the rate we were going, we were still roughly five years from finishing. Everyone has seen how this sort of thing comes about: it's meaningful work for engineering but never business/product critical so it just drags along. Seems like this time around someone just went "I wonder if we could do…

This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting.

> Back in 2022, we set out to migrate Asana's frontend test suite off Enzyme, our aging testing library, and onto React Testing Library (RTL).

Your telling me they had a full team of engineers at Asana, doing nothing but rewriting tests for 4 years until AI came along and did the last year in a couple days? I'm extremely doubtful. I don't doubt for a minute they were one-track to take 5 years, but not because it was 5 years of engineering effort for humans.

Far more likely they finally cleared up some tech debt they had been plugging at off and on for 4 years, then a PR flack got ahold of it and it became a breathless "AI did 5 years of work in a couple days".

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#70

Post by red-iron-pine »

internet_points wrote: post #60
crote wrote: post #24

Earlier quoted context omitted.

> I'm not sure why these types of news are still coming out when we all have AI at work. Because OpenAI is burning $15 billion/year, and outrageous stories like those get parroted in the media. It's free marketing for a company desperate to get middle management to believe that a $500/mo subscription is absolutely crucial for every single employee.

> parroted in the media ugh just recently I saw a news article gushing about how AI had helped with some health/medicine study, and various patient organizations were all like "oh yeah this is a Good use of AI" and then I click the link to read the study and it's decision trees and clustering on a tiny dataset that you could analyze with a ten year old laptop. I mean, sure at some point decision trees and clustering…

yeah but you'd need skills to operate the 10 year old laptop. probably have to copy and paste and understand excel and maybe whatever the hell a confidence interval is

this allows said middle managers to do the analysis themselves and feel mostly confident in the results.

Post reply on HN