Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asana cleared 5 years of engineering work in 2 weeks with Codex

openai.com

Post reply on HN
Print view View original

91–100 of 103 posts

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#91

Post by Volundr »

Supermancho wrote: post #73
Volundr wrote: post #69

Earlier quoted context omitted.

This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting. > Back in 2022, we set out to migrate Asana's frontend test suite…

> I can't see the code before, I can't see the code after. I can't verify they do the same thing. The ultimate bad faith interpretation. "Unless I can verify the results that contradict my worldview, I don't acknowledge them." > https://www.mikekasberg.com/blog/2026/08/19/hacking-with-cla... "I haven't done this, so this doesn't prove it." > https://www.bbc.com/news/articles/clyq011414eo "I haven't seen the paper tra…

The comment I was replying to said

> But you _can_ verify if you had bothered

I think pointing out I can't is indeed fair.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#94

Post by Volundr »

NorthSouthNorth wrote: post #75
Volundr wrote: post #69

Earlier quoted context omitted.

This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting. > Back in 2022, we set out to migrate Asana's frontend test suite…

Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify (I mean that is the nature of them lol). With enough budget this seems a rather reasonable and fun task.

I'm not sure I agree with the extremely easy to verify part. How do you validate that your new test suite covers the exact same edge cases as your old?

But yes, LLMs are great at tests. I'm not doubting that an LLM migrated some legacy tests, or that it was a task taking a long time. I am doubting the way it was presented in the article, that this was taking a full team of engineers dedicated to nothing but this, 5 years to accomplish (and presumably already spent 4 million working on this, since the total estimate was 6 million, and they've been at it for 4 year).

I think some PR flack got ahold of the fact that an LLM wrapped up migrating some legacy tests that a team had been slowly chipping away at for 4 years, between their normal feature work, and were on track to finish in 5, then wrote it up like it was that teams entire focus instead of a piece of tech debt.

That makes far more sense to me than spending millions for a team of engineers dedicated to nothing but rewriting an existing test suite.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#95

Post by avgDev »

amluto wrote: post #88
avgDev wrote: post #81

Earlier quoted context omitted.

It has it risks too. I've used chatGPT for a feature I am working on. It works, I have already resolved some bugs the it didn't pick up. However, the code quality is odd, I've had some hardcoded types as it is unlikely they will ever change. For some of them chatGPT created enums but only for some. I also realized that review generated code sucks, I can generate a whole app quickly, but I have no understanding of sma…

If you mean chatgpt.com, try with a real “agent” harness. The exact same model on chatgpt.com frequently performs much worse than it does with a proper harness, not to mention that the chatgpt.com frontend itself is quite bad. codex-rs, popularly known as Codex or maybe the Codex CLI, is a fine choice.

Codex has been renamed to chatGPT.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#96

Post by amluto »

avgDev wrote: post #95
amluto wrote: post #88

Earlier quoted context omitted.

If you mean chatgpt.com, try with a real “agent” harness. The exact same model on chatgpt.com frequently performs much worse than it does with a proper harness, not to mention that the chatgpt.com frontend itself is quite bad. codex-rs, popularly known as Codex or maybe the Codex CLI, is a fine choice.

Codex has been renamed to chatGPT.

Since when?

https://learn.chatgpt.com/docs/codex/cli

https://github.com/openai/codex

Looks like it's still a more-or-less-open-source product called Codex CLI.

FWIW, there have been issues with various versions of ChatGPT not knowing what Codex is -- at least in the (IIRC) GPT-5.3 timeframe, there were serious issues with the knowledge cutoff.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#97

Post by avgDev »

amluto wrote: post #96
avgDev wrote: post #95

Earlier quoted context omitted.

Codex has been renamed to chatGPT.

Since when? https://learn.chatgpt.com/docs/codex/cli https://github.com/openai/codex Looks like it's still a more-or-less-open-source product called Codex CLI. FWIW, there have been issues with various versions of ChatGPT not knowing what Codex is -- at least in the (IIRC) GPT-5.3 timeframe, there were serious issues with the knowledge cutoff.

They did it about a month ago for the desktop Codex app. It is actually really confusing and there are many discussions on reddit.

First, the app was called Codex, then they combined them and now the app is called chatGPT, but recently inside the app you can switch between ChatGPT and Codex. lol.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#98

Post by kelnos »

firefoxd wrote: post #15

Unfortunately, this is the type of statements we can't verify. I'm not sure why these types of news are still coming out when we all have AI at work. Whenever someone does such a huge drastic change like this, it's by ignoring a large chunk of code that most people were afraid to touch for good reasons. Now, that code is gone, AI is celebrated, things will break, people will work very hard in the background to fix it…

> I'm not sure why these types of news are still coming out when we all have AI at work.

Because "we all" is a bubble, and many people do not have AI at work, or at least not the level of usage that many people here have the budget for at their company.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#99

Post by kelnos »

NorthSouthNorth wrote: post #75
Volundr wrote: post #69

Earlier quoted context omitted.

This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting. > Back in 2022, we set out to migrate Asana's frontend test suite…

Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify (I mean that is the nature of them lol). With enough budget this seems a rather reasonable and fun task.

> Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify

I don't think that's true. All you see is all the test pass. You don't know that the tests still cover everything that they used to.

LLMs excel where there is an excellent test suite, and you ask them to modify the thing that the test suite tests, and forbid them from changing the tests.

Re: Asana cleared 5 years of engineering work in 2 weeks with Codex

#100

Post by kelnos »

Supermancho wrote: post #73
Volundr wrote: post #69

Earlier quoted context omitted.

This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting. > Back in 2022, we set out to migrate Asana's frontend test suite…

> I can't see the code before, I can't see the code after. I can't verify they do the same thing. The ultimate bad faith interpretation. "Unless I can verify the results that contradict my worldview, I don't acknowledge them." > https://www.mikekasberg.com/blog/2026/08/19/hacking-with-cla... "I haven't done this, so this doesn't prove it." > https://www.bbc.com/news/articles/clyq011414eo "I haven't seen the paper tra…

How is it bad faith to question corporate marketing blog posts? That's not bad faith, that's table stakes for critical thinking.

We absolutely should be skeptical when an AI company makes big claims. The fact that the company the AI company is talking about also claims the same thing doesn't change that. OpenAI is getting awareness and marketing out of this, and I'm sure Asana is getting something out of it too.

And I'm not even saying OpenAI or Asana are necessarily lying. Asana might not find out for months or years that some tests had been rewritten poorly, and don't sufficiently test the thing they were supposed to test anymore. For example. If they truly had 5 years of work, then I find it hard to believe that in two weeks of the LLM churning, they had the time to review all the new tests. They spot-checked, at best.

Maybe everything is great. Maybe the LLM did a wonderful job, and this was awesome for Asana. But we have no idea, and we're unlikely to ever find out. Unless, of course, it's in Asana's interest from a marketing perspective to tell us.

(Not sure what the URLs you posted in your comment are supposed to prove. They're unrelated to the issue at hand.)

Post reply on HN