Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What it feels like to work with Mythos

oneusefulthing.org

Post reply on HN
Print view View original

311–320 of 337 posts

Re: What it feels like to work with Mythos

#311

Post by brazzy »

eithed wrote: post #262
brazzy wrote: post #257

Earlier quoted context omitted.

> Don't forget that you can adjust your requirements (either via plan or skill) to ensure the mistakes do not happen. No, you can't. Adjusting prompts ensures absolutely nothing.

I disagree. What I should have added is that with agents (as well as humans) you do need to have tests that verify what was done.

That assumes you can write automated tests that reliably identify the mistakes over an entire codebase. Nice idea in theory. If it were actually possible, we would long since have generalized libraries of tests to catch every significant security and performance gotcha. What we have are static code analysis tools, fuzzers, etc. None of which have come close to eliminating security and performance problems. I don't see how AI somehome changes that.

Re: What it feels like to work with Mythos

#312

Post by queenkjuul »

chickensong wrote: post #140
eithed wrote: post #38

What I find fascinating that there is so little substance in this article about the quality of produced code and the medium. Is the code documented and tested? Is it understandable and extendable? Is it secure? What language, framework, database was used? Author mentions judgement and taste - well, is the code tasteful? Will the model rearchitecture the entire thing if I ask it to add new functionality, spending anot…

You probably don't care about the ingredients or engineering of asphalt, only if the road does its job well or is filled with potholes. Outside of the software industry, nobody gives a shit about code or databases.

I care that the engineer followed industry standard best practices and used high quality asphalt. How could i not care about that? How do you think potholes aren't related to the engineering of asphalt?

Re: What it feels like to work with Mythos

#313

Post by reedlaw »

CobrastanJorji wrote: post #198
jstummbillig wrote: post #56

Earlier quoted context omitted.

Less fascinating when you consider that this is a non-coders perspective.

It's still fascinating, but for a different reason. The "Concord" tool that got created bills itself as "Instrument-grade measurement of qualitative text. Explore in minutes, publish with honest statistics." Instrument-grade! How wonderful! That presumably means its accuracy has been ensured, and it's been carefully calibrated, right? What, nobody's ever measured or even examined the code? Well, no matter, let's go a…

Yeah, the README looks like slop to me.

Re: What it feels like to work with Mythos

#314

Post by eithed »

bluegatty wrote: post #205
olafmol wrote: post #64

This little line from the article scares me: "but a software engineer would iron out the remaining potential bugs that I could not find quickly" Every sw dev knows this is a very dangerous, and unrealistic, assumption.

it's basically a tiny statement that kind of hand waves all the 'actual stuff'.

https://www.danielzarick.com/uploads/2018-05-draw-the-owl.jp...

Re: What it feels like to work with Mythos

#315

Post by neilv »

an0malous wrote: post #114
eithed wrote: post #38

What I find fascinating that there is so little substance in this article about the quality of produced code and the medium. Is the code documented and tested? Is it understandable and extendable? Is it secure? What language, framework, database was used? Author mentions judgement and taste - well, is the code tasteful? Will the model rearchitecture the entire thing if I ask it to add new functionality, spending anot…

These posts are never written by software engineers, it’s always some tech exec, retired engineer, or VC. This author is apparently a professor at the Wharton School of Management? None of these people have to ship or maintain real products, they’re just making side projects. The only decent software engineering perspective I’ve seen has been from Mitchell Hashimoto.

Relevant quote:

> I am sure it is not perfect (I only spent an hour working with the results), but a software engineer would iron out the remaining potential bugs that I could not find quickly [...]

People have said things like this many times in the past, and, in the past (perhaps not now), it's always been a misunderstanding of what is good and bad, what's difficult and easy.

For example, someone would draw a UI in a GUI painter that generates code (or a resource file), and a manager would see it and think the majority of the work towards the product is done. (Incidentally, then there seemed to be a reaction, towards making your UI mockups look abstract or otherwise different from runnable code, helping the nontechical to understand that this isn't 90% of the finished product.)

Or a student intern hacks out a homework-grade demo, and a manager who understands neither software engineering nor product domain says "we just need some engineers to polish it up for production", and thinks the student is a star and why can't their engineers be as brilliant and productive. (I might have once been that energetic intern, who was happy for the encouragement, but then learned more, and saw it was a thing.)

This common misunderstanding was sometimes self-correcting -- when trying to ship became a disaster of misery and regretted-attrition, or the product was poorly received by the market because it wasn't thought through nor implemented well, or building subsequent functionality atop it was a nightmare. (But adverse effects of bad approaches is one of the reasons for management and ICs to job-hop, before the unwanted effects affect them personally.)

What might be different now is that some of these AI tools are outputting better-engineered work than some software engineers, and much faster.

At the back of my mind, I'm wondering how the really great software engineers will continue to stand out, as the discipline is being devalued in the minds of most leadership, and anyone can prompt an AI to generate something that superficially appears to them like what they assume a great software engineer would produce. (Even if the great engineer would do much better quality of implementation, have innovative ideas that ML from open source code would not, and maybe arrive at better product concepts as they worked through the problems.)

Re: What it feels like to work with Mythos

#316

Post by fractorial »

I’ve been falling back to Opus 4.6 since 4.7 and 4.8. I recently found success in using Opus 4.6 for cheap orchestration and reasoning while Opus 4.8 High/Max agents do the work.

I made serious progress towards repairing a proof for a conjecture that was published 10 days ago but kept running into a wall with one of the Lemmas.

I threw Fable 5 Max at it with the same subagent set up and in an hour it claimed to have disproved a core theorem of the paper.

The Lean construction looks correct, but I still need to verify it rigorously. This is certainly not something Opus 4.6 Max could do and it’s likely something Opus 4.8 Max could do with more delicate orchestration and time. However, the “one-shot” Fable 5 did give me pause.

Re: What it feels like to work with Mythos

#317

Post by orangecat »

geraneum wrote: post #207
mawadev wrote: post #66

Isn't it weird that we started to gauge the quality of a model by checking the vibe of the vibe coding?

You can see this all over the place. Under the Fable post in HN, you have simonw talking about the “feel” of working with Fable and how much better it is. If I believed in conspiracies, I’d have said it’s all orchestrated marketing…

And yet when there are objective results like "we ported Bun to Rust and 100% of the test suite passes" the response is "so what, the code is obviously crap and has tons of bugs that the tests don't cover".

Re: What it feels like to work with Mythos

#318

Post by xnx »

gofreddygo wrote: post #192
skydhash wrote: post #165

Earlier quoted context omitted.

The thing is that a lot of code rely on multiple layers of abstractions with their own correctness and failure states. And then you overlay the domain correctness and failure cases on top of that. But all of those correctness are imaginary. The hardware only enforce a few (and it may be buggy). The OS adds some more (and it’s buggy). The compiler/interpreter may have bugs (but that’s rarely a nuisance) and the librar…

> There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. That is so real. Brilliant !

Source: https://www.goodreads.com/quotes/21638-there-are-two-ways-of...

Re: What it feels like to work with Mythos

#319

Post by eithed »

brazzy wrote: post #311
eithed wrote: post #262

Earlier quoted context omitted.

I disagree. What I should have added is that with agents (as well as humans) you do need to have tests that verify what was done.

That assumes you can write automated tests that reliably identify the mistakes over an entire codebase. Nice idea in theory. If it were actually possible, we would long since have generalized libraries of tests to catch every significant security and performance gotcha. What we have are static code analysis tools, fuzzers, etc. None of which have come close to eliminating security and performance problems. I don't se…

Ah, I see what you mean now. Yes, my mind went straight to static analysis and testing (unit, feature, uat, mutation). Thanks for expanding on your point!

Re: What it feels like to work with Mythos

#320

Post by munksbeer »

skydhash wrote: post #310
munksbeer wrote: post #298

Earlier quoted context omitted.

> Are you seeing a quality increase? Less customer bugs, less outages, faster resolution? Are you measuring those? We're not at the stage to measure yet. We may be behind others, not sure. Actually, this isn't quite true. I was interested, so a created an ad-hoc report (with AI) on PRs landed per week over time. This has gone up over the last 6 momths. But that is hard to say why that is. It might just be people are…

> And then I come to HN or reddit and I see people telling us that they cannot write decent production code, and this is just wrong. This isn't opinion wrong, it is objectively wrong So how can you justify this comment of yours from your reply if you’re not measuring anything? Mind you, I can easily get good results from AI tools, but I don’t like the experience and the code is often over-engineered and drifts away f…

Ok. I get what you're saying. I definitely agree with losing touch with the code, but I still review everything the agent writes, and I steer it heavily. My perspective is from what I observe. The entire industry (my industry) is embracing agentic coding. And I don't believe developers are doing it because they're stupid.

I think it is going to continue to get better, and I don't think we'll be having this argument in two years time. Our entire industry will look very different.

Post reply on HN