My main issue with many of these tests and reviews is that most of the results focus on testing the harness (in this case, likely Claude Code) rather than evaluating the model’s inherent performance.
What it feels like to work with Mythos
321–330 of 337 posts
Re: What it feels like to work with Mythos
#322Been working on my pet project today with Fable; it seems pretty solid but not too far removed from 4.8; same hallucinating, same type of bugs, same focus in large projects on just doing what you ask and just ignoring whatever that may touch/break/influence. Running tests in the beginning but when fuller context, just 'will run later' and never doing it in the end unless you tell it to (using some assorted swear word…
I'm having the opposite experience. Fable seems to anticipate everything and do it all without asking. It's been very impressive and great to work with.
Not exactly strange behavior, Opus acted just like this too when I first subscribed. The popupar meme is Anthropic nerfed Opus during their capacity cruch. No idea if it's true, but I do wonder if Fable will fall victim to the same fate.
Re: What it feels like to work with Mythos
#323Earlier quoted context omitted.
> Does AI make incredibly inefficient code most of the time? Yup. But it does it at lightspeed with minimal effort. This hits the nail in the head. Detractors often hang on to examples of coding assistants making mistakes or output subpar code, but they somehow miss the fact that coding assistants can also be prompted again and refactor whole swaths of code just as fast as they introduce oopsies. This means that the…
I think this is overlooking the fact that assigning a coding assistant to fix the bugs it re-introduces for all eternity just leads to spiraling token costs, which might cost more than just hiring a competent engineer in the first place.
Maybe. We will see.
I think computers are incredibly cheap compared to humans. These models and infrastructure to run them are going to only get more efficient in time. Right now we are still using (for the most part) entire hardware architectures mostly shoehorned from one purpose (graphics) into another. As purpose-built hardware becomes more prevalent and the SOTA starts to slow down I can't imagine a $100k hardware box not being able to handle a small team of developer's needs for many things.
I do think there will be a place for the top 20% of software engineers forever. But most people are not in that top 20%, and the quality when you get below average is not a linear progression. It will not be that difficult for AI generated code to beat the "bottom end" of the industry since tbh it's hard for me to tell the difference between LLM generated code and some of the shit I've seen over the years. I've ran across code written by folks who don't know what an array is more than once.
Most software is not built by MIT and Stanford grads making $500k/yr in the Valley. It's built by work-a-day programmers in the middle of nowhere making $80k/yr to keep some niche small business going with hyper-specific software that was first designed for Windows 95. Or stuff like making horribly designed Wordpress plugins. Or Shopify integrations. etc. etc.
I've also seen these small businesses totally held back by incompetent programmers, and despite their best efforts and huge amounts (for them!) of investment they can never seem to fix it. These types of enterprises are having AI run circles around their current engineering practices, even if it would make most FAANG engineers gasp in horror.
Either way it will certainly be interesting to watch! I just wish I was closer to retirement.
Re: What it feels like to work with Mythos
#324As a software engineer and solution provider, I do not feel threatened by this. I do not fear that management will get tools like Mythos and then not need people like me. Most of the value I provide is in translating what the management/client _thinks_ they need into what is the real problem and solution. That's not an insult to them, it's just pointing out that they see only their problem, and they imagine what woul…
I currently see the problem as follows: The knowledge worker like you sees the need for people like themselves to still be hired, and can reasonably argue for it. However, the management dudes and investors do not understand it, and it is difficult to make them understand, when their (short to medium term) profits depend on not understanding it. So whether you feel threatened or not, is just a matter of you feeling b…
You are exactly right. Regardless of whether AI is better than a human or not is irrelevant if the bad, unqualified corporate leaders are making rash silver bullet decisions that cost workers jobs.
The problem is much broader though - consolidation of wealth and power have enabled, frankly, idiots to be able to control how the world works - from politics to business. Greed and stupidity is eating the world.
I don't see any solution. This is like a disease that will either eventually kill the body or take a long time to heal, leaving deep scars and forever changing humanity.
Maybe War Games was right - the only way to win is not to play. Therefore, find something you love (even if it doesn't pay well), and do that.
(I spent two years looking for a tech job. My 30 years of broad and meaningful experience is apparently not interesting to at least the 200 companies I applied to. So now I'm a teacher, and I'm quite happy.)
Re: What it feels like to work with Mythos
#325So, the author gave the model single sentence prompts like "Balatro, but for the game of coin flips" plus generic encouragement like "make it better" three times ended up with netlify-hostable web games each time? That is hard for me to believe.
It's likely that at least some amount of additional context was provided to the model to enable it to reliably create the desired form factor. This introduces the caveat that the author probably views some amount of context as being trivial / beneath the level of mentioning. But then the question becomes where they draw the line.
Re: What it feels like to work with Mythos
#326Earlier quoted context omitted.
This is really insightful, but I think it also extends to making the project either low stakes or low complexity. I have this lurking feeling that the preferable architecture for software will change as a result of LLMs because they're good at working on low complexity modular components more than they are on high complexity million-line code bases.
You'll just shift complexity to the orchestration of the modular components. Monoliths vs micro-services.
There's some truth here. But in carefully orchestrated scenarios (the minority, to be sure), it can work surprisingly well, I think.
Re: What it feels like to work with Mythos
#327Earlier quoted context omitted.
> The problem is that this approach is not sustainable. Errors compound. The cost to fix one issue might seem small at first, but over a stretch of time all these "oopsies" become architectural spaghetti that can only be fixed with a complete rewrite, which will certainly become more expensive than getting the code "organically" developed. That's so far been called software development. All software developed by peop…
> Where exactly is the novelty? The compounding speed. Your devs might reach a point where they have to rewrite and refactor, in a decade. Your LLM, with its higher throughput, may put you in that game breaking situation next week.
> The compounding speed. Your devs might reach a point where they have to rewrite and refactor, in a decade.
I think that this is exactly why this scaremongering breaks down. If you believe the compounding speed is that greater, wouldn't you be compelled to accept that refactoring and cleaning things up is just as fast and effortless?
I mean, you have a tool that writes software for you following your commands. If you are that concerned with maintainability then what can possibly compell you to not invest any effort in it?
Re: What it feels like to work with Mythos
#328Earlier quoted context omitted.
> The problem is that this approach is not sustainable. Errors compound. The cost to fix one issue might seem small at first, but over a stretch of time all these "oopsies" become architectural spaghetti that can only be fixed with a complete rewrite, which will certainly become more expensive than getting the code "organically" developed. That's so far been called software development. All software developed by peop…
> All software developed by people suffers from this issue. And that’s pretty much where you are wrong. Take any long running open source project and you can see the craftsmanship that goes into it. It may not be perfect, but hacks are clearly marked as such.
> And that’s pretty much where you are wrong. Take any long running open source project and (...)
I think you are demonstrating a clear lack of insight and experience in software development settings, including FLOSS projects. I can name you a dozen of fairly known FLOSS projects which are a big ball of mud. Just go to the likes of GitHub, check out the list of popular projects, and peek at their code. You will get a very mixed set of results.
Re: What it feels like to work with Mythos
#329Earlier quoted context omitted.
I currently see the problem as follows: The knowledge worker like you sees the need for people like themselves to still be hired, and can reasonably argue for it. However, the management dudes and investors do not understand it, and it is difficult to make them understand, when their (short to medium term) profits depend on not understanding it. So whether you feel threatened or not, is just a matter of you feeling b…
You are exactly right. Regardless of whether AI is better than a human or not is irrelevant if the bad, unqualified corporate leaders are making rash silver bullet decisions that cost workers jobs. The problem is much broader though - consolidation of wealth and power have enabled, frankly, idiots to be able to control how the world works - from politics to business. Greed and stupidity is eating the world. I don't s…
Written like you read my mind, honestly.
I have a mere 10y of experience, but also already looking for 1 year and also considering maybe I should become a teacher. Dealing with unruly children might be nerve wracking, and the tech level will be very basic, but I have always enjoyed enabling others to understand things and see them grow and having done my part in that. Also solid well taught foundations are very important. Currently, my only obstacle is, that it is not so easy to become a teacher where I live. Certain requirements kind of like certifications, that you can't just easily get, but need to invest a lot of time into.
Good to read of someone, who went that way. How did you manage the transition?
Re: What it feels like to work with Mythos
#330The outputs are impressive because a machine made them, but they're not impressive in themselves.
Given a tool that is supposed to unlock creativity and excitement, he made a series of worse clones of things.
Again, technically impressive, but the world has never needed the ability to make Balatro but less polished and coherent. We already have Balatro.
I'd be more convinced if people made things that didn't already exist; show me that these tools enable something you actually want.