Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The port I couldn't ship

ammil.industries

Post reply on HN
Print view View original

71–80 of 101 posts

Re: The port I couldn't ship

#71

Post by gyomu »

debois wrote: post #47

I recognize this part: > I don’t recall what happened next. I think I slipped into a malaise of models. 4-way split-paned worktrees, experiments with cloud agents, competing model runs and combative prompting. You’re trying to have the LLM solve some problem that you don’t really know how to solve yourself, and then you devolve into semi-random prompting in the hope that it’ll succeed. This approach has two problems:…

When I teach programming, some students, when stuck, will start flailing around - deleting random lines of code, changing call order, adding more functions, etc - and just hoping one of those things will “fix it” eventually.

This feels like the LLM-enabled version of this behavior (except that in the former case, students will quickly realize that what they’re doing is pointless and ask a peer or teacher for help; whereas maybe the LLM is a little too good at hijacking that and making its user feel like things are still on track).

The most important thing to teach is how to build an internal model of what is happening, identify which assumptions in your model are most likely to be faulty/improperly captured by the model, what experiments to carry out to test those assumptions…

In essence, what we call an “engineering mindset” and what good education should strive to teach.

Re: The port I couldn't ship

#72

Post by akrauss »

It is really important that such posts exist. There is the risk that we only hear about the wild successes and never the failures. But from the failures we learn much more.

One difference between this story and the various success stories is that the latter all had comprehensive test suites as part of the source material that agents could use to gain feedback without human intervention. This doesn’t seem to exist in this case, which may simply be the deal breaker.

Re: The port I couldn't ship

#73

Post by vbezhenar »

gyomu wrote: post #71
debois wrote: post #47

I recognize this part: > I don’t recall what happened next. I think I slipped into a malaise of models. 4-way split-paned worktrees, experiments with cloud agents, competing model runs and combative prompting. You’re trying to have the LLM solve some problem that you don’t really know how to solve yourself, and then you devolve into semi-random prompting in the hope that it’ll succeed. This approach has two problems:…

When I teach programming, some students, when stuck, will start flailing around - deleting random lines of code, changing call order, adding more functions, etc - and just hoping one of those things will “fix it” eventually. This feels like the LLM-enabled version of this behavior (except that in the former case, students will quickly realize that what they’re doing is pointless and ask a peer or teacher for help; wh…

Can LLM ask to help? I didn't see that. They can ask to clarify something, but they'll never admit they need help.

Re: The port I couldn't ship

#74

Post by dotancohen »

> The port I couldn't ship

It turns out that having a "trainer" to "coach" you is not a coincidence: these two words evolved together from the rail industry to the gym. Do "port" and "ship" have a similar history, evolving together from the maritime industry to software?

Re: The port I couldn't ship

#75

Post by dotancohen »

ruszki wrote: post #70
simonw wrote: post #67

Earlier quoted context omitted.

MicroQuickJS has 7, kind of: https://github.com/bellard/mquickjs/tree/main/tests Though if you look in those files some of them run a ton of test functions and assertions. My new Python library executes copies of the tests from that mquickjs repo - but those only count as 7 of the 400+ other tests.

Were the tests generated by an AI then? How do you know whether they are really comprehensive?

I'm now having Claude Code build the tests for my voicenotes organization application. For the most basic implementation - just a single text field - I wrote in English which tests I know I need, there were about two dozen. Approaching size limits, unicode, normalization, nonprinting characters, Hebrew vowel points, empty strings vs NULL strings, Exceeded byte length without exceeded character length, etc etc. I then threw Claude Code at it.

Claude Code found more edge cases to write tests for than I ever would have thought of. And I've been doing this for 20 years.

Re: The port I couldn't ship

#76

Post by magicalhippo »

johnfn wrote: post #21
jeffrallen wrote: post #12

Earlier quoted context omitted.

This is true, but also: everything I try works! I simply cannot come up with tasks the LLMs can't do, when running in agent mode, with a feedback loop available to them. Giving a clear goal, and giving the agent a way to measure it's progress towards that goal is incredibly powerful. With the problem in the original article, I might have asked it to generate 100 test cases, and run them with the original Perl. Then I…

Really, you haven't found a single task they can't do? I like agents, but this seems a little unrealistic? Recently, I asked Codex and Claude both to "give me a single command to capture a performance profile while running a playwright test". Codex worked on this one for at least 2 hours and never succeeded, even though it really isn't that hard.

I think I was using Grok Code 1 Fast with Cline, and had it trying to fix some code. Came back a bit later and found out that after not being able to make progress on fixing the code, it decided to "fix" the test by replacing it with a trivial test.

That made the test pass of course, leaving the code as broken as it ever was. Guess that one was on me though, I never specified it shouldn't do that...

Re: The port I couldn't ship

#77

Post by enraged_camel »

akrauss wrote: post #72

It is really important that such posts exist. There is the risk that we only hear about the wild successes and never the failures. But from the failures we learn much more. One difference between this story and the various success stories is that the latter all had comprehensive test suites as part of the source material that agents could use to gain feedback without human intervention. This doesn’t seem to exist in…

>> This doesn’t seem to exist in this case, which may simply be the deal breaker.

Perhaps, but perhaps not. The reason tests are valuable in these scenarios is they are actually a kind of system spec. LLMs can look at them to figure out how a system should (and should not) behave, and use that to guide the implementation.

I don’t see why regular specs (e.g. markdown files) could not serve the same purpose. Of course, most GitHub projects don’t include such files, but maybe that will change as time goes on.

Re: The port I couldn't ship

#78

Post by dotancohen »

simonw wrote: post #62
krackers wrote: post #38

Earlier quoted context omitted.

You should release it, it'd be quite useful.

https://pypi.org/project/micro-javascript/ - https://github.com/simonw/micro-javascript Here's the transcript showing how I built it: https://static.simonwillison.net/static/2025/claude-code-mic...

I see that you're no longer copying and pasting from the terminal, I remember those two gnarly code sessions to get a previous transcript. How are you generating that transcript now? I'd certainly like to use that for my own record keeping.

Re: The port I couldn't ship

#79

Post by 1317 »

simonw wrote: post #6

Funny to see this show up today since coincidentally I've had Claude code running for the past ~15 hours attempting to port MicroQuickJS to pure dependency-free Python, mainly as an experiment in how far a porting project can go but also because a sandboxed (memory constrained, to us time limits) JavaScript interpreter that runs in Python is something I really want to exist. I'm currently torn on whether to actually…

yt-dlp/youtube-dl used a python javascript interpreter to run youtube's JS until recently

idk how complete it is but it solved youtube's challenges etc for a long time

https://github.com/yt-dlp/yt-dlp/blob/6d92f87ddc40a319590976...

Re: The port I couldn't ship

#80

Post by [deleted] »

dotancohen wrote: post #75
ruszki wrote: post #70

Earlier quoted context omitted.

Were the tests generated by an AI then? How do you know whether they are really comprehensive?

I'm now having Claude Code build the tests for my voicenotes organization application. For the most basic implementation - just a single text field - I wrote in English which tests I know I need, there were about two dozen. Approaching size limits, unicode, normalization, nonprinting characters, Hebrew vowel points, empty strings vs NULL strings, Exceeded byte length without exceeded character length, etc etc. I then…

[deleted]
Post reply on HN